search

dib-lab / farm-notes

notes on the farm cluster
16 stars 9 forks source link

putting conda environments outside your home directory #70

Open ctb opened 1 month ago

ctb commented 1 month ago

The way farm is working, going forward, is to put all new accounts in their own 20G home directory -

% df -h ~/
Filesystem                        Size  Used Avail Use% Mounted on
nas-4-1-ib:/nas-4-1/home/ctbrown   20G   14G  6.4G  69% /home/ctbrown

This is as opposed to most of you, who are on the old-style “let’s just dump their home directory on a working disk”. (I asked for mine to be transitioned so that I would experience the frustrations of new users :wink:

There’s lots of good reasons for this, the main two being (1) it frees people from being dependent on a particular group, and (2) it means that even when the working disk gets filled up, your home directory remains open for business - and, importantly, your space is under your control.

The problem is that 20G is not that much if you’re doing a lot with conda!

So anyway what Sam needs to know is not just how to clean up conda space and so on -- which in this case is hard because she’s not the only person on the disk -- but how to put conda environments somewhere else.

There are two answers to give here -

FIRST, you can and should use conda clean -a routinely. It will delete lots of unused stuff.

SECOND, you can put conda environments in any free space you like! You can do something like: mamba create -p /tmp/conda-test -y python=3.12 and it will happily put your conda env there. And then you can do conda activate /tmp/conda-test.

HTH :)

mr-eyes commented 1 month ago

I have this new system in my genome center HPC account. I symlinked the conda directory to my home directory to continue working without minding any path changes.

ctb commented 1 month ago

so in this case you're talking about just relocating the entire conda directory, right @mr-eyes? That can be a bit more involved - there's a .conda directory sometimes, too. We should look into exactly what is stored where :)

mr-eyes commented 1 month ago

Yes, exactly! I haven't encountered any issues until now. And yes, there is a .conda/, but keeping it in the home directory wouldn't be a problem and will work smoothly with the conda dir.