ModuleNotFoundError: No module named 'conda._vendor.auxlib'

[B-1991-ing]

Dear binny support team,

I used binny, but error happened as shown below. Could you help me check what is the reason?

Error screenshot

Error file binny_25.err-25.txt

Log file binny_25.log-25.txt

Job script binny_25.sh.txt

Best,

Bing

[ohickl]

Dear bing, this could be a mamba/conda version issue as in e.g. this one. You could try updating one/both and see if that resolves it.

Best Oskar

[B-1991-ing]

Dear Oskar,

Thank you very much for your reply.

Yeah, I saw this, I also tried "conda update conda-build" under my own project directory, didn't work.

So, I want to try to install under my own directory again, as the HPC admin insisted that it won's work like the current way.

Best,

Bing

[B-1991-ing]

Dear Oskar,

Is it the form of the VARIABLE_CONFIG only worked on the SCHEDULER system - slurm, slurm_simple or sge?

Best,

Bing

[B-1991-ing]

Dear Oskar,

What I tried under my PROJECT directory three minutes ago.

Best,

Bing

[B-1991-ing]

I just tried the 10 versions of the miniconda as shown below. Same error every time.

[ohickl]

Yeah, I saw this, I also tried "conda update conda-build" under my own project directory, didn't work.

Hm, and that happens when letting binny create a Snakemake environment? You could try to do a fresh conda install.

Is it the form of the VARIABLE_CONFIG only worked on the SCHEDULER system - slurm, slurm_simple or sge?

Atm we ship these as default. Well check, how we can support moab/torque and get back to you. In the meantime you could submit a job using snakemake directly or use the wrapper in laptop mode (-l) inside a bash job script.

What I tried under my PROJECT directory three minutes ago.

It seems to have trouble loading your conda module. Does it work fine, if you just call it yourself? There is also a bug in the wrapper atm, regarding the correct order of loading the modules and needing conda, if the user has no conda in their path. It will be fixed soon. You could try loading the conda module before calling the wrapper, might work.

[B-1991-ing]

It seems to have trouble loading your conda module. Does it work fine, if you just call it yourself?

I think it works if I directly load any miniconda on the HPC terminal under my PROJECT directory as shown below. module load tools computerome_utils/2.0 module load miniconda3/4.12.0

I also changed "module load miniconda3/4.12.0" to "module load tools computerome_utils/2.0 miniconda3/4.12.0".

Then, I got screenshot as shown below.

[B-1991-ing]

Atm we ship these as default. Well check, how we can support moab/torque and get back to you. In the meantime you could submit a job using snakemake directly or use the wrapper in laptop mode (-l) inside a bash job script.

If I submit a job using snakemake directly, I still need to use the parameter "--use conda"? Actually, I tried to use the installed binny on the HPC cluster using the command line below, not worked. The output was attached at the beginning.

CONFIGFILE=/home/projects/env10000/people/binson/MTB/binning/binny${PBS_ARRAYID}/config.yaml snakemake -s /services/tools/binny/20220610/Snakefile --configfile ${CONFIGFILE} --use-conda --cores 1

[ohickl]

Ok, the loading looks fine now. It crashes because of the bug i mentioned. The wrapper currently loads the envs after it tries to use conda/mamba to create the snakemake env. well fix it asap and get back to you.

[ohickl]

If I submit a job using snakemake directly, I still need to use the parameter "--use conda"?

Yes. And id also use --conda-prefix to specify the conda envs dir so you could place it in a shared dir, if you wanted to.

The output was attached at the beginning.

Ah, the ModuleNotFoundError: No module named 'conda._vendor.auxlib' issue? Looking at the submission script, I am unsure, which module loads mamba. Maybe one of the modules loads a version that is incompatible with the conda version loaded?

[B-1991-ing]

Yes. And id also use --conda-prefix to specify the conda envs dir so you could place it in a shared dir, if you wanted to.

For the binny I just tried to installed under my PROJECT directory - /home/projects/env_10000/people/binson/binny, I don's think there is a well created conda env.

For the binny installed on the HPC cluster by my HPC admin - /home/projects/env_10000/people/binson/binny, I tried to set the parameter - conda_source: "/services/tools/binny/20220610/conda-meta" or the parameter - conda_source: "/services/tools/binny/20220610/condabin" in the config.yaml, then I got the output attached at the beginning.

I think I first need to have a well created conda env, then I can use the --conda-prefix?

[B-1991-ing]

Ah, the ModuleNotFoundError: No module named 'conda._vendor.auxlib' issue? Looking at the submission script, I am unsure, which module loads mamba. Maybe one of the modules loads a version that is incompatible with the conda version loaded?

I think maybe the snakemake command line firstly wanted to use the conda make some necessary env, but failed, then it tried to use the mamba created necessary env?

[B-1991-ing]

Ah, the ModuleNotFoundError: No module named 'conda._vendor.auxlib' issue? Looking at the submission script, I am unsure, which module loads mamba. Maybe one of the modules loads a version that is incompatible with the conda version loaded?

Here are the modules I loaded for running the shell script.

Load all required modules for the job

module load tools module load ngs module load anaconda3/2019.10 module load snakemake/6.0.4 module load binny/20220610

[ohickl]

For the binny I just tried to installed under ... I don's think there is a well created conda env.

No, as it failed creating them. the dir is probably empty.

I tried to set the parameter - conda_source ...

conda_source sets the path to a conda environments dir, its like --conda-prefix. Im unsure, what your admin did for the module. since there are conda and database dirs, I suspect he succesfully installed binny in that dir? Are there files/dirs in those?

I think maybe the snakemake command line firstly wanted to use the conda make some necessary env, but failed, then it tried to use the mamba created necessary env?

Snakemake uses mamba by default nowadays. So it strangely fails to use mamba internally, because of a problem with conda. I have not seen this problem with binny before. Ill check if there is something that can be done fromn our side. Can you check the snakemake, conda and mamba versions in the binny/snakemake env you used?

[ohickl]

Also, I just saw :

module load snakemake/6.0.4

binny needs at least 6.9.1, this could also be the/a source for the problem(s)

[B-1991-ing]

conda_source sets the path to a conda environments dir, its like --conda-prefix. Im unsure, what your admin did for the module. since there are conda and database dirs, I suspect he succesfully installed binny in that dir? Are there files/dirs in those?

I think he successfully installed it, but he just thinks it can't be working in this share installation.

Folder conda is empty.

Folder condabin.

Folder conda-meta.

database

[ohickl]

Ok, the database dir looks good. it seems like the binny module is supposed to supply conda and mamba already? So if you just load that one module, which versions of conda and mamba do you have?

[B-1991-ing]

conda_source sets the path to a conda environments dir, its like --conda-prefix. Im unsure, what your admin did for the module. since there are conda and database dirs, I suspect he succesfully installed binny in that dir? Are there files/dirs in those?

Should I set the conda source as condabin or conda-meta?

[ohickl]

Neither, ideally set it to the absolute path of conda, if you have write access to the binny module dir. might not be the case though. Othwise I'd choose somwhere appropriate in your user space.

[B-1991-ing]

Ok, the database dir looks good. it seems like the binny module is supposed to supply conda and mamba already? So if you just load that one module, which versions of conda and mamba do you have?

I can't see the conda version the admin used when he installed, from the VARIABLE_CONFIG.

[ohickl]

Maybe he didnt load any module, because he has conda in his PATH already. I meant, after you load the binny module with module load binny/20220610, what do you get e.g. with: which conda/which mamba

[B-1991-ing]

Neither, ideally set it to the absolute path of conda, if you have write access to the binny module dir. might not be the case though. Othwise I'd choose somwhere appropriate in your user space.

Yeah, as the HPC admin installed binny on the HPC cluster, and created the condabin and conda-meta dirs under the binny directory already. Only if I install the binny under my own PROJECT directory, then I can set the conda PATH under my own PROJECT directory.

[B-1991-ing]

Maybe he didnt load any module, because he has conda in his PATH already. I meant, after you load the binny module with module load binny/20220610, what do you get e.g. with: which conda/which mamba

Maybe not this way?

[B-1991-ing]

Maybe he didnt load any module, because he has conda in his PATH already. I meant, after you load the binny module with module load binny/20220610, what do you get e.g. with: which conda/which mamba

But nothing in the directory - /services/tools/binny/20220610/conda.

[ohickl]

Maybe not this way?

This actually looks like it should, I think. You get both, conda and mamba by just loading the binny module. Now if you run binny -i path/to/my_config.init.yaml (I assume the module also adds binny to your PATH) with conda_source: "absolute/path/to/conda/env/dir/in/user/space" in my_config.init.yaml it still fails? Since the database folder is already there it should only need to write to you chosen conda dir. You can also check the mamba and conda versions with mamba --version, just to be sure they installed recent versions in that module.

[B-1991-ing]

conda_source: "absolute/path/to/conda/env/dir/in/user/space"

The PATH I want to put the conda env?

I set conda_source: "/home/projects/env_10000/people/binson/binny/conda" config.init.yaml.txt

[B-1991-ing]

mamba --version

[ohickl]

Ok, theres the problem. Something seems to be wrong with the modules mamba/conda versions, either broken or incompatible, I assume. The config file looks good. Once we fix the wrapper bug (hopefully today or tomorrow) you could maybe ask the admin to rebuild the module, putting the conda envs in the modules conda dir and ensuring e.g.: mamba>=0.22.1, conda-build>=3.21.8 and conda>=4.12.0. Otherwise you can just load a recent conda module and do the personal install again, which then should work.

[B-1991-ing]

Okay, once you fix the bug, I will then message the HPC admin. Thank you very much.

[B-1991-ing]

Hi Oskar,

I saw you made some changes 2 hours ago.

Did you already fix the wrapper bug?

Best,

Bing

[ohickl]

Hi Bing,

Yes, you can give it a try.

[B-1991-ing]

One thing is that when I change the default kmers: '2,3,4' to kmers: '21,33,55' the running time would be longer or shorter?

[ohickl]

It would be extremely long, take incredible amounts of ram, and also would probably have terrible performance. I dont recommend it. Id stick with the default for now. You could add 5, or only go with 4/5, but I saw worse performance with that.

[B-1991-ing]

I tried to install binny under my PROJECT directory with miniconda3/4.12.0, with error, I will try to install binny with other versions.

[ohickl]

Never saw that before. Found this though. Its pretty recent, might be a Mamba bug. You could try this. Are you able to install anything with that miniconda module? E.g. do mamba create -n 'test' python=3.10 (you can delete it after).

[B-1991-ing]

I think I can install something with the miniconda module.

[ohickl]

Mamba definitely should work with anaconda3. At this point i think its just easier for you to install you own conda, its pretty straight forward. e.g. with miniconda or mambaforge. As i am not familiar with your HPCs setup and module environments, its hard to tell which combination of things works and is stable. If you have your own, you dont need to worry about that.

[B-1991-ing]

Actually I already tried to install the miniconda under my own PROJECT directory - /home/projects/env_10000/people/binson/, but it showed that no space left. I think the miniconda automatically saved something under my PERSONAL directory - /home/people/binson/.

Thank you very much anyway. I also already asked the HPC admin to have a reinstallation of the binny.

[ohickl]

but it showed that no space left

It should not need a lot. You should ask you admins, which space the most appropriate to install user tools to is. There should not be any space problems with a basic (mini)conda install.

I also already asked the HPC admin to have a reinstallation of the binny

Ok, hopefully it will work out.

[B-1991-ing]

Hi Oskar,

My HPC admin installed the new version of binny. But, when I left empty for the location of snakemake, why the binny will install the snakemake env into my PERSONAL directory (maximum memory 10GB) instead of my PROJECT directory? Normally, the HPC users are required to install and store all things under their PROJECT directory - /home/projects/env_10000/people/binson/.

The modules I loaded and the command lines I ran: module load tools module load ngs module load miniconda3/4.12.0 module load snakemake/7.8.2 module load binny/20220623 CONFIGFILE=/home/projects/env10000/people/binson/MTB/binning/binny${PBS_ARRAYID}/config.yaml snakemake -s /services/tools/binny/20220623/Snakefile --configfile ${CONFIGFILE} --use-conda --cores 20

CONFIGFILE=/home/projects/env10000/people/binson/MTB/binning/binny${PBS_ARRAYID}/config.yaml config.yaml.txt

Best,

Bing

[ohickl]

I would only load the binny module since its supposed to have conda and mamba. It looks like snakemake is already installed in the binny environment and wants to do some caching in your home dir, not install. But it should take very little space and be fine. Could it be that your user dir is completely full? if so, id clear up some space and try again. You can also check if theres a snakemake_env dir in /services/tools/binny/20220623/conda

[B-1991-ing]

When I set snakemake_env: "/services/tools/binny/20220623/conda/snakemake_env", it showed like below.

config.yaml.txt

[B-1991-ing]

You can also check if theres a snakemake_env dir in /services/tools/binny/20220623/conda

snakemake_env is under /services/tools/binny/20220623/conda, but I *didn't find yaml file** here.

[B-1991-ing]

I set conda_source: "/services/tools/binny/20220623/condabin", as I saw conda is under this dir.

[ohickl]

When I set snakemake_env: "/services/tools/binny/20220623/conda/snakemake_env", it showed like below.

This is strange, .snakemake is created relative to the work directory, which in this case is your output dir in your project folder. And I assume you have write access to that one. You could try to set it manually by supplying the snakemake call with the parameter --shadow-prefix /path/to/my/dir to put it there instead

snakemake_env is under /services/tools/binny/20220623/conda, but I *didn't find yaml file here.

That is correct, snakemake copies the yaml files from the workflow envs only and the snakemake env is not part of that. But I see that there are no other environments in there. So the admin did not run the setup? Or it failed, since the snakemake environment is there. Strange.

I set conda_source: "/services/tools/binny/20220623/condabin", as I saw conda is under this dir.

conda_source is the path to/to put the conda environments needed, not to the actual conda executable. You need to set it to a directory that you have write access to. /services/tools/binny/20220623/condabin will not work.

Also, I saw that you are still loading modules, while also loading the binny module, in your VARIBALE_CONFIG, which I think is unnecessary and might lead to conflicts.

In general, I dont see the module approach working out very well atm, especially, since there are no conda envs. I would create a directory in your project folder, which should have penty of space, e.g. called local_tools, install miniconda/miniforge to that location and then also load and setup binny there. That should ensure no permission or space problems. I am away until Tuesday and wont be able to respond in the meantime. If it still doesnt work by then, we can see what to do next. I am sorry, that you are having such difficulties. Hope it will work out now

[B-1991-ing]

This is strange, .snakemake is created relative to the work directory, which in this case is your output dir in your project folder. And I assume you have write access to that one. You could try to set it manually by supplying the snakemake call with the parameter --shadow-prefix /path/to/my/dir to put it there instead

Then, I don't need to set snakemake_env: "/services/tools/binny/20220623/conda/snakemake_env" in my config.yaml?

[B-1991-ing]

That is correct, snakemake copies the yaml files from the workflow envs only and the snakemake env is not part of that. But I see that there are no other environments in there. So the admin did not run the setup? Or it failed, since the snakemake environment is there. Strange.

For the prokka_env and mantis_env, I found they are under the directory - /services/tools/binny/20220623/workflow/envs.

[B-1991-ing]

I would create a directory in your project folder, which should have penty of space, e.g. called local_tools, install miniconda/miniforge to that location and then also load and setup binny there.

For this, I already tried to install miniconda under my PROJECT directory. I will try it again maybe.

[ohickl]

Then, I don't need to set snakemake_env: "/services/tools/binny/20220623/conda/snakemake_env" in my config.yaml?

For the module you should. Otherwise it tries to install snakemake in your conda_source location, if theres no snakemake in PATH.

[B-1991-ing]

Thank you for your reply, again.

I tried to install the miniconda three times, always got no space error, although I already specified my PROJECT directory.

Is any way we just run the binny script and directly load the necessary modules - snakemake, prokka, and mantis, etc, instead of creating different conda environments?