issue with tb-profiler + conda (+snakemake)

[pmenzel]

This is a fringe issue that I ran into when running TB-profiler through a conda installation that is shared by multiple users. I just want to write it down for posterity, so that the error message can be found by others (or myself) in the future.

Steps:

• have a miniconda3 installation that is used by multiple users (all in the same group) for multiple environments, using bioconda
• as user X do:

  conda create -n tbprofiler420
  conda activate tbprofiler420
  mamba install tb-profiler=4.2.0 -y
  tb-profiler update_tbdb --branch who

  Next, user Y, using the same miniconda3 installation, runs a snakemake workflow containing a rule that loads a conda yaml file, which also contains tb-profiler 4.2.0, using snakemake --use-conda --conda-prefix /opt/conda/by-snakemake ..., and executes the command tb-profiler update_tbdb --branch who.
  This will result in the error:

  
  TBProfiler version 4.2.0

Running command: set -u pipefail; git clone https://github.com/jodyphelan/tbdb.git

Running command: set -u pipefail; git checkout who

Running command: set -u pipefail; git pull

Running command: set -u pipefail; tb-profiler create_db --load Traceback (most recent call last): File "/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/bin/tb-profiler", line 566, in args.func(args) File "/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/bin/tb-profiler", line 200, in main_update_tbdb pp.run_cmd("tb-profiler create_db %s --load" % tmp) File "/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/lib/python3.9/site-packages/pathogenprofiler/utils.py", line 391, in run_cmd raise ValueError("Command Failed:\n%s\nstderr:\n%s" % (cmd,stderr.decode())) ValueError: Command Failed: set -u pipefail; tb-profiler create_db --load stderr: ESC[94mConverting 580 mutationsESC[0m

Running command: set -u pipefail; snpEff ann Mycobacterium_tuberculosis_h37rv 0eacf2ff-ac5e-426e-8a20-1fc06504de2e ESC[94mConverting 689 mutationsESC[0m

Running command: set -u pipefail; snpEff ann Mycobacterium_tuberculosis_h37rv 6c7cb704-1f82-4e16-b5ad-ac54eee90c18

Running command: set -u pipefail; git log | head -4 ESC[94mCopying file: tbdb.fasta ---> /opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/share/tbprofiler/tbdb.fastaESC[0m Traceback (most recent call last): File "/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/bin/tb-profiler", line 566, in args.func(args) File "/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/bin/tb-profiler", line 210, in main_create_db pp.create_db(args,extra_files=extra_files) File "/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/lib/python3.9/site-packages/pathogenprofiler/db.py", line 679, in create_db shutil.copy(val,target) File "/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/lib/python3.9/shutil.py", line 428, in copy copymode(src, dst, follow_symlinks=follow_symlinks) File "/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/lib/python3.9/shutil.py", line 317, in copymode chmod_func(dst, stat.S_IMODE(st.st_mode)) PermissionError: [Errno 1] Operation not permitted: '/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/share/tbprofiler/tbdb.fasta' Cleaning up after failed run [...]

The issue is that copying `tbdb.fasta` to `/opt/conda/by-snakemake/17feee8282d3b1f4544d1531017653a2/share/tbprofiler/tbdb.fasta` fails, because the latter still belongs to user X, even so a new conda environment was created by user Y.
The used `shutil.copy()` function also wants to overwrite the file permissions, which is not allowed due to file ownership, see also this issue: https://stackoverflow.com/questions/11835833/why-would-shutil-copy-raise-a-permission-exception-when-cp-doesnt

The solution would either be to remove all conda environments containing tb-profiler 4.2.0, cleaning all conda caches, and running snakemake again as Y.

Or one could replace all calls to `shutil.copy()` in `lib/python3.9/site-packages/pathogenprofiler/db.py` by `shutil.copyfile()`, which does not try to change the permissions of the target file, which will just successfully copy the file.

So, it is not a tb-profiler issue per se, but maybe it is still worthwhile to replace `shutil.copy()` by `shutil.copyfile()` in `pathogenprofiler/db.py` to prevent this or similar permission errors.

[tseemann]

Users sharing conda environments is a tricky situation.

One way to handle it is to alter tb-profiler to use a user-local version of the database first, and if that does not exist, use the global ome.

The local place would normally be $HOME/.local/share/tb-profiler/ folder.

[jodyphelan]

Oh yeah that is a good idea.

Is there any situation in which tb-profiler wouldn't have permission to create/write to $HOME/.local/share/tb-profiler/? or is .local a pretty safe bet, permissions-wise?