ncov_tools bad interpreter: Permission denied error

[dgaston]

Hello,

I keep getting this error now on multiple machines and with fresh installs/set-ups of signal. Running the command:

python signal.py --configfile config.yaml --cores 24 ncov_tools

I get this error on the following step: make_amplicon_steps with the error message:

/bin/bash: /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.snakemake/conda/47aef19e0702f2843c364d0c2fe3ede9/bin/primers_to_amplicons.py: /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.: bad interpreter: Permission denied [Mon May 30 10:25:22 2022] Error in rule make_amplicon_bed: jobid: 3 output: bed/amplicon.bed shell: primers_to_amplicons.py --primers /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/resources/primer_schemes/freed/nCoV-2019.bed --offset 0 --bed_type unique_amplicons --output bed/amplicon.bed --primer_prefix nCoV-2019 (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

To test I ran chmod -R 777 on the .snakemake directory so that all conda environments and files would be given maximal permissions and it doesn't make any difference.

[rdeborja]

@dgaston what happens if you run primers_to_amplicons.py on the command line outside signal and ncov-tools?

cd /path/to/qc/directory
primers_to_amplicons.py --primers /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/resources/primer_schemes/freed/nCoV-2019.bed --offset 0 --bed_type unique_amplicons --output bed/amplicon.bed --primer_prefix nCoV-2019

Also, even though you manually set the permissions to the entire conda environment, what are the permissions for the script?

ls -la /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.snakemake/conda/47aef19e0702f2843c364d0c2fe3ede9/bin/primers_to_amplicons.py

I just checked our and it should be:

-rwxrwxr-x

[dgaston]

For the permissions it looks like it inherited the 777 as expected:

-rwxrwxrwx

If I copy the primers_to_amplicons.py script into another directory and run with my base conda environment, or the base signal conda environment:

-bash: ./primers_to_amplicons.py: /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.: bad interpreter: Permission denied

The interpreter on the shebang line of the primers_to_amplicons.py script is:

/mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.snakemake/conda/47aef19e0702f2843c364d0c2fe3ede9/bin/python

and if I run that on its own it loads up the python interpreter just fine with no errors:

Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) [GCC 10.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

If I change that to a more standard #!/usr/bin/env python line, the script will then execute (I get an input error of course as the ncov module isn't in the same directory)

[jts]

This looks like a conda environment issue, the script is looking up the python interpreter using /usr/bin/env python, like the rest of the scripts in ncov-tools. I don't know why this is resolving to /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/. instead of the python path. @fmaguire?

[dgaston]

Yeah, if I set the shebang line of primers_to_amplicons.py, get_qc.py and collect_qc_summary.py all to /usr/bin/env python it runs and completes fine

[DarianHole]

It might be that Dan is using the fork I had for the NML (or it looks like it) which I've only run and tested with the bash script that was made before signal.py was available.

[dgaston]

Ah that could be it, yes I am using the fork from the NML for signal itself. Although I was running into the same problem previously with what I think was the straight signal, not the nml fork, and running ncov_tools with the bash script. I ended up creating this new clean environment to see if it would fix the original error

[rdeborja]

@dgaston did this end up working correctly? Unfortunately I don't much about signal but more than happy to investigate further if the bash script still isn't resolved.

[fmaguire]

Yeah, its an issue with the wrong python interpreter being used (and for some reason that pointing to a non-existent/or non exec permission intrepreter on your system). Two solutions:

• Add the env shebang to all the ncov-tool scripts/deps (as env will correctly use the activated conda env python interpreter)

• Alternatively, I just tend to run signal+ncov-tools (via signal.py or directly calling snakemake) from within the activated ncov-qc conda environment (created for ncov-tools as per the README). This means ncov-tools gets the python version and signal should work as expected as it activates separate rule-specific conda envs to execute rules.

[rdeborja]

All the scripts from ncov-parser and ncov-tools specify shebang as #!/usr/bin/env python. The scripts from ncov-parser seem to be overwritten upon installation via pip and point to the python interpreter from the conda env.

Sounds like running via signal.py is the way to go.

[fmaguire]

Good catch @rdeborja!

[dgaston]

Just wanted to clarify that I was running via signal.py. But yes, changing the shebang line in the 3 scripts to correct them back to #!/usr/bin/env python fixed things

[rdeborja]

Closing issue as a workaround has been put in place. Please re-open if there are any issues.