Closed dgaston closed 1 year ago
@dgaston what happens if you run primers_to_amplicons.py
on the command line outside signal
and ncov-tools
?
cd /path/to/qc/directory
primers_to_amplicons.py --primers /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/resources/primer_schemes/freed/nCoV-2019.bed --offset 0 --bed_type unique_amplicons --output bed/amplicon.bed --primer_prefix nCoV-2019
Also, even though you manually set the permissions to the entire conda environment, what are the permissions for the script?
ls -la /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.snakemake/conda/47aef19e0702f2843c364d0c2fe3ede9/bin/primers_to_amplicons.py
I just checked our and it should be:
-rwxrwxr-x
For the permissions it looks like it inherited the 777 as expected:
-rwxrwxrwx
If I copy the primers_to_amplicons.py script into another directory and run with my base conda environment, or the base signal conda environment:
-bash: ./primers_to_amplicons.py: /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.: bad interpreter: Permission denied
The interpreter on the shebang line of the primers_to_amplicons.py script is:
/mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.snakemake/conda/47aef19e0702f2843c364d0c2fe3ede9/bin/python
and if I run that on its own it loads up the python interpreter just fine with no errors:
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) [GCC 10.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.
If I change that to a more standard #!/usr/bin/env python line, the script will then execute (I get an input error of course as the ncov module isn't in the same directory)
This looks like a conda environment issue, the script is looking up the python interpreter using /usr/bin/env python
, like the rest of the scripts in ncov-tools. I don't know why this is resolving to /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.
instead of the python path. @fmaguire?
Yeah, if I set the shebang line of primers_to_amplicons.py, get_qc.py and collect_qc_summary.py all to /usr/bin/env python it runs and completes fine
It might be that Dan is using the fork I had for the NML (or it looks like it) which I've only run and tested with the bash script that was made before signal.py
was available.
Ah that could be it, yes I am using the fork from the NML for signal itself. Although I was running into the same problem previously with what I think was the straight signal, not the nml fork, and running ncov_tools with the bash script. I ended up creating this new clean environment to see if it would fix the original error
@dgaston did this end up working correctly? Unfortunately I don't much about signal
but more than happy to investigate further if the bash
script still isn't resolved.
Yeah, its an issue with the wrong python interpreter being used (and for some reason that pointing to a non-existent/or non exec permission intrepreter on your system). Two solutions:
Add the env shebang to all the ncov-tool scripts/deps (as env
will correctly use the activated conda env python interpreter)
Alternatively, I just tend to run signal+ncov-tools (via signal.py
or directly calling snakemake) from within the activated ncov-qc
conda environment (created for ncov-tools as per the README). This means ncov-tools gets the python version and signal should work as expected as it activates separate rule-specific conda envs to execute rules.
All the scripts from ncov-parser
and ncov-tools
specify shebang as #!/usr/bin/env python
. The scripts from ncov-parser
seem to be overwritten upon installation via pip
and point to the python interpreter from the conda env.
Sounds like running via signal.py
is the way to go.
Good catch @rdeborja!
Just wanted to clarify that I was running via signal.py. But yes, changing the shebang line in the 3 scripts to correct them back to #!/usr/bin/env python fixed things
Closing issue as a workaround has been put in place. Please re-open if there are any issues.
Hello,
I keep getting this error now on multiple machines and with fresh installs/set-ups of signal. Running the command:
python signal.py --configfile config.yaml --cores 24 ncov_tools
I get this error on the following step: make_amplicon_steps with the error message:
/bin/bash: /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.snakemake/conda/47aef19e0702f2843c364d0c2fe3ede9/bin/primers_to_amplicons.py: /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/.: bad interpreter: Permission denied [Mon May 30 10:25:22 2022] Error in rule make_amplicon_bed: jobid: 3 output: bed/amplicon.bed shell: primers_to_amplicons.py --primers /mnt/shared-data/clin/public_health/sars-cov-2/analysis/covid-19-signal-nml/resources/primer_schemes/freed/nCoV-2019.bed --offset 0 --bed_type unique_amplicons --output bed/amplicon.bed --primer_prefix nCoV-2019 (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
To test I ran chmod -R 777 on the .snakemake directory so that all conda environments and files would be given maximal permissions and it doesn't make any difference.