search

DyogenIBENS / SCORPIOS

SCORPiOs is a synteny-guided gene tree correction pipeline for clades that have undergone a whole-genome duplication event.
GNU General Public License v3.0
19 stars 3 forks source link

error report when run snakemake #13

Closed sadiexiaoyu closed 1 month ago

sadiexiaoyu commented 1 month ago

Hi,

When I ran the sample file using snakemake --configfile config_example.yaml --use-conda --cores 4 --scheduler=greedy I got error report like this:

Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 4 Rules claiming more threads will be scaled down. Traceback (most recent call last): File "/miniconda3/envs/scorpios_new/lib/python3.7/site-packages/snakemake/init.py", line 783, in snakemake keepmetadata=keep_metadata, File "/miniconda3/envs/scorpios_new/lib/python3.7/site-packages/snakemake/workflow.py", line 1039, in execute logger.run_info("\n".join(dag.stats())) File "/miniconda3/envs/scorpios_new/lib/python3.7/site-packages/snakemake/dag.py", line 2159, in stats yield tabulate(rows, headers="keys") File "/miniconda3/envs/scorpios_new/lib/python3.7/site-packages/tabulate/init.py", line 2049, in tabulate tabular_data, headers, showindex=showindex File "/miniconda3/envs/scorpios_new/lib/python3.7/site-packages/tabulate/init.py", line 1471, in _normalize_tabular_data rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows)) File "/miniconda3/envs/scorpios_new/lib/python3.7/site-packages/tabulate/init.py", line 1471, in rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows)) File "/miniconda3/envs/scorpios_new/lib/python3.7/site-packages/tabulate/init.py", line 107, in _is_separating_line (len(row) >= 1 and row[0] == SEPARATING_LINE) File "/miniconda3/envs/scorpios_new/lib/python3.7/site-packages/snakemake/rules.py", line 1127, in eq return self.name == other.name and self.output == other.output AttributeError: 'str' object has no attribute 'name'

Could you give any suggestion?

eparey commented 1 month ago

Hello,

This seems to be related to a known snakemake issue (https://github.com/snakemake/snakemake/issues/1899): versions >0.8 of the package tabulate are incompatible with this version of snakemake.

I updated the conda env file to (hopefully) resolve the issue. Could you please try to re-install the scorpios conda env with the updated envs/scorpios.yaml file?

Best, Elise

eparey commented 1 month ago

Hi, is the issue resolved?

Let me know if you need more info on how to reinstall the environment.

sadiexiaoyu commented 1 month ago

Hi, Thanks for the reply! Yes, I also found the issue which said that it might be a problem with the version of tabulate. So I am currently reinstalling it, and it's still in the installation process, which is a bit slow. I hope it goes smoothly and I will let you known once it is done.

sadiexiaoyu commented 1 month ago

Hi, I think I have installed snakemake successfully. However, when I run the example files snakemake --configfile config_example.yaml --use-conda --cores 4 --scheduler=greedy I got the following error report, could you give me any suggestion? Thanks!

Activating conda environment: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a /usr/bin/bash: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: /SCORPIOS/: bad interpreter: No such file or directory /usr/bin/bash: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: /SCORPIOS/: bad interpreter: No such file or directory sed: can't read SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh: No such file or directory sed: can't read SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh: No such file or directory rm: cannot remove 'SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh.bak': No such file or directory rm: cannot remove 'SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh.bak': No such file or directory /usr/bin/bash: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: /SCORPIOS/: bad interpreter: No such file or directory [Tue Oct 1 20:01:47 2024] sed: can't read SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh: No such file or directory Error in rule build_tree_polytomysolver: jobid: 90 output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh conda-env: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a shell:

    if [ -s SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697250_Lepisosteus.oculatus.phy ]; then
        profileNJ -s SCORPiOs_example/sptree_no_anc_0.nwk -g SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102697250_Lepisosteus.oculatus.nh --sep '_' -d SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697250_Lepisosteus.oculatus.phy -o SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh                      --slimit 1 --spos postfix  >&2;
    else touch SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh;
    fi;
    sed -i.bak 1d SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh; rm SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh.bak;

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Tue Oct 1 20:01:47 2024]

Error in rule build_tree_polytomysolver: rm: cannot remove 'SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh.bak': No such file or directory jobid: 96 output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh conda-env: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a shell:

    if [ -s SCORPiOs_example/fastdist_mat_Clupeocephala_0_102685883_Lepisosteus.oculatus.phy ]; then
        profileNJ -s SCORPiOs_example/sptree_no_anc_0.nwk -g SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102685883_Lepisosteus.oculatus.nh --sep '_' -d SCORPiOs_example/fastdist_mat_Clupeocephala_0_102685883_Lepisosteus.oculatus.phy -o SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh                      --slimit 1 --spos postfix  >&2;
    else touch SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh;
    fi;
    sed -i.bak 1d SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh; rm SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh.bak;

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Tue Oct 1 20:01:47 2024] Error in rule build_tree_polytomysolver: jobid: 99 output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh conda-env: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a shell:

    if [ -s SCORPiOs_example/fastdist_mat_Clupeocephala_0_102695880_Lepisosteus.oculatus.phy ]; then
        profileNJ -s SCORPiOs_example/sptree_no_anc_0.nwk -g SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102695880_Lepisosteus.oculatus.nh --sep '_' -d SCORPiOs_example/fastdist_mat_Clupeocephala_0_102695880_Lepisosteus.oculatus.phy -o SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh                      --slimit 1 --spos postfix  >&2;
    else touch SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh;
    fi;
    sed -i.bak 1d SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh; rm SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh.bak;

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

/usr/bin/bash: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: /SCORPIOS/: bad interpreter: No such file or directory sed: can't read SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh: No such file or directory rm: cannot remove 'SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh.bak': No such file or directory [Tue Oct 1 20:01:47 2024] Error in rule build_tree_polytomysolver: jobid: 87 output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh conda-env: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a shell:

    if [ -s SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697450_Lepisosteus.oculatus.phy ]; then
        profileNJ -s SCORPiOs_example/sptree_no_anc_0.nwk -g SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102697450_Lepisosteus.oculatus.nh --sep '_' -d SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697450_Lepisosteus.oculatus.phy -o SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh                      --slimit 1 --spos postfix  >&2;
    else touch SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh;
    fi;
    sed -i.bak 1d SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh; rm SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh.bak;

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /SCORPIOS/.snakemake/log/2024-10-01T200143.328649.snakemake.log

eparey commented 1 month ago

Hello,

I just quickly tested installing scorpios from scratch and was unable to reproduce the issue.

It seems that one rule-specific environment (envs/polytomysolver.yaml) didn't install properly.

sadiexiaoyu commented 1 month ago

Hi, I tried the scripts that you have suggested.

"You can force scorpios to reinstall it by deleting your broken polytomysolver env (rm -r SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a) and re-running scorpios (snakemake --configfile config_example.yaml --use-conda --cores 4 --scheduler=greedy). You should then see a message showing the env is getting created ("Creating conda environment envs/polytomysolver.yaml...")."

Yes, after deleting the broken polytomysolver env, and re-running scorpios, I got the following messages (seems the same as previous error report). And everyhing is ok with script: mamba env create -f envs/polytomysolver.yaml. So could there be anything wrong with profileNJ?

Updating job get_scores_ortho. Updating job add_families_with_synteny_evidence. Creating conda environment envs/polytomysolver.yaml... Downloading and installing remote packages. Environment for envs/polytomysolver.yaml created (location: .snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a) Using shell: /usr/bin/bash Provided cores: 4 Rules claiming more threads will be scaled down. Job stats: job count min threads max threads

Target 1 1 1 build_test_tree_treebest 8 1 1 build_tree_polytomysolver 8 1 1 clean_up 1 1 1 correct_input_trees 1 4 4 cut_orthology_graphs 3 4 4 gene_trees_to_correct 1 1 1 list_treeb_inputs_for_parsing 2 1 1 pass_clean_interm 1 1 1 successfully_corrected_trees 2 1 1 test_polyS 8 1 1 total 36 1 4

[Wed Oct 2 14:49:09 2024] rule build_tree_polytomysolver: input: SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102685883_Lepisosteus.oculatus.nh, SCORPiOs_example/fastdist_mat_Clupeocephala_0_102685883_Lepisosteus.oculatus.phy, SCORPiOs_example/sptree_no_anc_0.nwk, SCORPiOs_example/Trees/subalis_0/Clupeocephala/102685883_Lepisosteus.oculatus.fa output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh jobid: 96 wildcards: wgd=Clupeocephala, ctrees=102685883_Lepisosteus.oculatus resources: tmpdir=/tmp

Activating conda environment: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a

[Wed Oct 2 14:49:09 2024] rule build_tree_polytomysolver: input: SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102697250_Lepisosteus.oculatus.nh, SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697250_Lepisosteus.oculatus.phy, SCORPiOs_example/sptree_no_anc_0.nwk, SCORPiOs_example/Trees/subalis_0/Clupeocephala/102697250_Lepisosteus.oculatus.fa output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh jobid: 90 wildcards: wgd=Clupeocephala, ctrees=102697250_Lepisosteus.oculatus resources: tmpdir=/tmp

Activating conda environment: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a

[Wed Oct 2 14:49:09 2024] rule build_tree_polytomysolver: input: SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102695880_Lepisosteus.oculatus.nh, SCORPiOs_example/fastdist_mat_Clupeocephala_0_102695880_Lepisosteus.oculatus.phy, SCORPiOs_example/sptree_no_anc_0.nwk, SCORPiOs_example/Trees/subalis_0/Clupeocephala/102695880_Lepisosteus.oculatus.fa output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh jobid: 99 wildcards: wgd=Clupeocephala, ctrees=102695880_Lepisosteus.oculatus resources: tmpdir=/tmp

Activating conda environment: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a

[Wed Oct 2 14:49:09 2024] rule build_tree_polytomysolver: input: SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102697450_Lepisosteus.oculatus.nh, SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697450_Lepisosteus.oculatus.phy, SCORPiOs_example/sptree_no_anc_0.nwk, SCORPiOs_example/Trees/subalis_0/Clupeocephala/102697450_Lepisosteus.oculatus.fa output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh jobid: 87 wildcards: wgd=Clupeocephala, ctrees=102697450_Lepisosteus.oculatus resources: tmpdir=/tmp

Activating conda environment: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a /usr/bin/bash: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: /SCORPIOS/: bad interpreter: No such file or directory /usr/bin/bash: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: /SCORPIOS/: bad interpreter: No such file or directory sed: can't read SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh: No such file or directory sed: can't read SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh: No such file or directory /usr/bin/bash: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: /SCORPIOS/: bad interpreter: No such file or directory rm: cannot remove 'SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh.bak': No such file or directory rm: cannot remove 'SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh.bak': No such file or directory sed: can't read SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh: No such file or directory rm: cannot remove 'SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh.bak': No such file or directory /usr/bin/bash: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: SCORPIOS/: bad interpreter: No such file or directory sed: can't read SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh: No such file or directory rm: cannot remove 'SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh.bak': No such file or directory [Wed Oct 2 14:49:09 2024] [Wed Oct 2 14:49:09 2024] Error in rule build_tree_polytomysolver: [Wed Oct 2 14:49:09 2024] Error in rule build_tree_polytomysolver: jobid: 99 [Wed Oct 2 14:49:09 2024] Error in rule build_tree_polytomysolver: jobid: 90 output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh Error in rule build_tree_polytomysolver: jobid: 87 output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh conda-env: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a jobid: 96 output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh conda-env: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a shell:

    if [ -s SCORPiOs_example/fastdist_mat_Clupeocephala_0_102695880_Lepisosteus.oculatus.phy ]; then
        profileNJ -s SCORPiOs_example/sptree_no_anc_0.nwk -g SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102695880_Lepisosteus.oculatus.nh --sep '_' -d SCORPiOs_example/fastdist_mat_Clupeocephala_0_102695880_Lepisosteus.oculatus.phy -o SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh                      --slimit 1 --spos postfix  >&2;
    else touch SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh;
    fi;
    sed -i.bak 1d SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh; rm SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102695880_Lepisosteus.oculatus.nh.bak;

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
output: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh
conda-env: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a
shell:

    if [ -s SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697450_Lepisosteus.oculatus.phy ]; then
        profileNJ -s SCORPiOs_example/sptree_no_anc_0.nwk -g SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102697450_Lepisosteus.oculatus.nh --sep '_' -d SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697450_Lepisosteus.oculatus.phy -o SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh                      --slimit 1 --spos postfix  >&2;
    else touch SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh;
    fi;
    sed -i.bak 1d SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh; rm SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697450_Lepisosteus.oculatus.nh.bak;

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

conda-env: /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a
shell:

    if [ -s SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697250_Lepisosteus.oculatus.phy ]; then
        profileNJ -s SCORPiOs_example/sptree_no_anc_0.nwk -g SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102697250_Lepisosteus.oculatus.nh --sep '_' -d SCORPiOs_example/fastdist_mat_Clupeocephala_0_102697250_Lepisosteus.oculatus.phy -o SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh                      --slimit 1 --spos postfix  >&2;
    else touch SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh;
    fi;
    sed -i.bak 1d SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh; rm SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102697250_Lepisosteus.oculatus.nh.bak;

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

shell:

    if [ -s SCORPiOs_example/fastdist_mat_Clupeocephala_0_102685883_Lepisosteus.oculatus.phy ]; then
        profileNJ -s SCORPiOs_example/sptree_no_anc_0.nwk -g SCORPiOs_example/Trees/ctrees_0/Clupeocephala/C_102685883_Lepisosteus.oculatus.nh --sep '_' -d SCORPiOs_example/fastdist_mat_Clupeocephala_0_102685883_Lepisosteus.oculatus.phy -o SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh                      --slimit 1 --spos postfix  >&2;
    else touch SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh;
    fi;
    sed -i.bak 1d SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh; rm SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102685883_Lepisosteus.oculatus.nh.bak;

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /SCORPIOS/.snakemake/log/2024-10-02T144403.147774.snakemake.log

eparey commented 1 month ago

What do these commands output (from within the SCORPIOS folder):

conda activate polytomysolver
which profileNJ
profileNJ --help
conda deactivate

and:

conda activate .snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a
which profileNJ
profileNJ --help
conda deactivate

and same with the absolute path (is SCORPIOS really cloned in the root folder or have you truncated it to not show personal info?)

conda activate /SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a
which profileNJ
profileNJ --help
conda deactivate
sadiexiaoyu commented 1 month ago

Hi,

the output of the first command is ok, but not for the 2nd and 3rd commands: (SCORPIOS was not cloned in the root folder, I just truncated it to not show personal info)

1st command:

(scorpios_new) [my ID]$ conda activate polytomysolver (polytomysolver) [my ID]$ which profileNJ ~/miniconda3/envs/polytomysolver/bin/profileNJ

(polytomysolver) [my ID] $ profileNJ --help usage: profileNJ [-h] -s SPECIETREE [-S SMAP] -g GENETREE [-d DISTFILE] [-o OUTFILE] [-gl GLINE] [-r {none,all,best}] [-n] [--sep GENE_SEP] [--spos {prefix,postfix}] [--nflagval MVAL] [-c {nj,upgma,rand}] [--slimit SOL_LIMIT] [--plimit PATH_LIMIT] [-v] [--try_hard] [--batch] [--seuil SEUIL] [--cap] [--parallelize] [--firstbest] [--cost COSTDL COSTDL | --scost SDLCOST] [--internalcost {mean,default}]

Polytomy solver with multiple solutions.

optional arguments: -h, --help show this help message and exit -s SPECIETREE, --sFile SPECIETREE Name of the file containing the species newick tree. (default: None) -S SMAP, --sMap SMAP Gene to species map. Use the standard format. (default: None) -g GENETREE, --gFile GENETREE Name of the file containing the gene newick tree. (default: None) -d DISTFILE, --dist DISTFILE Name of the file containing the distances between each pair of genes (The gene set should be the same for the leaf set of the genetree). (default: None) -o OUTFILE, --output OUTFILE Name of your output files with the corrected tree. When batch is specified, each corrected genetree will be printed in the appropriate output file. The genetree is printed on stdout if omitted. (default: None) -gl GLINE, --gLine GLINE Index of the line in the gene tree file that corresponds to the current gene tree, starting by 1. When the flag batch is used, each line of gFile will be used as genetree (default: 1) -r {none,all,best}, --reroot {none,all,best} Enable/Disable root mode. none: disable reroot mode, correct the input polytomies and return the result. all: enable reroot mode, reroot the genetree at each node and return all polytomy corrected version for each rooted tree. best: enable reroot mode, rerrot the genetree at each node and return all polytomy corrected version for the rooted tree with the smallest Dup-Lost score (First one if not unique).

                     (default: none)

-n, -nf, --nnflag Treat negative distances as large distances. (default: False) --sep GENE_SEP Gene-Specie separator for each leaf name in the genetree. PolytomySolver will guess by default in a very limited list of special character. ';;' is not a valid separator for the newick format! IF YOUR SEPARATOR IS ";;", DON'T USE THIS FLAG. THE PROGRAM WILL AUTOMATICALLY GUESS. (default: None) --spos {prefix,postfix} The position of the specie name according to the separator. Supported option are prefix and postfix (default: prefix) --nflagval MVAL Set largest value in the distance matrix. Entries on the main diagonal and negative values will be replaced by mValue. (default: 1e+305) -c {nj,upgma,rand}, --cluster {nj,upgma,rand} Set the clustering methods. upgma: UPGMA (Unweighted Pair Group Method with Arithmetic Mean) clustering algo. nj: neighbor joining clustering method, (slower). rand: A random clustering method (should be faster than upgma).

                     (default: nj)

--slimit SOL_LIMIT Set the max number of solution per genetree. Possible values are -1 to return all the solution or n, n>0 for a specific number of solution. Setting this argument to -1 is computationally expensive. (default: 30) --plimit PATH_LIMIT Set the max number of solution for each polytomy in the genetree. Possible values are -1 to explore all the solution or n, n>0 for a specific number of solution. Setting this argument to -1 is also computationally expensive. (default: -1) -v Output verbosity (default: False) --try_hard Try correcting errors due to common mistake before raising an exception (default: False) --batch Use this flag to enable batch mode. In batch mode, gLine value is discarded, --dist should be a file whose line link to the distance matrix of the genetree at the same line number in your genetree file (default: False) --seuil SEUIL Branch contraction threshold, when the tree is binary. Use only when the tree is binary. (default: None) --cap Capitalize the species name of the genetree leaves to match each species. Almost all functions are case sensitive. (default: False) --parallelize Use parallisation (default: False) --firstbest Only output solution for the first root with the best dl score. (default: False) --cost COSTDL COSTDL D L : 2 float values, duplication and loss cost in this order (default: None) --scost SDLCOST A file which contain the DL cost for each species. If a specie is not found in this list, its DL cost will be set to default value (1) (default: None) --internalcost {mean,default} internal Node duplication and loss cost. Set this value to mean in order to take the mean duplication and loss from the leaf; or set it to default (1) or value obtained from cost. (default: default) conda deactivate #################################################################################

2nd command: (scorpios_new) [my ID]$ conda activate .snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a (my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a) [my ID]$ which profileNJ ~/my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ

(my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a) [my ID]$ profileNJ --help -bash: my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: my path/SCORPIOS/: bad interpreter: No such file or directory

(my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a) [my ID]$ conda deactivate #################################################################################

3rd command: (scorpios_new) [my ID]$ conda activate my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756^C8cbd9f80ad6f0a Not a conda environment: my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756^C8cbd9f80ad6f0a

eparey commented 1 month ago

Thanks!

There is a typo in your third command (there is a '^' that shouldn't be here). Can you please run it again?

From what I can see from command 2, the environment was correctly created and profileNJ was installed by snakemake in the specific conda env, as expected, but there is an issue when profileNJ is being invoked.

Does it work if you use the relative path, that is, if you directly try from within the SCORPIOS folder (using the scorpios_new env): .snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ --help? Is there any space or special character in your path "my path"?

If you tryhead my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ, what do you get?

sadiexiaoyu commented 1 month ago

Hi, thanks for the quick reply.

Here are the outputs: There is a typo in your third command (there is a '^' that shouldn't be here). Can you please run it again? (scorpios_new) [my ID]$ conda activate my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a (my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a) [my ID]$ which profileNJ ~my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ (my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a) [my ID]$ profileNJ --help -bash: my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: my path/SCORPIOS/: bad interpreter: No such file or directory

From what I can see from command 2, the environment was correctly created and profileNJ was installed by snakemake in the specific conda env, as expected, but there is an issue when profileNJ is being invoked.

Does it work if you use the relative path, that is, if you directly try from within the SCORPIOS folder (using the scorpios_new env): .snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ --help? Is there any space or special character in your path "my path"? (scorpios_new) [my ID]$ .snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ --help -bash: .snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: my path/SCORPIOS/: bad interpreter: No such file or directory

(scorpios_new) [my ID]$ my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ --help -bash: my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ: my path/SCORPIOS/: bad interpreter: No such file or directory

there is no special character in my path except underscore.

If you try head my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ, what do you get? (scorpios_new) [my ID]$ head my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ

!my path/SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/python

import argparse import linecache from profileNJ.PolytomySolver import * from profileNJ.TreeLib import TreeUtils, TreeClass, params import sys import time from operator import itemgetter from math import log10 from multiprocessing.pool import Pool

eparey commented 1 month ago

Thank you! Sorry I am a bit slow to figure this out, but I'm sure we'll get there eventually!

I suspect there might me something wrong with python in this conda env (or with its path in the sheebang of profileNJ). What do you have with:

conda activate SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a
which python
python SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ --help

or (scorpios_new) ~/miniconda3/envs/polytomysolver/bin/python SCORPIOS/.snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ --help

eparey commented 1 month ago

Hello,

I think I finally know what's going wrong.

How long is your path my path? I suspect that your path is so long that the shebang auto-generated by conda exceeds the OS hard limit (see for instance https://github.com/conda/conda/issues/977).

To fix the issue, what I suggest is that you either (i) manually edit .snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ to replace the shebang (first line) by #!/usr/bin/env python or (ii) install SCORPIOS in another location with shorter path.

sadiexiaoyu commented 1 month ago

Hi, eparey,

Thank you very much for the reply! I manually edit .snakemake/conda/a0c747b3e48a6756aa8cbd9f80ad6f0a/bin/profileNJ to replace the shebang (first line) by #!/usr/bin/env python, and it works!

Now I can run snakemake --configfile config_example.yaml --use-conda --cores 4 --scheduler=greedy successfully, and got the output SCORPiOs_example/SCORPiOs_output_0.nhx.

However, there are still some warnings during the execution of the program as follows: First of all, there seems something wrong with the RAxML path. [Sat Oct 5 21:03:48 2024] rule test_polyS: input: SCORPiOs_example/Corrections/PolyS_0/Clupeocephala/102686286_Lepisosteus.oculatus.nh, SCORPiOs_example/Trees/subalis_0/Clupeocephala/102686286_Lepisosteus.oculatus.fa, SCORPiOs_example/Trees/subtrees_0/Clupeocephala/102686286_Lepisosteus.oculatus.nh output: SCORPiOs_example/Corrections/Res_polylk_0/Clupeocephala/Res_102686286_Lepisosteus.oculatus.txt, SCORPiOs_example/Trees/subalis_0/Clupeocephala/102686286_Lepisosteus.oculatus_a.lk jobid: 92 wildcards: wgd=Clupeocephala, ctrees=102686286_Lepisosteus.oculatus resources: tmpdir=/tmp

Warning, you specified a working directory via "-w" Keep in mind that RAxML only accepts absolute path names, not relative ones! Warning, you specified a working directory via "-w" Keep in mind that RAxML only accepts absolute path names, not relative ones! Warning, you specified a working directory via "-w" Warning, you specified a working directory via "-w" Keep in mind that RAxML only accepts absolute path names, not relative ones! Keep in mind that RAxML only accepts absolute path names, not relative ones!

RAxML can't, parse the alignment file as phylip file it will now try to parse it as FASTA file RAxML can't, parse the alignment file as phylip file it will now try to parse it as FASTA file RAxML can't, parse the alignment file as phylip file it will now try to parse it as FASTA file RAxML can't, parse the alignment file as phylip file it will now try to parse it as FASTA file

Using BFGS method to optimize GTR rate parameters, to disable this specify "--no-bfgs" Warning, you specified a working directory via "-w" Keep in mind that RAxML only accepts absolute path names, not relative ones! RAxML can't, parse the alignment file as phylip file it will now try to parse it as FASTA file

Using BFGS method to optimize GTR rate parameters, to disable this specify "--no-bfgs" Using BFGS method to optimize GTR rate parameters, to disable this specify "--no-bfgs" This is RAxML version 8.2.12 released by Alexandros Stamatakis on May 2018. This is RAxML version 8.2.12 released by Alexandros Stamatakis on May 2018. This is RAxML version 8.2.12 released by Alexandros Stamatakis on May 2018.

With greatly appreciated code contributions by: With greatly appreciated code contributions by: Andre Aberer (HITS) Simon Berger (HITS) Alexey Kozlov (HITS) Kassian Kobert (HITS) David Dao (KIT and HITS) Sarah Lutteropp (KIT and HITS) Nick Pattengale (Sandia) Wayne Pfeiffer (SDSC) Akifumi S. Tanabe (NRIFS) Charlie Taylor (UF)

Secondly, warnings about the file removing. [Sat Oct 5 21:05:20 2024] rule clean_up: input: SCORPiOs_example/.int_cleanup_0 output: SCORPiOs_example/.cleanup_0 jobid: 1 resources: tmpdir=/tmp

rm: cannot remove 'SCORPiOs_example/Pairwise_SyntenyOrthoPred_0': No such file or directory Touching output file SCORPiOs_example/.cleanup_0. [Sat Oct 5 21:05:20 2024] Finished job 1.

Looking forward to your further suggestion!

eparey commented 1 month ago

Great!

These are inconsequential warnings, not errors.(SCORPiOs does use absolute path for RAxML and the remove warning is just an extra security to catch files which may not have been previously removed).

Closing this as there are no more errors.