Error in job metametamerge while creating output

Thomieh73 commented 6 years ago

Hi Vitor,

I have made custom database following your example using only fungi sequences. Next I classified my sequences with this database and this ran until almost the end. So databases were created for the different tools and the classification for each tool was run.

I then ran into a problem with the metametamerge.sm script.

This is the rule that was created for the step:


rule metametamerge:
    input: /work/users/thhaverk/MM_databases/fungi_database/kaiju_db_check.done, /work/users/thhaverk/MM_databases/fungi_
database/clark_db_check.done, /work/users/thhaverk/MM_databases/fungi_database/kraken_db_check.done, /work/users/thhaverk
/MM_databases/fungi_database/dudes_db_check.done, /work/users/thhaverk/MM_databases/names.dmp, K1T0/profiles/fungi_databa
se/kaiju.profile.out, K1T0/profiles/fungi_database/clark.profile.out, K1T0/profiles/fungi_database/kraken.profile.out, K1
T0/profiles/fungi_database/dudes.profile.out, /work/users/thhaverk/MM_databases/merged.dmp, /work/users/thhaverk/MM_datab
ases/nodes.dmp, K1T0/profiles/fungi_database/kaiju_clean_files.done, K1T0/profiles/fungi_database/clark_clean_files.done,
 K1T0/profiles/fungi_database/kraken_clean_files.done, K1T0/profiles/fungi_database/dudes_clean_files.done
    output: K1T0/metametamerge/fungi_database/final.metametamerge.profile.out
    log: K1T0/log/fungi_database/metametamerge.log
    benchmark: K1T0/log/fungi_database/metametamerge.time
    wildcards: sample=K1T0, database=fungi_database

This is the error that I get:

Error in job metametamerge while creating output file K1T0-B/metametamerge/fungi_database/final.metametamerge.profile.out
.
ClusterJobException in line 12 of /usit/abel/u1/thhaverk/miniconda3/envs/metameta/opt/metameta/scripts/metametamerge.sm:
Error executing rule metametamerge on cluster (jobid: 1, jobscript: /work/users/thhaverk/MMresults_test/.snakemake/tmp.ox
ye9o6w/snakejob.metametamerge.1.sh). For detailed error see the cluster log.
Job failed, going on with independent jobs.
Error in job metametamerge while creating output file K1T0/metametamerge/fungi_database/final.metametamerge.profile.out.
ClusterJobException in line 12 of /usit/abel/u1/thhaverk/miniconda3/envs/metameta/opt/metameta/scripts/metametamerge.sm:
Error executing rule metametamerge on cluster (jobid: 0, jobscript: /work/users/thhaverk/MMresults_test/.snakemake/tmp.ox
ye9o6w/snakejob.metametamerge.0.sh). For detailed error see the cluster log.
Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message

So next I look at the clusterlog for this job file: metametamerge.18496961.out

Error in job metametamerge while creating output file K1T0/metametamerge/fungi_database/final.metametamerge.profile.out.
RuleException:
CalledProcessError in line 31 of /usit/abel/u1/thhaverk/miniconda3/envs/metameta/opt/metameta/scripts/metametamerge.sm:
Command 'MetaMetaMerge.py --input-files K1T0/profiles/fungi_database/dudes.profile.out K1T0/profiles/fungi_database/clark.profile.out K1T0/profiles/fungi_database/kaiju.profile.out K1T0/profiles/fungi_database/kraken.profile.out --database-profiles /work/users/thhaverk/MM_databases/fungi_database/dudes.dbprofile.out /work/users/thhaverk/MM_databases/fungi_database/clark.dbprofile.out /work/users/thhaverk/MM_databases/fungi_database/kaiju.dbprofile.out /work/users/thhaverk/MM_databases/fungi_database/kraken.dbprofile.out --tool-identifier 'dudes,clark,kaiju,kraken' --tool-method 'p,b,b,b' --names-file /work/users/thhaverk/MM_databases/names.dmp --nodes-file /work/users/thhaverk/MM_databases/nodes.dmp --merged-file /work/users/thhaverk/MM_databases/merged.dmp --bins 4 --cutoff 0.0001 --mode 'linear' --ranks 'species' --output-file K1T0/metametamerge/fungi_database/final.metametamerge.profile.out   --output-parsed-profiles > K1T0/log/fungi_database/metametamerge.log 2>&1' returned non-zero exit status 1
  File "/usit/abel/u1/thhaverk/miniconda3/envs/metameta/opt/metameta/scripts/metametamerge.sm", line 31, in __rule_metametamerge
  File "/usit/abel/u1/thhaverk/miniconda3/envs/metameta/lib/python3.5/concurrent/futures/thread.py", line 55, in run
Exiting because a job execution failed. Look above for error message

So with this I am not sure what is happening here, so I also checked the metametamerge.log file:

- - - - - - - - - - - - - - - - - - - - -
           MetaMetaMerge 1.1
- - - - - - - - - - - - - - - - - - - - -
Input files: 
 dudes (p) K1T0/profiles/fungi_database/dudes.profile.out /work/users/thhaverk/MM_databases/fungi_database/dudes.dbprofile.out
 clark (b) K1T0/profiles/fungi_database/clark.profile.out /work/users/thhaverk/MM_databases/fungi_database/clark.dbprofile.out
 kaiju (b) K1T0/profiles/fungi_database/kaiju.profile.out /work/users/thhaverk/MM_databases/fungi_database/kaiju.dbprofile.out
 kraken (b) K1T0/profiles/fungi_database/kraken.profile.out /work/users/thhaverk/MM_databases/fungi_database/kraken.dbprofile.out
Taxonomy: 
 /work/users/thhaverk/MM_databases/names.dmp, /work/users/thhaverk/MM_databases/nodes.dmp, /work/users/thhaverk/MM_databases/merged.dmp
Bins: 4
Cutoff: 0.0001
Mode: linear
Ranks: species
Output file (type): K1T0/metametamerge/fungi_database/final.metametamerge.profile.out (bioboxes)
Verbose: False
Detailed: False

- - - - - - - - - - - - - - - - - - - - -

Parsing taxonomy (names, nodes, merged) ... 

Reading database profiles ...
 - /work/users/thhaverk/MM_databases/fungi_database/dudes.dbprofile.out (tsv)
        species - 0 entries (0 ignored)
        (WARNING) no valid entries found [species]
Traceback (most recent call last):
  File "/usit/abel/u1/thhaverk/miniconda3/envs/metameta/bin/MetaMetaMerge.py", line 338, in <module>
    main()
  File "/usit/abel/u1/thhaverk/miniconda3/envs/metameta/bin/MetaMetaMerge.py", line 113, in main
    db = Databases(database_file, parse_files(database_file, 'db', all_names_scientific, all_names_other, nodes, merged, ranks, args.verbose), ranks)           
  File "/usit/abel/u1/thhaverk/miniconda3/envs/metameta/lib/python3.5/site-packages/metametamerge/Databases.py", line 8, in __init__
    Profile.__init__(self, profile, ranks)
  File "/usit/abel/u1/thhaverk/miniconda3/envs/metameta/lib/python3.5/site-packages/metametamerge/Profile.py", line 15, in __init__
    self.profilerank[rankid] = ProfileRank(profile[np.ix_(profile[:,1]==rankid, [0,2,3])],rankid,sum(profile[:,1]==rankid))
IndexError: too many indices for array

The output mentions that no valied entries were found. But the dudes.profile.out file contains a tabular output of classifications. I changed the ranks in the yaml file to genus, but that gave the same error. Next I tried to not use the output from dudes to see if there was the problem, but then clark gave the same error. So I think it is not depending on the tool but on the parsing of the profile files.

So the error has to do with the last line from the metametamerge.log. Any idea on what is causing the error and how to solve it?

pirovc commented 6 years ago

Hi Stephan,

Apparently the profiles (.profile.out) were generated just fine as you checked. Looks like MetaMeta failed to generate the database profiles (`/work/users/thhaverk/MM_databases/fungi_database/.dbprofile.out`). Can you check if those files were created or have any information?

If possible, please send me the following log files: /work/users/thhaverk/MM_databases/fungi_database/log/*_db_custom_profile.log /work/users/thhaverk/MM_databases/fungi_database/log/*_database_profile.log

Best, Vitor

tadKeys commented 6 years ago

Dear Vitor,

I'm getting the same exception with my dataset as Thomieh73 (using the archaea_bacteria_201503 database).

dbprofile.out are present

~/work/01_meta_meta_db/archaea_bacteria_201503$ ls -l
total 318
drwxr-xr-x 3 stephan stephan     4 Jun 27  2017 clark_db
-rw-rw-r-- 1 stephan stephan     0 Feb 15 14:40 clark_db_check.done
-rwxr-xr-x 1 stephan stephan 57901 Feb 15 14:40 clark.dbprofile.out
drwxr-xr-x 2 stephan stephan     9 Jun 27  2017 dudes_db
-rw-rw-r-- 1 stephan stephan     0 Feb 14 09:40 dudes_db_check.done
-rwxr-xr-x 1 stephan stephan 60361 Feb 14 09:40 dudes.dbprofile.out
drwxr-xr-x 2 stephan stephan    18 Jun  8  2017 gottcha_db
-rw-rw-r-- 1 stephan stephan     0 Feb 14 09:01 gottcha_db_check.done
-rwxr-xr-x 1 stephan stephan 58584 Feb 14 09:01 gottcha.dbprofile.out
drwxr-xr-x 2 stephan stephan     5 Jun  8  2017 kaiju_db
-rw-rw-r-- 1 stephan stephan     0 Feb 14 09:01 kaiju_db_check.done
-rwxr-xr-x 1 stephan stephan 85115 Feb 14 09:01 kaiju.dbprofile.out
drwxr-xr-x 3 stephan stephan     5 Jun  8  2017 kraken_db
-rw-rw-r-- 1 stephan stephan     0 Feb 15 11:55 kraken_db_check.done
-rwxr-xr-x 1 stephan stephan 60361 Feb 15 11:54 kraken.dbprofile.out
drwxrwxr-x 2 stephan stephan    37 Feb 15 14:40 log
drwxr-xr-x 2 stephan stephan     2 Jun  8  2017 motus_db
-rw-rw-r-- 1 stephan stephan     0 Feb 13 15:22 motus_db_check.done
-rwxr-xr-x 1 stephan stephan 89199 Feb 13 15:22 motus.dbprofile.out

output/sample1/log/archaea_bacteria_201503/metametamerge.log
```
   MetaMetaMerge 1.1
```
Input files: kraken (p) sample1/profiles/archaea_bacteria_201503/kraken.profile.out /home/stephan/work/01_meta_meta_db/archaea_bacteria_201503/kraken.dbprofile.out clark (p) sample1/profiles/archaea_bacteria_201503/clark.profile.out /home/stephan/work/01_meta_meta_db/archaea_bacteria_201503/clark.dbprofile.out kaiju (p) sample1/profiles/archaea_bacteria_201503/kaiju.profile.out /home/stephan/work/01_meta_meta_db/archaea_bacteria_201503/kaiju.dbprofile.out Taxonomy: /home/stephan/work/01_meta_meta_db/taxonomy/names.dmp, /home/stephan/work/01_meta_meta_db/taxonomy/nodes.dmp, /home/stephan/work/01_meta_meta_db/taxonomy/merged.dmp Bins: 4 Cutoff: 0.0001 Mode: linear Ranks: species Output file (type): sample1/metametamerge/archaea_bacteria_201503/final.metametamerge.profile.out (bioboxes) Verbose: False Detailed: False

Parsing taxonomy (names, nodes, merged) ...

Reading database profiles ...

/home/stephan/work/01_meta_meta_db/archaea_bacteria_201503/kraken.dbprofile.out (tsv) species - 1529 entries (0 ignored) 2 taxons with merged entries [94694,1905730] Total - 1527 taxons
/home/stephan/work/01_meta_meta_db/archaea_bacteria_201503/clark.dbprofile.out (tsv) species - 1461 entries (0 ignored) 2 taxons with merged entries [94694,1905730] Total - 1459 taxons
/home/stephan/work/01_meta_meta_db/archaea_bacteria_201503/kaiju.dbprofile.out (tsv) species - 2419 entries (0 ignored) 4 taxons with merged entries [1405,94694,1334193,1905730] Total - 2415 taxons

Reading profiles ...

sample1/profiles/archaea_bacteria_201503/kraken.profile.out (tsv) species - 0 entries (0 ignored) (WARNING) no valid entries found [species] Traceback (most recent call last): File "/xxx/output/.snakemake/conda/266fca67/bin/MetaMetaMerge.py", line 338, in main() File "/xxx/output/.snakemake/conda/266fca67/bin/MetaMetaMerge.py", line 131, in main tool = Tools(input_file, identifiers[idx], methods[idx], parse_files(input_file, methods[idx], all_names_scientific, all_namesother, nodes, merged, ranks, args.verbose), ranks, args.verbose) File "/xxx/output/.snakemake/conda/266fca67/lib/python3.5/site-packages/metametamerge/Tools.py", line 11, in init Profile.init(self, profile, ranks) File "/xxx/output/.snakemake/conda/266fca67/lib/python3.5/site-packages/metametamerge/Profile.py", line 15, in init self.profilerank[rankid] = ProfileRank(profile[np.ix(profile[:,1]==rankid, [0,2,3])],rankid,sum(profile[:,1]==rankid)) IndexError: too many indices for array


Is this the reason for the error?

species - 0 entries (0 ignored) (WARNING) no valid entries found [species]



Thanks in advance,
Stephan

pirovc commented 6 years ago

Hi Stephan,

The previous error was related to the custom dabases profiles, which is due to a dependency on NCBI e-utils web-services. This step is failing for some people and a fix using local files is being developed.

The new error is different, since your database profiles look fine and were loaded correctly. What is actually happening is that there were no identifications for this dataset with kraken and metametamerge fails to load it. There's also a planned fix for that. What you can do for now is to disable kraken for this run. But once you have some identifications from kraken this should not happen.

Best, Vitor

Thomieh73 commented 5 years ago

Hi Vitor, Since I am no longer working on this issue, I think it is good to close it.

Best Thomas

pirovc / metameta

Error in job metametamerge while creating output #8