search

allind / EukDetect

MIT License
40 stars 15 forks source link

Error in `filter` mode #32

Closed shibataryohei closed 1 year ago

shibataryohei commented 1 year ago

Hi Abigail,

Thank you very much for your support in issue #30. I successfully finished analysis in a dataset, but newly encountered in another dataset during filter mode.

After aln mode has successfully finished in all files, run mode failed in some files (not all files) with the error messages attached below in the log file. Do you have any idea for a solution?

Best, Rio

[Fri Dec  2 17:11:24 2022]
Finished job 798.
832 of 3223 steps (26%) done

[Fri Dec  2 17:11:24 2022]
rule taxonomize:
    input: /MYDIRECTORYFORFILE/filtering/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_read_counts_and_mismatches.txt, /MYDIRECTORYFOREUKDETECT/EukDetect/eukdetect/count_primary_and_secondary_eukfrac.py, /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/busco_taxid_link.txt, /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/specific_and_inherited_markers_per_taxid.txt, /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/taxa.sqlite, /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/taxid_cumulativelength.txt
    output: /MYDIRECTORYFORFILE/filtering/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_all_hits_table.txt, /MYDIRECTORYFORFILE/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_filtered_hits_table.txt, /MYDIRECTORYFORFILE/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_filtered_hits_eukfrac.txt
    jobid: 291
    wildcards: output_dir=/MYDIRECTORYFORFILE, sample=XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003

OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 30 current, 30 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 30 current, 30 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 30 current, 30 max
Traceback (most recent call last):
  File "/MYDIRECTORYFOREUKDETECT/EukDetect/eukdetect/count_primary_and_secondary_eukfrac.py", line 4, in <module>
    from ete3 import NCBITaxa
  File "/MYDIRECTORYFOREUKDETECT/.conda/envs/eukdetect/lib/python3.6/site-packages/ete3/__init__.py", line 47, in <module>
    import numpy
  File "/MYDIRECTORYFOREUKDETECT/.conda/envs/eukdetect/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/MYDIRECTORYFOREUKDETECT/.conda/envs/eukdetect/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/MYDIRECTORYFOREUKDETECT/.conda/envs/eukdetect/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/MYDIRECTORYFOREUKDETECT/.conda/envs/eukdetect/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/MYDIRECTORYFOREUKDETECT/.conda/envs/eukdetect/lib/python3.6/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
SystemError: initialization of multiarray raised unreported exception
/bin/bash: line 1: 34134 Segmentation fault      python /MYDIRECTORYFOREUKDETECT/EukDetect/eukdetect/count_primary_and_secondary_eukfrac.py --dbfile /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/taxa.sqlite --taxid_link /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/busco_taxid_link.txt --inherited_markers /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/specific_and_inherited_markers_per_taxid.txt --taxid_genelens /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/taxid_cumulativelength.txt --readcounts /MYDIRECTORYFORFILE/filtering/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_read_counts_and_mismatches.txt --primarytab /MYDIRECTORYFORFILE/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_filtered_hits_table.txt --eukfrac /MYDIRECTORYFORFILE/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_filtered_hits_eukfrac.txt --alltab /MYDIRECTORYFORFILE/filtering/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_all_hits_table.txt
[Fri Dec  2 17:11:25 2022]
Error in rule taxonomize:
    jobid: 291
    output: /MYDIRECTORYFORFILE/filtering/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_all_hits_table.txt, /MYDIRECTORYFORFILE/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_filtered_hits_table.txt, /MYDIRECTORYFORFILE/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_filtered_hits_eukfrac.txt

RuleException:
CalledProcessError in line 185 of /MYDIRECTORYFOREUKDETECT/EukDetect/rules/eukdetect_eukfrac.rules:
Command ' set -euo pipefail;  python /MYDIRECTORYFOREUKDETECT/EukDetect/eukdetect/count_primary_and_secondary_eukfrac.py --dbfile /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/taxa.sqlite --taxid_link /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/busco_taxid_link.txt --inherited_markers /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/specific_and_inherited_markers_per_taxid.txt --taxid_genelens /MYDIRECTORYFOREUKDETECT/EukDetect/eukdb/taxid_cumulativelength.txt --readcounts /MYDIRECTORYFORFILE/filtering/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_read_counts_and_mismatches.txt --primarytab /MYDIRECTORYFORFILE/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_filtered_hits_table.txt --eukfrac /MYDIRECTORYFORFILE/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_filtered_hits_eukfrac.txt --alltab /MYDIRECTORYFORFILE/filtering/XCSNH_20200131_A00904_IL100150648_N2UD-F09_L003_all_hits_table.txt ' returned non-zero exit status 139.
  File "/MYDIRECTORYFOREUKDETECT/EukDetect/rules/eukdetect_eukfrac.rules", line 185, in __rule_taxonomize
  File "/MYDIRECTORYFOREUKDETECT/.conda/envs/eukdetect/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[Fri Dec  2 17:11:27 2022]
Finished job 352.
833 of 3223 steps (26%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
allind commented 1 year ago

Hi Rio,

I suspect this error is happening not directly because of eukdetect but because the number of requested threads is too high for the system you're running on. (This is suggested by the error OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable, OpenBLAS blas_thread_init: RLIMIT_NPROC 30 current, 30 max). Try re-running again requesting fewer cores.

shibataryohei commented 1 year ago

Hi Abigail,

Thank you very much for your advice. By using fewer cores, I've just confirmed my run was successfully finished. Sorry for the delay!

Best, Rio