search

edgardomortiz / Captus

Assembly of Phylogenomic Datasets from High-Throughput Sequencing data
https://edgardomortiz.github.io/captus.docs/
GNU General Public License v3.0
21 stars 5 forks source link

Issues on Cleaning step of pipeline #18

Open TiagoVieiraBotany opened 3 weeks ago

TiagoVieiraBotany commented 3 weeks ago

Hi Edgardo and colleagues,

I'm facing an issue when trying to run the first - cleaning - step of the pipeline. Here's my slurm script:

!/bin/bash #

SBATCH -p sapphire # partition (queue)

SBATCH -c 12 # number of cores

SBATCH --mem 128000 # memory pool for all cores

SBATCH -t 0-18:00 # time (D-HH:MM)

SBATCH -o /n/holyscratch01/davis_lab/tlvieira/CAPTUS/captus.clean.out

SBATCH -e /n/holyscratch01/davis_lab/tlvieira/CAPTUS/captus.clean.err

SBATCH --mail-type=END

SBATCH --mail-user=tiagolvs@gmail.com

source /n/home10/tlvieira/miniconda3/etc/profile.d/conda.sh

conda activate captus

Clean threshold of 13

captus_assembly clean \ --reads /n/holyscratch01/davis_lab/tlvieira/RAW_DATA \ --out /n/holyscratch01/davis_lab/tlvieira/CAPTUS/01_clean_reads/ \ --adaptor_set Illumina \ --trimq 13 \ --maq 16 \ --qc_program fastqc \ --threads 12

It doesn't move further from the adaptor removal step of the protocol due to issues with Java for preparing reports, apparently. The Slurm .err file indicates:

Traceback (most recent call last): File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc return self._engine.get_loc(casted_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'after'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/n/home10/tlvieira/miniconda3/envs/captus/bin/captus_assembly", line 10, in sys.exit(main()) ^^^^^^ File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/captus/captus_assembly.py", line 1424, in main CaptusAssembly() File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/captus/captus_assembly.py", line 90, in init getattr(self, args.command)() File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/captus/captus_assembly.py", line 310, in clean clean(full_command, args) File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/captus/clean.py", line 387, in clean qc_html_report, qc_html_msg = build_qc_report(out_dir, qc_extras_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/captus/report.py", line 58, in build_qc_report read_len_pct = avg_read_len.loc[:, "after"] / avg_read_len.loc[:, "before"] * 100


  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexing.py", line 1184, in __getitem__
    return self._getitem_tuple(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexing.py", line 1368, in _getitem_tuple
    return self._getitem_lowerdim(tup)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexing.py", line 1041, in _getitem_lowerdim
    return self._getitem_nested_tuple(tup)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexing.py", line 1124, in _getitem_nested_tuple
    return cast(_LocIndexer, self)._handle_lowerdim_multi_index_axis0(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexing.py", line 1394, in _handle_lowerdim_multi_index_axis0
    raise ek
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexing.py", line 1388, in _handle_lowerdim_multi_index_axis0
    return self._get_label(tup, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexing.py", line 1381, in _get_label
    return self.obj.xs(label, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/generic.py", line 4290, in xs
    loc, new_index = index._get_loc_level(key, level=0)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexes/multi.py", line 3257, in _get_loc_level
    loc_level = self._get_level_indexer(k, level=i)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexes/multi.py", line 3391, in _get_level_indexer
    idx = self._get_loc_single_level_index(level_index, key)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexes/multi.py", line 2980, in _get_loc_single_level_index
    return level_index.get_loc(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/tlvieira/miniconda3/envs/captus/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: 'after'

--------------------------------------------------------------------------------------------------------------

The program also outputs some .log files (e.g. hs_err_pid282039.log), with the following header content:

 There is insufficient memory for the Java Runtime Environment to continue.
 Native memory allocation (mmap) failed to map 1070554152960 bytes for committing reserved memory.
 Possible reasons:
   The system is out of physical RAM or swap space
 Possible solutions:
   Reduce memory load on the system
   Increase physical memory or swap space
   Check if swap backing store is full
   Decrease Java heap size (-Xmx/-Xms)
   Decrease number of Java threads
   Decrease Java thread stack sizes (-Xss)
   Set larger code cache with -XX:ReservedCodeCacheSize=
 This output file may be truncated or incomplete.

  Out of Memory Error (os_linux.cpp:2643), pid=282039, tid=282040

 JRE version:  (20.0.2) (build )
 Java VM: OpenJDK 64-Bit Server VM (20.0.2-internal-adhoc..src, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, linux-amd64)
 Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /n/holyscratch01/davis_lab/tlvieira/CAPTUS/core.$

Any clues of what might be going on concerning the issue? The idea is to use the pipeline cleaning steps instead of FASTP

All the best,
Tiago
edgardomortiz commented 3 weeks ago

Hi @TiagoVieiraBotany ,

Reserving 128GB of RAM should be more than plenty, unfortunately the JAVA components of Captus are not able to access them. I found this depends on the configuration of the cluster for which you have to consult the administrators of your HPC. Sorry I can't be of more help...

Edgardo

TiagoVieiraBotany commented 3 weeks ago

Thanks for replying @edgardomortiz . I will stick with Fastp for this duty then. Anyway, thanks for putting together such a nice and well documented pipeline! All the best, Tiago

edgardomortiz commented 3 weeks ago

Hi @TiagoVieiraBotany ,

Fastp was my first option, I think I will include it as alternative in a following version. I didn't use it, however because it tends to detect real repeats as adaptor, check my experiment here: https://edgardomortiz.github.io/captus.docs/assembly/clean/

As I said, some cluster configuration don't allow for automatic detection of available RAM, which messes up with Java. Maybe, you could try installing Captus through Micromamba (I saw you were using the classic conda environment), I don't know if this will help though...

Best,

Edgardo

edgardomortiz commented 2 weeks ago

Hi @TiagoVieiraBotany ,

I ran into this error in our cluster too (not exactly the same one, but also related to memory), my node also has 128Gb of RAM, then I added --ram 100 to my Captus command and it finished successfully.

I hope this helps,

Edgardo