search

medema-group / BiG-SCAPE

Similarity networks of biosynthetic gene clusters
GNU Affero General Public License v3.0
61 stars 26 forks source link

OSError: [Errno 12] Cannot allocate memory #175

Open PuziJiang opened 2 weeks ago

PuziJiang commented 2 weeks ago

Dear Sir, I used 64 CPUs and provided 2TB of memory to run BiG-SCAPE-1.1.5, for clustering 140,000 BGCs. However, I encountered the following error: OSError: [Errno 12] Cannot allocate memory. The complete output file is shown below: WARNING! This alignment consists of closely-related and very-long sequences. WARNING! FastTree (or other standard maximum-likelihood tools) may not be appropriate for aligments of very closely-related sequences like this one, as FastTree does not account for recombination or gene conversion

WARNING! This alignment consists of closely-related and very-long sequences. WARNING! FastTree (or other standard maximum-likelihood tools) may not be appropriate for aligments of very closely-related sequences like this one, as FastTree does not account for recombination or gene conversion

WARNING! This alignment consists of closely-related and very-long sequences. WARNING! FastTree (or other standard maximum-likelihood tools) may not be appropriate for aligments of very closely-related sequences like this one, as FastTree does not account for recombination or gene conversion

WARNING! This alignment consists of closely-related and very-long sequences. WARNING! FastTree (or other standard maximum-likelihood tools) may not be appropriate for aligments of very closely-related sequences like this one, as FastTree does not account for recombination or gene conversion

WARNING! This alignment consists of closely-related and very-long sequences. WARNING! FastTree (or other standard maximum-likelihood tools) may not be appropriate for aligments of very closely-related sequences like this one, as FastTree does not account for recombination or gene conversion

WARNING! This alignment consists of closely-related and very-long sequences. WARNING! FastTree (or other standard maximum-likelihood tools) may not be appropriate for aligments of very closely-related sequences like this one, as FastTree does not account for recombination or gene conversion

WARNING! This alignment consists of closely-related and very-long sequences. WARNING! FastTree (or other standard maximum-likelihood tools) may not be appropriate for aligments of very closely-related sequences like this one, as FastTree does not account for recombination or gene conversion

WARNING! This alignment consists of closely-related and very-long sequences. WARNING! FastTree (or other standard maximum-likelihood tools) may not be appropriate for aligments of very closely-related sequences like this one, as FastTree does not account for recombination or gene conversion

Exception in thread Thread-16: Traceback (most recent call last): File "/share/appspace_data/sharedgroups/conda/env/bigscape/lib/python3.6/threading.py", line 916, in bootstrap_inner self.run() File "/share/appspace_data/shared_groups/conda/env/bigscape/lib/python3.6/threading.py", line 864, in r un self._target(*self._args, **self._kwargs) File "/share/appspace_data/shared_groups/conda/env/bigscape/lib/python3.6/multiprocessing/pool.py", lin e 405, in _handle_workers pool._maintain_pool() File "/share/appspace_data/shared_groups/conda/env/bigscape/lib/python3.6/multiprocessing/pool.py", lin e 246, in _maintain_pool self._repopulate_pool() File "/share/appspace_data/shared_groups/conda/env/bigscape/lib/python3.6/multiprocessing/pool.py", lin e 239, in _repopulate_pool w.start() File "/share/appspace_data/shared_groups/conda/env/bigscape/lib/python3.6/multiprocessing/process.py", line 105, in start self._popen = self._Popen(self) File "/share/appspace_data/shared_groups/conda/env/bigscape/lib/python3.6/multiprocessing/context.py", line 277, in _Popen return Popen(process_obj) File "/share/appspace_data/shared_groups/conda/env/bigscape/lib/python3.6/multiprocessing/popen_fork.py ", line 19, in init self._launch(process_obj) File "/share/appspace_data/shared_groups/conda/env/bigscape/lib/python3.6/multiprocessing/popen_fork.py ", line 66, in _launch self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory

Traceback (most recent call last): File "/share/appspace_data/shared_groups/software/BiG-SCAPE-1.1.5/bigscape.py", line 3261, in htmlFolder=network_html_folder) File "/share/appspace_data/shared_groups/software/BiG-SCAPE-1.1.5/bigscape.py", line 1648, in clusterJs onBatch p = subprocess.Popen(command, stdout=newick_file, shell=False) File "/share/appspace_data/shared_groups//conda/env/bigscape/lib/python3.6/subprocess.py", line 729, in init restore_signals, start_new_session) File "/share/appspace_data/shared_groups/conda/env/bigscape/lib/python3.6/subprocess.py", line 1295, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory Job 11861687 stderr output:

PuziJiang commented 2 weeks ago

And I have another question: The file "Network_Annotations_Full.tsv" shows the number of RiPPs is 40698, but the BGC numbers in the file "Network_Annotations_RiPPs.tsv" is 41263. Which results should I use?

jorgecnavarrom commented 2 weeks ago

Hi

About the first question: yes, unfortunately BiG-SCAPE 1 is very memory intensive. One option here would be to break down the analysis by running with --banned_classes so that you only calculate a single class at the time

I don't really know for the second question though; the class annotations should be a subset of the full one. Can you know which lines appear in "Full" that don't appear in "RiPPs"? Perhaps a comparison software like kdiff3, WinMerge or CompareMerge2

PuziJiang commented 2 weeks ago

Dear Sir, Thanks for your reply! I solved the memory problem when I used the parameter "--banned_classes".

About the number of BiG-SCAPE classes, I checked the Network_Annotations_Full.tsv and Network_Annotations of each class, and I found that some BGCs did not appear in "Full" but appear in "RiPPs". These BGCs are annotated as the hybrids of RiPPs and other type. I did not set any parameter, and BiG-SCAPE still put the hybrid BGCs in every different individual class, right?

WechatIMG1379 WechatIMG1380 WX20240829-100425@2x

github-actions[bot] commented 3 days ago

This issue has not seen activity for 14 days and has been marked as stale. Please comment with additional information if this issue is still relevant.