search

compomics / peptide-shaker

Interpretation of proteomics identification results
http://compomics.github.io/projects/peptide-shaker.html
47 stars 19 forks source link

MS file exceeds memory mapped reader max buffer size. #468

Closed brvpuyve closed 3 months ago

brvpuyve commented 2 years ago

Hi,

I am trying to process large .mgf files (approximately 10Gb), but unfortunately SearhGUI crashes. The memory assigned to Java was already increased to 250 Gb, but no luck. Any suggestions on how to resolve this?

Best, SearchGUI 4.1.1 log.txt Bart

mvaudel commented 2 years ago

Hi Bart,

If nothing is suspicious about these mgf files (acquisition and peak-picking are ok), the low hanging fruit is to split your mgf files. I can fix our handling of the spectra to accommodate larger MS files, but I don't expect to have the time to look into it before October.

Apologies for the inconvenience,

Marc

kbarylyuk commented 5 months ago

Hi Marc and the team, I am running in the same error with a roughly 5.5 Gb mzML file, and I am wondering if there has been any development going on to enable handling larger files. I am not sure if a workflow with converting the data to MGF files and then splitting them is going to be optimal in my setting, but I am happy to give it a try if you could kindly provide a bit more detailed guidance on how to do it. Thank you! Konstantin PeptideShaker log:

Tue Jan 23 14:09:31 CET 2024: PeptideShaker version 3.0.5.
Memory given to the Java virtual machine: 34359738368.
Total amount of memory in the Java virtual machine: 218103808.
Free memory: 163524640.
Java version: 21.0.1.
1714 script command tokens
(C) 2009 Jmol Development
Jmol Version: 12.0.43  2011-05-03 14:21
java.vendor: Microsoft
java.version: 21.0.1
os.name: Mac OS X
memory: 68.0/167.8
processors available: 10
useCommandThread: false
java.io.IOException: File exceeds memory mapped reader max buffer size.
    at com.compomics.util.experiment.io.mass_spectrometry.cms.CmsFileWriter.addSpectrum(CmsFileWriter.java:109)
    at com.compomics.util.experiment.io.mass_spectrometry.MsFileHandler.writeCmsFile(MsFileHandler.java:224)
    at com.compomics.util.experiment.io.mass_spectrometry.MsFileHandler.register(MsFileHandler.java:150)
    at eu.isas.peptideshaker.gui.NewDialog$24.run(NewDialog.java:1005)
mvaudel commented 5 months ago

Hi Konstantin,

Time is flying! Apologies, I have not been able to prioritize this and am not sure when I can finally refactor this code. In the meantime splitting the file would be the way to go.

Sorry that I cannot help more at this stage,

Marc

hbarsnes commented 3 months ago

I've just released SearchGUI version 4.3.8 that should take care of the problem with the buffer size. If this is not the case, please let me know and I'll reopen the issue.