Closed marioernestovaldes closed 5 years ago
marioernestovaldes
commented
5 years ago I have been checking the error and I found that .xml file inside QcSearchData folder doesn't contain anything, hence making unable to read the spectra...any suggestion?
kind regards!
kevinkovalchik
commented
5 years ago Sorry I haven't had a chance to look into this yet. I believe the IdentiPy search is failing for some reason, and then when RawTools tries to process the results it doesn't find them. Hence the empty XML file. But I don't think this is IdentiPy's fault. Likely there is a problem in how a directory path is set up in RawTools. We've had similar problems using identipy before. I'll see if I can replicate the issue and get it fixed, but in the meantime try using absolute paths instead of going from the home directory. That might help.
Kevin
On Jan 11, 2019 2:01 PM, "marioernestovaldes" notifications@github.com wrote:
I have been checking the error and I found that .xml file inside QcSearchData folder doesn't contain anything, hence making unable to read the spectra...any suggestion?
kind regards!
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kevinkovalchik/RawTools/issues/22#issuecomment-453671147, or mute the thread https://github.com/notifications/unsubscribe-auth/APT50OXYvGvxs-lFOCFfBDGMKZjASWdAks5vCQnMgaJpZM4Z6k1D .
marioernestovaldes
commented
5 years ago It's complicated because is the only step where I have problems... I tried with absolute paths, and get the same error... all the other commands work perfectly... let me know if you need anything...
thanks a lot!
best!
kevinkovalchik
commented
5 years ago Hm. Yeah, looking through the error message it looks like all the folders were created where they should have been. Could you try searching the generated MGF file directly with IdentiPy, just to confirm that RawTools is causing the issue?
Thanks, Kevin
marioernestovaldes
commented
5 years ago thanks for your quick answer... let me see that right now and I let you know...
marioernestovaldes
commented
5 years ago It seems like IdentiPy cann't generate the .xml file. Take a look, when I try to read .mgf file with IdentiPy...
mario@mario-biosystem:~/Programs/p/RawTools-1.1.0/qc_output/QcSearchData$ identipy /home/mario/Programs/p/RawTools-1.1.0/qc_output/QcSearchData/ch_23Aug2018_HeLa_Std_1.raw.mgf -db /home/mario/Programs/p/RawTools-1.1.0/uniprot_human-crap_sep2018_FWD.fasta
INFO: [17:58:52] Reading defaults from /home/mario/.local/lib/python2.7/site-packages/identipy/default.cfg
INFO: [17:58:52] Setting legend: {}
INFO: [17:58:52] Writing /home/mario/Programs/p/RawTools-1.1.0/qc_output/QcSearchData/ch_23Aug2018_HeLa_Std_1.raw.pep.xml ...
INFO: [17:58:52] Reading spectra ...
Traceback (most recent call last):
File "/home/mario/.local/bin/identipy", line 11, in
any suggestion?...
thanks...
kevinkovalchik
commented
5 years ago If I get a chance this weekend I'll try using IdentiPy on my system with an mgf file from 1.4.0 to see if I get the same error. I have an older version of IdentiPy, so that would tell us if the problem is something new in IdentiPy or if it is the MGF.
Kevin
On Fri, Jan 11, 2019, 5:03 PM marioernestovaldes <notifications@github.com wrote:
It seems like IdentiPy cann't generate the .xml file. Take a look, when I try to read .mgf file with IdentiPy...
mario@mario-biosystem:~/Programs/p/RawTools-1.1.0/qc_output/QcSearchData$ identipy /home/mario/Programs/p/RawTools-1.1.0/qc_output/QcSearchData/ch_23Aug2018_HeLa_Std_1.raw.mgf -db /home/mario/Programs/p/RawTools-1.1.0/uniprot_human-crap_sep2018_FWD.fasta
INFO: [17:58:52] Reading defaults from /home/mario/.local/lib/python2.7/site-packages/identipy/default.cfg INFO: [17:58:52] Setting legend: {} INFO: [17:58:52] Writing /home/mario/Programs/p/RawTools-1.1.0/qc_output/QcSearchData/ch_23Aug2018_HeLa_Std_1.raw.pep.xml ... INFO: [17:58:52] Reading spectra ... Traceback (most recent call last): File "/home/mario/.local/bin/identipy", line 11, in load_entry_point('identipy==0.2', 'console_scripts', 'identipy')() File "/home/mario/.local/lib/python2.7/site-packages/identipy/cli.py", line 219, in run utils.write_output(inputfile, settings, main.process_file(inputfile, settings)) File "/home/mario/.local/lib/python2.7/site-packages/identipy/utils.py", line 1402, in write_output return writer(inputfile, settings, results) File "/home/mario/.local/lib/python2.7/site-packages/identipy/utils.py", line 1086, in write_pepxml results = [x for x in results if x['candidates'].size] File "/home/mario/.local/lib/python2.7/site-packages/identipy/peptide_centric.py", line 230, in process_peptides kwargs = prepare_peptide_processor(fname, settings) File "/home/mario/.local/lib/python2.7/site-packages/identipy/peptide_centric.py", line 66, in prepare_peptide_processor for spec in utils.iterate_spectra(fname): File "/home/mario/.local/lib/python2.7/site-packages/identipy/utils.py", line 269, in iterate_spectra for x in f: File "/home/mario/.local/lib/python2.7/site-packages/pyteomics/auxiliary/file_helpers.py", line 170, in next return next(self._reader) File "/home/mario/.local/lib/python2.7/site-packages/pyteomics/mgf.py", line 295, in _read spectrum = self._item_from_offsets(offsets) File "/home/mario/.local/lib/python2.7/site-packages/pyteomics/mgf.py", line 291, in _item_from_offsets return self._read_spectrum_lines(lines) File "/home/mario/.local/lib/python2.7/site-packages/pyteomics/mgf.py", line 218, in _read_spectrum_lines 'Error when parsing %s. Line:\n%s' % (getattr(self._source, 'name', 'MGF file'), line)) pyteomics.auxiliary.structures.PyteomicsError: Pyteomics error, message: u'Error when parsing /home/mario/Programs/p/RawTools-1.1.0/qc_output/QcSearchData/ch_23Aug2018_HeLa_Std_1.raw.mgf. Line:\n132,1356 18,2373'
any suggestion?...
thanks...
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kevinkovalchik/RawTools/issues/22#issuecomment-453704854, or mute the thread https://github.com/notifications/unsubscribe-auth/APT50HC5k4jHhv6avKn_kagFHIgVEVSTks5vCTRKgaJpZM4Z6k1D .
marioernestovaldes
commented
5 years ago Ok! I'll be in touch... Have a nice weekend
Best!
kevinkovalchik
commented
5 years ago So everything worked fine when I tried it. Then I updated IdentiPy to the latest release, and everything worked fine again, so I'm not sure what is going on.
Earlier you said:
I have been checking the error and I found that .xml file inside QcSearchData folder doesn't contain anything, hence making unable to read the spectra
Is the .pep.xml file empty or the .mgf file? The xml file is generated by identipy and will hold the search results, so we would expect it to be empty if IdeniPy is crashing. The MGF file holds the spectra, so if it is empty that is a problem.
Can you share the MGF file with me? If it is rather large, you can make a small one using -N 100 or something as a qc parameter.
Kevin
marioernestovaldes
commented
5 years ago hope you had a nice weekend! .mgf file is fine... the .xml contains only these lines:
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="pepXML_std.xsl"?>
when I tried to process .mgf file with IdentiPy, I got this error...
identipy ch_23Aug2018_HeLa_Std_1.raw.mgf -db human.fasta
INFO: [10:21:26] Reading defaults from /home/mario/.local/lib/python2.7/site-packages/identipy/default.cfg
INFO: [10:21:26] Setting legend: {}
INFO: [10:21:26] Writing ch_23Aug2018_HeLa_Std_1.raw.pep.xml ...
INFO: [10:21:26] Reading spectra ...
Traceback (most recent call last):
File "/home/mario/.local/bin/identipy", line 11, in
So, looks like the problem is with IdentiPy. What version of IdentiPy are you using?
marioernestovaldes
commented
5 years ago here is the .mgf file I am using...
Please, let me know if you need other info...
thanks in advance...
El lun., 14 ene. 2019 a las 9:47, Kevin Kovalchik (notifications@github.com) escribió:
So everything worked fine when I tried it. Then I updated IdentiPy to the latest release, and everything worked fine again, so I'm not sure what is going on.
Earlier you said:
I have been checking the error and I found that .xml file inside QcSearchData folder doesn't contain anything, hence making unable to read the spectra
Is the .pep.xml file empty or the .mgf file? The xml file is generated by identipy and will hold the search results, so we would expect it to be empty if IdeniPy is crashing. The MGF file holds the spectra, so if it is empty that is a problem.
Can you share the MGF file with me? If it is rather large, you can make a small one using -N 100 or something as a qc parameter.
Kevin
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kevinkovalchik/RawTools/issues/22#issuecomment-454075175, or mute the thread https://github.com/notifications/unsubscribe-auth/AsbHnUc77pwutjZK1AJ8L2ZSWBlcmGVSks5vDLS4gaJpZM4Z6k1D .
kevinkovalchik
commented
5 years ago sorry, I don't see the file. Could you try attaching it again? GitHub might not accept something with a .mgf extension, so you can try putting it in a zip file if it isn't working.
Kevin
marioernestovaldes
commented
5 years ago sorry...here it is
regards!
El lun., 14 ene. 2019 a las 10:42, Kevin Kovalchik (< notifications@github.com>) escribió:
sorry, I don't see the file. Could you try attaching it again? GitHub might not accept something with a .mgf extension, so you can try putting it in a zip file if it isn't working.
Kevin
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kevinkovalchik/RawTools/issues/22#issuecomment-454094756, or mute the thread https://github.com/notifications/unsubscribe-auth/AsbHnbiS9Lfzsi70dHy8ZmkRpH037GwZks5vDMGBgaJpZM4Z6k1D .
marioernestovaldes
commented
5 years ago kevinkovalchik
commented
5 years ago Oh, interesting. So your MGF file uses commas for decimals and mine uses dots. I assume this must be because of your computer's language settings or something similar. I tried searching it and got the same error. I now notice the problem with the commas in the error message:
pyteomics.auxiliary.structures.PyteomicsError: Pyteomics error, message: u'Error when parsing
ch_23Aug2018_HeLa_Std_1.raw.mgf. Line:\n134,13287 2,1495'That last bit with Line:\n134,13287 2,1495.
Try this file: ch_23Aug2018_HeLa_Std_1_test.raw.zip I replaced all the commas in your file with periods. If this works for you, I will try to get RawTools to always write this format, regardless of the system's country/language settings.
marioernestovaldes
commented
5 years ago It's working!!! Sorry, I should have foreseen that... Thank you so much for your time and patience... I really appreciate that...
kind regards!!!
Mario E.
kevinkovalchik
commented
5 years ago Great! Oddly, this hasn't come up before. I will try to get a fix for this into a new release soon.
Kevin
kevinkovalchik
commented
5 years ago Hi Mario,
Can you try this version out and see if it works for you? RawTools_issue22.zip
I tried setting the cultural context for the application to default to "invariant", which is generally based on English practices. Hopefully this will fix the issue with the MGF files. If it works, though, it will change the output for everything, so any other file the program writes will also use "." rather than "," for decimals. Let me know if that is a problem.
Kevin
marioernestovaldes
commented
5 years ago sure! I will prove it right now and let you know...
cheers!
Mario
El mar., 15 ene. 2019 a las 15:26, Kevin Kovalchik (< notifications@github.com>) escribió:
Hi Mario,
Can you try this version out and see if it works for you? RawTools_issue22.zip https://github.com/kevinkovalchik/RawTools/files/2761705/RawTools_issue22.zip
I tried setting the cultural context for the application to default to "invariant", which is generally based on English practices. Hopefully this will fix the issue with the MGF files. If it works, though, it will change the output for everything, so any other file the program writes will also use "." rather than "," for decimals. Let me know if that is a problem.
Kevin
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kevinkovalchik/RawTools/issues/22#issuecomment-454575880, or mute the thread https://github.com/notifications/unsubscribe-auth/AsbHnefiqpXEUfBQpYnWWukrwigTGCMTks5vDlWWgaJpZM4Z6k1D .
marioernestovaldes
commented
5 years ago So, here it is... I already had solved the problem by setting up my system to "English", so the .mgf file would not have "," After your request, I turned around my system into Spanish and proved the new version... Everything is working perfectly, even with the system in Spanish, so, it seems the problem is fixed... please, let me know if I can help you in anything else...
cheers!
Mario
El mar., 15 ene. 2019 a las 16:14, Mario Ernesto (marioe911116@gmail.com) escribió:
sure! I will prove it right now and let you know...
cheers!
Mario
El mar., 15 ene. 2019 a las 15:26, Kevin Kovalchik (< notifications@github.com>) escribió:
Hi Mario,
Can you try this version out and see if it works for you? RawTools_issue22.zip https://github.com/kevinkovalchik/RawTools/files/2761705/RawTools_issue22.zip
I tried setting the cultural context for the application to default to "invariant", which is generally based on English practices. Hopefully this will fix the issue with the MGF files. If it works, though, it will change the output for everything, so any other file the program writes will also use "." rather than "," for decimals. Let me know if that is a problem.
Kevin
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kevinkovalchik/RawTools/issues/22#issuecomment-454575880, or mute the thread https://github.com/notifications/unsubscribe-auth/AsbHnefiqpXEUfBQpYnWWukrwigTGCMTks5vDlWWgaJpZM4Z6k1D .
kevinkovalchik
commented
5 years ago Great! Thanks for letting me know.
Take care, Kevin
Hello, Thanks for developing such a friendly tool!! I am following each one of the steps you describe in the wiki... I am working with Linux... Almost was perfect until "Quality control analysis with a database search"... this is the command I am using in version 1.4.0:
mono RawTools.exe qc -d ~/Programs/RawTools-1.4.0/ -q ~/Programs/RawTools-1.4.0/qc_output_wSearch/ -s identipy --db ~/Programs/RawTools-1.4.0/uniprot_human-crap_sep2018_FWD.fasta --fmods 57.0214@C --xmod 15.9949@M -N 1000 -p /usr/bin/python -I /usr/local/bin/identipy
and the output is as follows (I tried with version 1.1.0 and I got the same error):
Checking Identipy dependencies
All Identipy dependencies satisfied!
1 file(s) to process
Processing /home/mario/Programs/RawTools-1.4.0/ch_23Aug2018_HeLa_Std_1.raw
Determing MS analysis order... Done! Extracting scan indices: 100%
Checking for orphaned scans: 100%
================ Scan indexing report ================
Total scans in file: 11378
Scans linked: 11378
Orphan scans:
None!
All scans accounted for!
======================================================
Extracting reaction events: 100%
Extracting scan data: 100%
Extracting trailer extras: 100%
Extracting precursor masses: 100%
Extracting retention times: 100%
Analyzing precursor peaks: 100%
Calculating meta data
MS1 isolation interference
MS2 scan cycle density
Ion injection time
Duty cycle
Intensity distribution
Summed intensities
Calculating metrics
Writing MGF file: 100%
Starting Identipy
INFO: [17:07:49] Reading defaults from /home/mario/.local/lib/python2.7/site-packages/identipy/default.cfg INFO: [17:07:49] Reading config from /home/mario/Programs/RawTools-1.4.0/iontrap_config.cfg INFO: [17:07:49] Setting legend: {'!': ('aab', 'M', ''), 'aabM': '!'} INFO: [17:07:49] Writing /home/mario/Programs/RawTools-1.4.0/qc_output_wSearch/QcSearchData/ch_23Aug2018_HeLa_Std_1.raw.pep.xml ... INFO: [17:07:49] Reading spectra ... Traceback (most recent call last): File "/usr/local/bin/identipy", line 11, in
load_entry_point('identipy==0.2', 'console_scripts', 'identipy')()
File "/home/mario/.local/lib/python2.7/site-packages/identipy/cli.py", line 219, in run
utils.write_output(inputfile, settings, main.process_file(inputfile, settings))
File "/home/mario/.local/lib/python2.7/site-packages/identipy/utils.py", line 1402, in write_output
return writer(inputfile, settings, results)
File "/home/mario/.local/lib/python2.7/site-packages/identipy/utils.py", line 1086, in write_pepxml
results = [x for x in results if x['candidates'].size]
File "/home/mario/.local/lib/python2.7/site-packages/identipy/peptide_centric.py", line 230, in process_peptides
kwargs = prepare_peptide_processor(fname, settings)
File "/home/mario/.local/lib/python2.7/site-packages/identipy/peptide_centric.py", line 66, in prepare_peptide_processor
for spec in utils.iterate_spectra(fname):
File "/home/mario/.local/lib/python2.7/site-packages/identipy/utils.py", line 269, in iterate_spectra
for x in f:
File "/home/mario/.local/lib/python2.7/site-packages/pyteomics/auxiliary/file_helpers.py", line 170, in next
return next(self._reader)
File "/home/mario/.local/lib/python2.7/site-packages/pyteomics/mgf.py", line 295, in _read
spectrum = self._item_from_offsets(offsets)
File "/home/mario/.local/lib/python2.7/site-packages/pyteomics/mgf.py", line 291, in _item_from_offsets
return self._read_spectrum_lines(lines)
File "/home/mario/.local/lib/python2.7/site-packages/pyteomics/mgf.py", line 218, in _read_spectrum_lines
'Error when parsing %s. Line:\n%s' % (getattr(self._source, 'name', 'MGF file'), line))
pyteomics.auxiliary.structures.PyteomicsError: Pyteomics error, message: u'Error when parsing /home/mario/Programs/RawTools-1.4.0/qc_output_wSearch/QcSearchData/ch_23Aug2018_HeLa_Std_1.raw.mgf. Line:\n123,19139 3,1177'
Unhandled Exception: System.Xml.XmlException: Root element is missing. at System.Xml.XmlTextReaderImpl.Throw (System.Exception e) [0x00027] in <76ac1921b67d491f9f54fe5ecfa699fe>:0 at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo (System.String res) [0x00017] in <76ac1921b67d491f9f54fe5ecfa699fe>:0 at System.Xml.XmlTextReaderImpl.ParseDocumentContent () [0x0035d] in <76ac1921b67d491f9f54fe5ecfa699fe>:0 at System.Xml.XmlTextReaderImpl.Read () [0x0008c] in <76ac1921b67d491f9f54fe5ecfa699fe>:0 at System.Xml.XmlReader.MoveToContent () [0x0003a] in <76ac1921b67d491f9f54fe5ecfa699fe>:0 at System.Xml.Linq.XElement.Load (System.Xml.XmlReader reader, System.Xml.Linq.LoadOptions options) [0x0000e] in <4f39e78504054ebf9f17cd311f2b71f4>:0 at System.Xml.Linq.XElement.Load (System.String uri, System.Xml.Linq.LoadOptions options) [0x0000f] in <4f39e78504054ebf9f17cd311f2b71f4>:0 at System.Xml.Linq.XElement.Load (System.String uri) [0x00000] in <4f39e78504054ebf9f17cd311f2b71f4>:0 at RawTools.QC.SearchQC.LoadSearchResults (RawTools.WorkFlows.WorkflowParameters parameters, System.String rawFileName) [0x00020] in <45b2eaefc00c42588472cf9f27ecda76>:0 at RawTools.QC.SearchQC.ParseSearchResults (RawTools.Data.Containers.SearchMetricsContainer searchMetrics, RawTools.WorkFlows.WorkflowParameters parameters, System.String rawFileName) [0x00000] in <45b2eaefc00c42588472cf9f27ecda76>:0 at RawTools.WorkFlows.WorkFlowsDDA.QcDDA (ThermoFisher.CommonCore.Data.Interfaces.IRawFileThreadManager rawFileThreadManager, RawTools.WorkFlows.WorkflowParameters parameters) [0x00150] in <45b2eaefc00c42588472cf9f27ecda76>:0 at RawTools.QC.QcWorkflow.DoQc (RawTools.WorkFlows.WorkflowParameters parameters) [0x0014a] in <45b2eaefc00c42588472cf9f27ecda76>:0 at RawTools.Program.DoStuff (RawTools.ArgumentParser.QcOptions opts) [0x00148] in <45b2eaefc00c42588472cf9f27ecda76>:0 at RawTools.Program+<>c.b__0_1 (RawTools.ArgumentParser.QcOptions opts) [0x00000] in <45b2eaefc00c42588472cf9f27ecda76>:0
at CommandLine.ParserResultExtensions.WithParsed[T] (CommandLine.ParserResult:0
at RawTools.Program.Main (System.String[] args) [0x000d9] in <45b2eaefc00c42588472cf9f27ecda76>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.Xml.XmlException: Root element is missing.
at System.Xml.XmlTextReaderImpl.Throw (System.Exception e) [0x00027] in <76ac1921b67d491f9f54fe5ecfa699fe>:0
at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo (System.String res) [0x00017] in <76ac1921b67d491f9f54fe5ecfa699fe>:0
at System.Xml.XmlTextReaderImpl.ParseDocumentContent () [0x0035d] in <76ac1921b67d491f9f54fe5ecfa699fe>:0
at System.Xml.XmlTextReaderImpl.Read () [0x0008c] in <76ac1921b67d491f9f54fe5ecfa699fe>:0
at System.Xml.XmlReader.MoveToContent () [0x0003a] in <76ac1921b67d491f9f54fe5ecfa699fe>:0
at System.Xml.Linq.XElement.Load (System.Xml.XmlReader reader, System.Xml.Linq.LoadOptions options) [0x0000e] in <4f39e78504054ebf9f17cd311f2b71f4>:0
at System.Xml.Linq.XElement.Load (System.String uri, System.Xml.Linq.LoadOptions options) [0x0000f] in <4f39e78504054ebf9f17cd311f2b71f4>:0
at System.Xml.Linq.XElement.Load (System.String uri) [0x00000] in <4f39e78504054ebf9f17cd311f2b71f4>:0
at RawTools.QC.SearchQC.LoadSearchResults (RawTools.WorkFlows.WorkflowParameters parameters, System.String rawFileName) [0x00020] in <45b2eaefc00c42588472cf9f27ecda76>:0
at RawTools.QC.SearchQC.ParseSearchResults (RawTools.Data.Containers.SearchMetricsContainer searchMetrics, RawTools.WorkFlows.WorkflowParameters parameters, System.String rawFileName) [0x00000] in <45b2eaefc00c42588472cf9f27ecda76>:0
at RawTools.WorkFlows.WorkFlowsDDA.QcDDA (ThermoFisher.CommonCore.Data.Interfaces.IRawFileThreadManager rawFileThreadManager, RawTools.WorkFlows.WorkflowParameters parameters) [0x00150] in <45b2eaefc00c42588472cf9f27ecda76>:0
at RawTools.QC.QcWorkflow.DoQc (RawTools.WorkFlows.WorkflowParameters parameters) [0x0014a] in <45b2eaefc00c42588472cf9f27ecda76>:0
at RawTools.Program.DoStuff (RawTools.ArgumentParser.QcOptions opts) [0x00148] in <45b2eaefc00c42588472cf9f27ecda76>:0
at RawTools.Program+<>c.b__0_1 (RawTools.ArgumentParser.QcOptions opts) [0x00000] in <45b2eaefc00c42588472cf9f27ecda76>:0
at CommandLine.ParserResultExtensions.WithParsed[T] (CommandLine.ParserResult:0
at RawTools.Program.Main (System.String[] args) [0x000d9] in <45b2eaefc00c42588472cf9f27ecda76>:0
1[T] result, System.Action1[T] action) [0x0001e] in1[T] result, System.Action1[T] action) [0x0001e] incan you figure out what the error is? is there a problem with "identipy"?
thanks in advance...