search

broadinstitute / oncotator

Other
67 stars 32 forks source link

Open File Descriptors Issue #342

Closed jacmarjorie closed 8 years ago

jacmarjorie commented 8 years ago

I have recently ran into a problem running Oncotator in a "trusted" environment with a restricted amount of file descriptors (ie. 1024). It appeared Oncotator was opening files, and when it proceeded to the 1024 file the program failed with a "operation not permitted error".

We were curious if there is a specific reason so many file descriptors need to be open at a given time? Also, curious if anyone else has run into this issue?

Thanks!

Error, specifically when it goes to read the reference chromosome file:

File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 86, in _annotate_mut
    return ds.annotate_mutation(m)
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/datasources/ReferenceDatasource.py", line 89, in annotate_mutation
    m.createAnnotation('ref_context', self.getRange(m['chr'], iStart - (self.windowSizeRef+1), iEnd + (self.windowSizeRef-1)), annotationSource=self.title)
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/datasources/ReferenceDatasource.py", line 129, in getRange
    result = fp.read(iEnd-iStart+1)
IOError: [Errno 1] Operation not permitted

2016-01-29 23:22:47,478 ERROR [oncotator.output.TcgaMafOutputRenderer:280] Error at mutation 3024 ['21', '48084230', '48084230', 'C', 'T']:
2016-01-29 23:22:47,478 ERROR [oncotator.output.TcgaMafOutputRenderer:281] Incomplete: rendered 3024 mutations.

Another instance:

2016-01-21 11:11:23,763 ERROR [oncotator.output.TcgaMafOutputRenderer:279] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/output/TcgaMafOutputRenderer.py", line 267, in renderMutations
    for m in mutations:
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 414, in _applyManualAnnotations
    for m in mutations:
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 422, in _applyDefaultAnnotations
    for m in mutations:
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 485, in _annotate_mutations_using_datasources
    m = self._annotate_func_ptr(m, datasource)
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 86, in _annotate_mut
    return ds.annotate_mutation(m)
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/datasources/ReferenceDatasource.py", line 89, in annotate_mutation
    m.createAnnotation('ref_context', self.getRange(m['chr'], iStart - (self.windowSizeRef+1), iEnd + (self.windowSizeRef-1)), annotationSource=self.title)
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/datasources/ReferenceDatasource.py", line 129, in getRange
    result = fp.read(iEnd-iStart+1)
IOError: [Errno 1] Operation not permitted

2016-01-21 11:11:23,764 ERROR [oncotator.output.TcgaMafOutputRenderer:280] Error at mutation 6554 ['20', '62904723', '62904723', 'C', 'T']:
2016-01-21 11:11:23,765 ERROR [oncotator.output.TcgaMafOutputRenderer:281] Incomplete: rendered 6554 mutations.
Traceback (most recent call last):
  File "/usr/bin/oncotator", line 9, in <module>
    load_entry_point('Oncotator==1.5.1.0', 'console_scripts', 'oncotator')()
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Oncotator.py", line 292, in main
    annotator.annotate()
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 403, in annotate
    filename = self._outputRenderer.renderMutations(mutations, metadata=metadata, comments=comments)
  File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/output/TcgaMafOutputRenderer.py", line 283, in renderMutations
    raise e
IOError: [Errno 1] Operation not permitted
LeeTL1220 commented 8 years ago

You are the first. I am surprised... are you sure it is the file descriptor count? How do you know? On Feb 19, 2016 5:45 PM, "Jaclyn Smith" notifications@github.com wrote:

I have recently ran into a problem running Oncotator in a "trusted" environment with a restricted amount of file descriptors (ie. 1024). It appeared Oncotator was opening files, and when it proceeded to the 1024 file the program failed with a "operation not permitted error".

We were curious if there is a specific reason so many file descriptors need to be open at a given time? Also, curious if anyone else has run into this issue?

Thanks!

Error, specifically when it goes to read the reference chromosome file:

File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 86, in _annotate_mut return ds.annotate_mutation(m) File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/datasources/ReferenceDatasource.py", line 89, in annotate_mutation m.createAnnotation('ref_context', self.getRange(m['chr'], iStart - (self.windowSizeRef+1), iEnd + (self.windowSizeRef-1)), annotationSource=self.title) File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/datasources/ReferenceDatasource.py", line 129, in getRange result = fp.read(iEnd-iStart+1) IOError: [Errno 1] Operation not permitted

2016-01-29 23:22:47,478 ERROR [oncotator.output.TcgaMafOutputRenderer:280] Error at mutation 3024 ['21', '48084230', '48084230', 'C', 'T']: 2016-01-29 23:22:47,478 ERROR [oncotator.output.TcgaMafOutputRenderer:281] Incomplete: rendered 3024 mutations.

Another instance:

2016-01-21 11:11:23,763 ERROR [oncotator.output.TcgaMafOutputRenderer:279] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/output/TcgaMafOutputRenderer.py", line 267, in renderMutations for m in mutations: File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 414, in _applyManualAnnotations for m in mutations: File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 422, in _applyDefaultAnnotations for m in mutations: File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 485, in _annotate_mutations_using_datasources m = self._annotate_func_ptr(m, datasource) File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 86, in _annotate_mut return ds.annotate_mutation(m) File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/datasources/ReferenceDatasource.py", line 89, in annotate_mutation m.createAnnotation('ref_context', self.getRange(m['chr'], iStart - (self.windowSizeRef+1), iEnd + (self.windowSizeRef-1)), annotationSource=self.title) File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/datasources/ReferenceDatasource.py", line 129, in getRange result = fp.read(iEnd-iStart+1) IOError: [Errno 1] Operation not permitted

2016-01-21 11:11:23,764 ERROR [oncotator.output.TcgaMafOutputRenderer:280] Error at mutation 6554 ['20', '62904723', '62904723', 'C', 'T']: 2016-01-21 11:11:23,765 ERROR [oncotator.output.TcgaMafOutputRenderer:281] Incomplete: rendered 6554 mutations. Traceback (most recent call last): File "/usr/bin/oncotator", line 9, in load_entry_point('Oncotator==1.5.1.0', 'console_scripts', 'oncotator')() File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Oncotator.py", line 292, in main annotator.annotate() File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/Annotator.py", line 403, in annotate filename = self._outputRenderer.renderMutations(mutations, metadata=metadata, comments=comments) File "/usr/lib/python2.7/site-packages/Oncotator-1.5.1.0-py2.7.egg/oncotator/output/TcgaMafOutputRenderer.py", line 283, in renderMutations raise e IOError: [Errno 1] Operation not permitted

— Reply to this email directly or view it on GitHub https://github.com/broadinstitute/oncotator/issues/342.

jacmarjorie commented 8 years ago

Sorry for the late reply, yes we are sure this is the problem since we were able to isolate the issue to when we increased the number of allowed file descriptors.

LeeTL1220 commented 8 years ago

Are you using a cache (-u)?

On Tue, Mar 1, 2016 at 4:34 PM, Jaclyn Smith notifications@github.com wrote:

Sorry for the late reply, yes we are sure this is the problem since we were able to isolate the issue to when we increased the number of allowed file descriptors.

— Reply to this email directly or view it on GitHub https://github.com/broadinstitute/oncotator/issues/342#issuecomment-190915138 .

Lee Lichtenstein Broad Institute 75 Ames Street, Room 7003EB Cambridge, MA 02142 617 714 8632

LeeTL1220 commented 8 years ago

I am not sure, but where it is failing, the datasource only opens one file handle per contig, which is not very many.

lbergelson commented 8 years ago

Is this on hg 20? Doesn't that have many more contigs than hg19?

LeeTL1220 commented 8 years ago

@jacmarjorie Are you all set? Can we close this issue? As far as we know, Oncotator does not open that many file descriptors. Also, you are using an old version of Oncotator, though I make no promises that an upgrade will fix anything.

jacmarjorie commented 8 years ago

We are running this on hg19. Posting this issue was more just to let you know that Oncotator (https://github.com/broadinstitute/oncotator/archive/v1.8.0.0.tar.gz) fails when linux kernel restricts open file descriptors to 1024.

The general process will run through the first set of mutations (on a single chromsome) at which point it has opened up the chr file; however, once a mutation is read on the next chromosome, oncotator attempts to open a new chr file and the program crashes (prevents reading of the file since the kernel will not allow an additional file descriptor to be open). This behavior seemed to be coming from the fact that the previous chromsome file had not been closed.

Feel free to close, we can only fix the issue by increasing the amount of file descriptors that the kernel allows. Other than that oncotator has been a great program.

Thank you!

LeeTL1220 commented 8 years ago

Yes, but this implies that 1023 file descriptors were open before Oncotator even started....

Closing....

On Fri, Apr 15, 2016 at 2:00 PM, Jaclyn Smith notifications@github.com wrote:

We are running this on hg19. Posting this issue was more just to let you know that Oncotator ( https://github.com/broadinstitute/oncotator/archive/v1.8.0.0.tar.gz) fails when linux kernel restricts open file descriptors to 1024.

The general process will run through the first set of mutations (on a single chromsome) at which point it has opened up the chr file; however, once a mutation is read on the next chromosome, oncotator attempts to open a new chr file and the program crashes (prevents reading of the file since the kernel will not allow an additional file descriptor to be open). This behavior seemed to be coming from the fact that the previous chromsome file had not been closed.

Feel free to close, we can only fix the issue by increasing the amount of file descriptors that the kernel allows. Other than that oncotator has been a great program.

Thank you!

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/broadinstitute/oncotator/issues/342#issuecomment-210566222

Lee Lichtenstein Broad Institute 75 Ames Street, Room 7003EB Cambridge, MA 02142 617 714 8632