Closed cmdcolin closed 9 years ago
@cmdcolin It looks like it opens 1-3 files per track loaded and doesn't seem to increase (most of the time) and will eventually release the file handles.
Loading the bam track (Forager RNA-Seq reads), I see that it loads around 14 . . . goes up to 20 or so and then plateaus.
With 3 BAM files I got it to around 80-100 . . and it kind of maxed out (moving back and forth and changing groups).
That being said, in the code there is this line . . can't but help to fix:
// TODO: move up so not recalculating each time
String pathTranslated = request.getPathTranslated();
String rootPath = pathTranslated.substring(0, pathTranslated.length() - request.getPathInfo().length());
String configPath = rootPath + "/config/config.properties";
File propertyFile = new File(configPath);
String filename = null;
should probably call propertyFile.close()
I wasn't sure whether we should put it in the try{} finally {} block or just manually close it
Actually we should probably just load the file into memory so that it doesn't have to do that on every get request right?
I think it is working now . . . It probably wouldn't hurt to have it in the init, but performance wise it doesn't make any difference and I am using the "request" explicitly.
That is what I ended up doing. I had a note to do that eventually.
Nathan
On Feb 6, 2015, at 5:18 PM, Colin Diesh notifications@github.com wrote:
Actually we should probably just load the file into memory so that it doesn't have to do that on every get request right?
— Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/162#issuecomment-73342220.
@cmdcolin If you think this fixed it I would say go ahead and close it.
Thanks it seems to work so i'll close it for now
It probably wouldn't hurt to have it in the init
I would prefer this, it looks a little messy currently, so...whenever we get a chance in our copious amounts of free time...refactor it
Reopen:
I thought this was a fluke but the first time you open webapollo after a deploy, it returns
SyntaxError: expected expression, got ')' when parsing data/trackList.json.
This is reproducable on two different servers but I haven't narrowed down the cause yet.
Hmm . . do you have the trackList.json? It could be something introduced by the JBrowse Servlet, but I suspect its a faulty tracklist.json. I just tested this and it worked great.
I'm fairly sure it is independant of trackList.json (i tested two different servers, local dev and production) and both have it
It only happens the first time you access it, after that, presumably, the config.properties is initialized (since it is initialized within the GET request handler)
For completeness though here is a basic trackList.json
==removed==
Again, the error goes away after you refresh again
I but it is having all of the asynchronous requests coming through. I guess we’ll have to move it to the init servlet method. Give me a few to fix this.
Nathan
On Feb 9, 2015, at 12:02 PM, Colin Diesh notifications@github.com wrote:
I'm fairly sure it is independant of trackList.json (i tested two different servers, local dev and production) and both have it
It only happens the first time you access it, after that, presumably, the config.properties is initialized (since it is initialized within the GET request handler)
For completeness though here is a basic trackList.json
{ "share_link" : 0, "plugins" : [ { "location" : "./plugins/WebApollo", "name" : "WebApollo" } ], "tracks" : [ { "chunkSize" : 20000, "storeClass" : "JBrowse/Store/Sequence/StaticChunked", "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-", "category" : "Reference sequence", "type" : "SequenceTrack", "label" : "DNA", "key" : "Reference sequence" }, { "autocomplete" : "none", "style" : { "className" : "annot", "renderClassName" : "annot-render", "subfeatureClasses" : { "non_canonical_three_prime_splice_site" : "noncanonical-splice-site", "wholeCDS" : null, "exon" : "container-100pct", "CDS" : "annot-CDS", "non_canonical_five_prime_splice_site" : "noncanonical-splice-site", "UTR" : "annot-UTR" }, "arrowheadClass" : "annot-arrowhead", "alternateClasses" : { "transposable_element" : { "renderClassName" : "blue-ibeam-render", "className" : "blue-ibeam" }, "pseudogene" : { "renderClassName" : "gray-center-30pct", "className" : "light-purple-80pct" }, "snRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "rRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "snoRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "repeat_region" : { "className" : "magenta-80pct" }, "tRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "ncRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "miRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" } }, "uniqueIdField" : "id", "centerSubFeature" : { "non_canonical_three_prime_splice_site" : false, "non_canonical_five_prime_splice_site" : false } }, "key" : "User-created Annotations", "storeClass" : "WebApollo/Store/SeqFeature/ScratchPad", "disableJBrowseMode" : false, "phase" : 0, "compress" : 0, "label" : "Annotations", "type" : "WebApollo/View/Track/AnnotTrack", "subfeatures" : 1 } ], "names" : { "url" : "names/", "type" : "Hash" }, "favicon" : "./plugins/WebApollo/img/webapollo_favicon.ico", "alwaysOnTracks" : "DNA,Annotations", "formatVersion" : 1 } https://cloud.githubusercontent.com/assets/6511937/6114505/3fbfeb1a-b064-11e4-9cc2-970de6742e1f.jpg Again, the error goes away after you refresh again
— Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/162#issuecomment-73579260.
There were a few problems . . . and it was serendipity that it was working in the first place (properties was checking for a global, but assigning a local).
Both of these are now fixed (doing in the init servlet).
Let me know if it doesn’t fix it for you. Still not sure why it ended up working for me.
Nathan
On Feb 9, 2015, at 12:02 PM, Colin Diesh notifications@github.com wrote:
I'm fairly sure it is independant of trackList.json (i tested two different servers, local dev and production) and both have it
It only happens the first time you access it, after that, presumably, the config.properties is initialized (since it is initialized within the GET request handler)
For completeness though here is a basic trackList.json
{ "share_link" : 0, "plugins" : [ { "location" : "./plugins/WebApollo", "name" : "WebApollo" } ], "tracks" : [ { "chunkSize" : 20000, "storeClass" : "JBrowse/Store/Sequence/StaticChunked", "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-", "category" : "Reference sequence", "type" : "SequenceTrack", "label" : "DNA", "key" : "Reference sequence" }, { "autocomplete" : "none", "style" : { "className" : "annot", "renderClassName" : "annot-render", "subfeatureClasses" : { "non_canonical_three_prime_splice_site" : "noncanonical-splice-site", "wholeCDS" : null, "exon" : "container-100pct", "CDS" : "annot-CDS", "non_canonical_five_prime_splice_site" : "noncanonical-splice-site", "UTR" : "annot-UTR" }, "arrowheadClass" : "annot-arrowhead", "alternateClasses" : { "transposable_element" : { "renderClassName" : "blue-ibeam-render", "className" : "blue-ibeam" }, "pseudogene" : { "renderClassName" : "gray-center-30pct", "className" : "light-purple-80pct" }, "snRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "rRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "snoRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "repeat_region" : { "className" : "magenta-80pct" }, "tRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "ncRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" }, "miRNA" : { "renderClassName" : "gray-center-30pct", "className" : "brightgreen-80pct" } }, "uniqueIdField" : "id", "centerSubFeature" : { "non_canonical_three_prime_splice_site" : false, "non_canonical_five_prime_splice_site" : false } }, "key" : "User-created Annotations", "storeClass" : "WebApollo/Store/SeqFeature/ScratchPad", "disableJBrowseMode" : false, "phase" : 0, "compress" : 0, "label" : "Annotations", "type" : "WebApollo/View/Track/AnnotTrack", "subfeatures" : 1 } ], "names" : { "url" : "names/", "type" : "Hash" }, "favicon" : "./plugins/WebApollo/img/webapollo_favicon.ico", "alwaysOnTracks" : "DNA,Annotations", "formatVersion" : 1 } https://cloud.githubusercontent.com/assets/6511937/6114505/3fbfeb1a-b064-11e4-9cc2-970de6742e1f.jpg Again, the error goes away after you refresh again
— Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/162#issuecomment-73579260.
Looks good now
If a bunch of tracks are opened at once, then many duplicate file handles are opened for config.properties, basically until it hits the system limit.
Normal operation: around 250 file handles open
After opening up about 10 bam tracks and scrolling around: over 800 file handles open, all duplicates of config.properties
It will just increase until the "too many open files error" or the garbage collector picks up the objects