biocore / LabControl

lab manager for plate maps and sequence flows
BSD 3-Clause "New" or "Revised" License
2 stars 15 forks source link

Prevent silent timeout failure when computing pool values on "Pool library plates" page #361

Open AmandaBirmingham opened 6 years ago

AmandaBirmingham commented 6 years ago

Observed in maptest.ucsd.edu against qiita-rc db: Usually computing pool values for a single plate works fine, but occasionally for a single plate, and reliably if more than one (96-well) plate is added, clicking the "compute pool values" button produces no result (except that the button becomes permanently grayed out, and cannot be re-enabled even by changing pooling values, which should usually trigger a re-enable). Examining the JavaScript console shows that the POST to compute_pool is timing out:

screen shot 2018-10-04 at 12 16 11 pm

Ideally, we'd make this stop timing out :) At the very least, we should provide some sort of information to the user about why the page suddenly died on them, and how to recover. (Right now, the only way I know of to recover is to exit the page, come back in, and start over :( )

AmandaBirmingham commented 5 years ago

When this issue is fixed, the temporary text added to the pool library plates page by issue #386 should be removed!

antgonza commented 5 years ago

@AmandaBirmingham, do you have instructions of how to reproduce this error? Thanks!

AmandaBirmingham commented 5 years ago

@antgonza per above, in maptest.ucsd.edu, go to "Pool library plates" page, add multiple 96-well plates, and click the "compute pool values" button. In my experience, it doesn't matter which plate type (amplicon or shotgun) is selected, or which pooling function (minimum or equal volume), or what robot is selected. In my tests, I have left all other values as default.

HOWEVER, as noted in the but report, this failure doesn't always happen. I just tried this four times, and it silently failed one time out of the four; the other three, it worked (i.e, the web request didn't time-out).

antgonza commented 5 years ago

@AmandaBirmingham, thanks for the explanation. Looked at the logs and seems like this is the actual error:

[E 180124 16:48:30 web:2063] 500 POST /process/compute_pool (137.110.106.241) 30.51ms
[E 180124 16:48:47 web:1590] Uncaught exception POST /process/compute_pool (137.110.106.241)
    HTTPServerRequest(protocol='http', host='maptest.ucsd.edu:8080', method='POST', uri='/process/compute_pool', version='HTTP/1.1', remote_ip='137.110.106.241', headers={'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'X-Requested-With': 'XMLHttpRequest', 'Accept-Language': 'en-US,en;q=0.9', 'Cookie': 'jlinkengads=engads; _ga=GA1.2.1689508911.1515106271; _gid=GA1.2.196178017.1516638277; __utma=57960238.1689508911.1515106271.1516655287.1516679873.11; __utmz=57960238.1516679873.11.9.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); __unam=aaacafd-161021be724-4956035f-3; user=2|1:0|10:1516741104|4:user|28:ImFkc3dhZmZvcmRAdWNzZC5lZHUi|7dd376be278c36addd02325664b562a58c25a5734a6f3da2694647a6bf3e1028; jlinkserver=act; jssoserver=davis; jlinksessionidx=z735d13fc80a0ab819646bc50accd8851', 'Host': 'maptest.ucsd.edu:8080', 'Accept': '*/*', 'Referer': 'http://maptest.ucsd.edu:8080/process/poollibraries', 'Content-Length': '262', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36', 'Connection': 'keep-alive', 'Origin': 'http://maptest.ucsd.edu:8080'})
    Traceback (most recent call last):
      File "/home/pmapper/.virtualenvs/labman/lib/python3.5/site-packages/tornado/web.py", line 1509, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/home/pmapper/.virtualenvs/labman/lib/python3.5/site-packages/tornado/web.py", line 2897, in wrapper
        return method(self, *args, **kwargs)
      File "/home/pmapper/labman/labman/gui/handlers/process_handlers/pooling_process.py", line 283, in post
        output = self._compute_pools(plate_info)
      File "/home/pmapper/labman/labman/gui/handlers/process_handlers/pooling_process.py", line 173, in _compute_pools
        quant_process.compute_concentrations(**params)
    AttributeError: 'NoneType' object has no attribute 'compute_concentrations'
[E 180124 16:48:47 web:2063] 500 POST /process/compute_pool (137.110.106.241) 26.42ms

and so far these are the plates that have raised that error:

/plate/176/
/plate/23/
/plate/246/
/plate/26/
/plate/44/
/plate/45/

Just as a confirmation, do any of this match the one that failed out of the 4 tests you did?

AmandaBirmingham commented 5 years ago

Hmmm, of these, plate 246 could have been one of the ones I tried. Interestingly, the 500 errors you highlight above are different than the 504 ("gateway time-out") errors that originally spawned this issue. I think that maybe we need to address @wasade 's comment on https://github.com/jdereus/labman/issues/405 : " I think the jQuery calls for testing terms need to be modified to pay attention to the HTTP status codes spit back by the API, which I believe requires performing AJAX calls instead." Clearly multiple pages within the LabControl interface are behaving badly because they don't allow us to handle different errors differently. I will start taking a look at this ... suggest tabling this issue until we address the larger one.