search

biocore / LabControl

lab manager for plate maps and sequence flows
BSD 3-Clause "New" or "Revised" License
2 stars 15 forks source link

Stress test the software #173

Open tanaes opened 6 years ago

tanaes commented 6 years ago

We need to stress test the database with synthetic data to assure it will scale.

ElDeveloper commented 6 years ago

Was this issue fixed by #192?

josenavas commented 6 years ago

I don't think so, since this specific issue talks about adding large quantities of data to the DB to test where it is slow. However, the functions added in #192 can be used for this.

tanaes commented 6 years ago

Update: using the stress_tests command of tester.py in PR #197, I added 1000 Amplicon plates to the DB to try it out. (I killed it before it added the shotgun plates b/c i was getting impatient)

Population the 'plate list / sample plates' table takes about 2 minutes, while the 'quantify plates / add plates' modal table takes about 3:45 to populate.

AmandaBirmingham commented 5 years ago

We have had numerous indications that the current code does not scale acceptably, including #364, #361, #385. It is not known right now if this is due to database performance issues, tornado performance issues, or something else. Someone is going to have to put in some serious effort to identify the source(s) of the problem and then figuring out how to fix them.

antgonza commented 5 years ago

@charles-cowart / @AmandaBirmingham, I think this issue can be closed, right? Currently the only outstanding issue from the list is https://github.com/jdereus/labman/issues/361, which is an issue by itself ...

unique-identifier commented 5 years ago

@antgonza the issue title is pretty broad; my guess is that there's more testing that could be done from now until full launch, even if #364, #361, and #385 are done.

AmandaBirmingham commented 5 years ago

So far, I am not convinced that this is done. Below is the description I have of what stress testing was going to be done; if these steps have been done, can someone mine the info on them out of whatever email trail they ended up in and put them here? If not, can we discuss why we no longer plan to do this work before we close this issue?

From Charlie, Amanda, Daniel meeting 03/19: We were asked to generate "a plan to profile / isolate the known performance issues". Our plan is to generate python code that can programmatically add 100,000 samples and their processed results to a test labcontrol installation, testing the performance of the system as we go and at the end of the process. We will insert the samples as mixes of 96- and 384-well plates in groups of 1000 at a time, tracking how long the programmatic insertion takes and testing the interface performance manually after each 1000. If we reach a point at which either (a) the programmatic insert process slows to an unacceptable speed or (b) the interface becomes unusable due to performance issues, we will stop the process (the person doing the testing will make the judgment call about whether either of these thresholds has been reached). After the test data load is completed (either to the full 100,000 samples or to the point where it was stopped for performance concerns) we will manually examine and/or profile the code to identify the source(s) of the slow-downs. The test load process may provide guidance on these source(s), since if the programmatic loads work well, then the performance issues are likely limited to the tornado interface, whereas if the programmatic load is slow, then database accesses are a likely culprit.