search

NAVADMC / ADSM

A simulation of disease spread in livestock populations. Includes detection and containment simulation.
Other
10 stars 5 forks source link

Load testing #1017

Open missyschoenbaum opened 3 years ago

missyschoenbaum commented 3 years ago

I'm trying to load test. I'm bouncing between Prod, Beta and Prod on muscle machine. I'm having fails more than expected. After fail, smaller counts fail when they were successful before.

Fail means throwing red memory error - note that fail handles graceful shutdown perfectly.

Prod 7/27/2020 - MDL03 - Modsim - 500 iterations pass 7/26/2020 - MSCHO- HPAI MN - 1000 iterations pass 7/27/2020 - HomePC - Modsim - 500 fail

Beta 7/24/2020 - MSCHO - TX Import - 20 iterations pass 7/24/2020 - MSCHO - TX Import - 50 iterations pass 7/24/2020 - MSCHO - TX Import - 500 iterations fail 7/26/2020 - MSCHO- HPAI MN - 50 iterations pass 7/26/2020 - MSCHO- HPAI MN - 1000 iterations pass

missyschoenbaum commented 3 years ago

Batch test - MDL03 - used 6 ModSim scenarios (just changed name, didn't even open). Ran 500 iterations each One - Four completed with no issue. Five failed with memory error about 20 iterations in - Says it completed so assume graceful write Six tried to initiate, but never started, still in error mode

missyschoenbaum commented 3 years ago

@BryanHurst could it be that we have an error that is not getting caught? The Vaccination validation was not always getting caught. In one test example, I just found there was a vaccination ring, but no trigger.

These omissions could have happened as we put in new functionality.

BryanHurst commented 3 years ago

The error coming from the CEngine is pretty specific about it being a memory allocation error.

I'm doing a big update to the CEngine and recompiling to see if I can get this one knocked out. I'll update when done if there is something to test.

BryanHurst commented 3 years ago

Unfortunately after updating all the dependencies, I'm still getting the error.

I have a question off to Neil to see if he responds in the next day or two.

missyschoenbaum commented 3 years ago

When I looked through the examples I had to run batch, all of them were missing a vaccination trigger. So, some of them validated and ran with no issue.

BryanHurst commented 3 years ago

@missyschoenbaum In going back through previous builds, I've found that this bug has been around since we merged the Vaccination Rings branch in. I've tried several things on the Python side to help out, but it is definitely a bug that was introduced into the CEngine with the new Vaccination Rings feature.

I did go and compile the CEngine on Linux since Neil usually works in Linux. This bug seems to only be present on Windows.

missyschoenbaum commented 3 years ago

@BryanHurst Wow, hard to believe I missed it that long. I'm tinkering with the scenarios I ran batch with, The first four pass at 500 iterations (even when it appears there is the weird vacc validation thing). However, when I ramp up parameters that would increase spread, it fails.

Example - detection at 10, 14 days passes, at 21 days it fails.

missyschoenbaum commented 3 years ago

I looked at the overloading VX queue. It appears that 1,000 may be the max units that can be in the queue. I have successful runs up to 985 Units.