search

microsoft / lis-test

Contains test infrastructure for testing Linux virtual machines on Windows Azure and Hyper-V.
32 stars 72 forks source link

Update Core_StressReloadModules to deeper stress #1097

Closed lixuemin2016 closed 6 years ago

lixuemin2016 commented 6 years ago

Hi Chris,

Currently there is one bug https://bugzilla.redhat.com/show_bug.cgi?id=1613655, if use original #100 times and sleep 1 for six times, it cannot catch the bug like https://bugzilla.redhat.com/show_bug.cgi?id=1613655, this case only reproduces on SMP, but not UP, so may need to add two cases. Do you want to change related cases.xml to add new test cases about StressReloadModules_UP and StressReloadModules_SMP? If yes, I will change it too?

Thank you so much.

chvalean commented 6 years ago

Yes @lixuemin2016 , I think that 2 test cases for UP and SMP would be the right way, especially given the origin of the ticket with that behavior. If we bump the loop count from 100 to 2000, what is the total running time of the test case? If it's over 30 minutes, I would put it though in the stress test category, or maybe split them into Core test with a limited number of repeats, and a more in-depth check with 2000 iterations that would go into the Stress tests area?

lixuemin2016 commented 6 years ago

@chvalean , updated case.xml to add two cases StressReloadModules_UP and StressReloadModules_SMP. After change to 2000 and remove 6 seconds sleep, running time about 4 minutes for one case. It should be ok to keep them in the Core test. Thank you so much.

lixuemin2016 commented 6 years ago

@chvalean, understand your consideration, if add sleep 1 second after unload/load, it will sleep 6 seconds for 3 modules, it needs about 12 minutes to finish 100 rounds. If only set 100 loop count, sometimes hard to detect this bug. If set as 200 loop count, it needs about 24 minutes to finish test, it looks too long run time in core test. What's your suggestion about this scenario? Add new cases to stress test as your original comment? Thank you.

chvalean commented 6 years ago

Unfortunately we have to keep the sleep for a proper module reload. I would suggest to maybe keep the SMP variation under the Core section, and under Stress add the same test but with a higher loop count and on single core. For stress maybe it might be worth checking in 2-3 hours how many iterations can be done. That might be sufficient as a baseline, and as you added the loop count as a param, it can be changed based on individual needs.

lixuemin2016 commented 6 years ago

@chvalean， thank you for your suggestion. Updated based on comment, change back the default loop count as 100, running time about 11 minutes. Also add duration time parameter, locally will set sleep time as 0 only to track regression issue in Core test. For stress test, will add sleep 1 second, we run core test for every compose, but not stress test.