search

STAMP-project / dspot

Automatically detect and generate missing assertions for Junit test cases (also known as test amplification)
https://dspot-demo.stamp-project.eu/
GNU Lesser General Public License v3.0
114 stars 28 forks source link

Progression bar not working well #622

Open vmassol opened 6 years ago

vmassol commented 6 years ago

Hi, I've started dspot on xwiki-component-api with all amplifiers and it's taking very long (expected). However the progression bar doesn't seem to be working. It's been stuck at 89% for more than 1 hour now:

[INFO] 2018-10-29 13:31:56 Amplification - Assertion amplification: 0 test method(s) has been successfully amplified.
[INFO] 2018-10-29 13:31:56 Amplification - iteration 2 / 3
[INFO] 2018-10-29 13:31:56 Amplification - 200 tests selected to be amplified over 200 available tests
[INFO] 2018-10-29 13:31:56 NoBudgetizer - Amplification of inputs...
 89% |================================================= ]\

It's now 14:48 and it was started at 13:31:56...

Would be great to have progress since right now I don't know if it's stuck or just long.

vmassol commented 6 years ago

Been using 700%+ of my CPU for over an hour now...

vmassol commented 6 years ago

How long should I wait before considering it's stuck? I hope the answer is not one week :)

danglotb commented 6 years ago

The progress bar uses the number of test methods to be amplified. So DSpot amplifies 89% of 200 test methods.

Would be great to have progress since right now I don't know if it's stuck or just long.

How do you suggest to predict the time that the input amplification will take? At which granularity?

Been using 700%+ of my CPU for over an hour now...

You should use remote VMs, not your laptop.

How long should I wait before considering it's stuck? I hope the answer is not one week :)

From my experience, DSpot can take half day to input amplifiy a test method. It depends a lot of thing such as: the number of amplifier used, the number of potential input modification of the test method, the amount of memory avalaible etc...

I guess it won't be stuck but it will grow in memory until, in the worst case, throws a memory error.

vmassol commented 6 years ago

How do you suggest to predict the time that the input amplification will take? At which granularity? From my experience, DSpot can take half day to input amplifiy a test method.

So this suggests that having a progress bar on the amplified tests is too coarse grained. When DSpot amplifies a test method, I guess it's trying different things, so you could count or display those smaller steps.

How do you suggest to predict the time that the input amplification will take?

I don't think you should do that, that's too complex and brings little value IMO. What's important is that the user sees progress on the screen to know it's not stuck.

From my experience, DSpot can take half day to input amplifiy a test method. I guess it won't be stuck but it will grow in memory until, in the worst case, throws a memory error.

How am I supposed to use DSpot? It seems like it's going to be very hard to use if I have to wait for a OOM. Right now it's still running and I've been waiting for 2 hours, and that's just for one small module of XWiki :)

I guess I could try with only 1 amplifier but there are 9 of them with all possible combinations... Seems like there's a serious usability issue.

vmassol commented 6 years ago

Killing Dspot after 4 hours ;) (was still at 89% and not moving)