search

web-platform-tests / results-collection

Other
41 stars 46 forks source link

Edge runs are too slow (20-30h) #623

Open foolip opened 6 years ago

foolip commented 6 years ago

Hovering the recent runs on https://wpt.fyi/test-runs?product=edge, Edge runs have taken 20-30h.

This is probably due to https://github.com/web-platform-tests/results-collection/commit/27c22db74aefb3e1ae93d34a76c7c9502ce48b64.

gsnedders commented 6 years ago

And despite @jugglinmike's theory, it hasn't made the results any more reliable AFAICT.

jugglinmike commented 6 years ago

Both the increase in time-to-results and the negligible effect on correctness were expected:

Do you think it'd be worth trying more aggressive restarting

The fact that this immediately interferes with almost every "chunk" makes me think that restarting will not resolve it, but it couldn't hurt to try (especially now that we're disabling Safari on Sauce Labs). We'll see what effect this has on time-to-results (it may get ugly).

As anticipated, the results continue to be invalid. From later on in that discussion thread:

We've completed a collection from Edge that involved restarting after every test failure. This increased build time from 15 hours to 24 hours, but it had no appreciable affect on the veracity of the results.

There's little value in improving the time-to-results without fixing the underlying problem. Since the fix will itself shorten the duration to some extent, I'm going to hold off on reverting the referenced commit.

foolip commented 6 years ago

https://wpt.fyi/results/?sha=349d418380&label=stable took 19 hours, so that below the range I observed, but very narrowly.