search

bocoup / test262-report-issue-tracker

Public issue tracker for the Test262 Report:
https://test262.report/
11 stars 1 forks source link

Unexplained regression in JavaScriptCore results #28

Open jugglinmike opened 3 years ago

jugglinmike commented 3 years ago

In recent (as of 2021-06-14) results published to test262.report, some of the data for JavaScriptCore appears to be inaccurate.

For example, the test at built-ins/Atomics/wait/nan-for-timeout.js passed on 2021-05-17 but failed on 2021-06-10. We're told that the JSC maintainers did not modify the implementation of Atomics during this period, and the test continues to report as "passing" in the project's own continuous integration environment.

This suggests that there is a flaw in the test262.report results collection mechanism.

jugglinmike commented 3 years ago

This appears to have been caused by a change in eshost, a utility which this project depends on transitively through test262-harness.

I've filed a pull request to address the problem. If/when that is merged and released, the results collection process will automatically receive the new version, and the subsequently-collected JavaScriptCore results should be more authentic.