search

web-platform-tests / wpt.fyi

web-platform-tests dashboard
https://wpt.fyi/
Other
188 stars 89 forks source link

History timeline missed test changes for Dec. 12th 2023 #3638

Open DanielRyanSmith opened 10 months ago

DanielRyanSmith commented 10 months ago

A timeline error was brought to our attention:

Hi, question about WPT. This test is failing in Chrome according to that website: https://wpt.fyi/results/css/css-contain/content-visibility/content-visibility-with-top-layer-005.html?label=master&label=experimental&product=chrome&product=firefox&product=safari&aligned&view=interop&q=label%3Ainterop-2022-contain%20or%20label%3Ainterop-2023-contain But if you click "show history timeline" it is green. That's a bug in the website right?

The timeline matches the data that is stored for the timeline but not the actual data. My initial guess is that somehow the datastore entities that populate the timeline were not recorded for this date. However, the fact that the timeline is consistently green for the entire week means that either the values are not being written each day, or that the failing status for this test was changed and saved in the "chrome_recent_statuses.json" that is used to track the most recently seen statuses.

DanielRyanSmith commented 10 months ago

Update: The history timeline seems to have updated with the correct value of "FAIL" for Chrome and Safari. However, the fail state was not recognized until Dec. 20th. This is likely caused by the fact that no aligned runs were available to from Dec. 11 to Dec. 19-20, therefore the history timeline was not updated.

This reveals an issue with the history timeline in which the date of the graph's data implicitly spans until the current day, regardless of the date of the last processed data. This missing data confusion might have been more obvious if the timeline's span was limited to the date of the last time the test/subtest's data was assessed.