web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
5k stars 3.1k forks source link

WPT report issue with parameter "::1" #41565

Open juliandescottes opened 1 year ago

juliandescottes commented 1 year ago

Mentioned by @thiagowfx at https://github.com/web-platform-tests/wpt/commit/bb6cf97f8d49604f274c019253ea6b08d2140464#r125054097

The following test is using "::1" as parametrized value: https://github.com/web-platform-tests/wpt/blob/3df1ed2d3a57ca4853ce996697fe7d8e9cfaa502/webdriver/tests/bidi/network/add_intercept/invalid.py#L138-L139

Logged at https://community-tc.services.mozilla.com/tasks/AUsx8q5ORmmu4f6J5rcjuA/runs/0/logs/public/logs/live.log#L10120

4:48.01 INFO STDOUT: webdriver/tests/bidi/network/add_intercept/invalid.py::test_params_url_patterns_pattern_hostname_invalid_value[::1] 4:48.01 pid:1196 1692618204034 RemoteAgent INFO Perform WebSocket upgrade for incoming connection from 127.0.0.1:40170 4:48.01 INFO STDOUT: PASSED

But when written in the wpt_report.json, the first part of the name gets removed, only 1] remains. wpt_report for the same job: https://community-tc.services.mozilla.com/api/queue/v1/task/AUsx8q5ORmmu4f6J5rcjuA/runs/0/artifacts/public%2Fresults%2Fwpt_report.json.gz . Relevant part:

{
  "name": "test_params_url_patterns_pattern_hostname_invalid_value[abc#com]", 
  "status": "PASS", 
  "message": null, 
  "known_intermittent": []
}, {
  "name": "test_params_url_patterns_pattern_hostname_invalid_value[abc:com]", 
  "status": "PASS", 
  "message": null,
  "known_intermittent": []
}, {
  "name": "1]", 
  "status": "PASS", 
  "message": null,
  "known_intermittent": []
}
thiagowfx commented 1 year ago

More specifically, the problem is the :: parsing. When adding another parameter, namely abc::com, the test name shows up as com] in the metadata, likewise.

gsnedders commented 1 year ago

This might be the fundamental test_id behaviour of pytest. There's an ids argument to paramterize: https://docs.pytest.org/en/latest/reference/reference.html#pytest.Metafunc.parametrize