search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.97k stars 3.09k forks source link

Interactive mode produces log that is confusing #41912

Open WeizhongX opened 1 year ago

WeizhongX commented 1 year ago

Mozlog have an interactive mode (see mozlog/mozlog/formatters/grouping.py), but the log produced at interactive mode is both confusing and incorrect.

Compared the attached log-non-interactive.txt and log-interactive.txt, (they are logs from same command except that for log-non-intertive.txt stdout is redirected to a text file), they are quite different. In log-interactive.txt, we can see a lot of "Restarting browser for new test type" without seeing any test running.

Also attached a video that captures the screen when running the command. I think we should not erase old lines on the screen.

WeizhongX commented 1 year ago

log-interactive.txt https://drive.google.com/file/d/1Ec72LHJz-6BBcX-D4kb0WaF722iPtKex/view?usp=drive_link&resourcekey=0-PheDWyEskiI_WrtSMcLYxA

log-non-interactive.txt https://drive.google.com/file/d/1gg9E20VY_mn0rpYERqCX6T0R7S7t1u3r/view?usp=drive_link&resourcekey=0-F87ZvwSY8LcCuFG9eYvZhQ

Wptrunner.mov https://drive.google.com/file/d/1OFc3DrvAiVKatcLVtNic-jnEHb838cPI/view?usp=drive_link&resourcekey=0-XZLZK0_6Cw2xpivU4LQKRA

WeizhongX commented 1 year ago

@jgraham

WeizhongX commented 1 year ago

It would also be nice if we could log the test runner number in the logs.

jgraham commented 1 year ago

FWIW the grouping formatter originates with @mrobinson's work in Servo.

mrobinson commented 1 year ago

@WeizhongX I think you need to give access to your Google Drive links.

We've made some advancement to the Servo version of this formatter (including fixing some bugs). There are still a few issues to fix. It would be possible to remove the erasing behavior of the formatter, but the idea here is that you can run 1000 tests and follow the failures in real-time without having to look at the scrollback.

WeizhongX commented 1 year ago

@mrobinson Thanks, can you request access so that I can grant the permission? I was not able to find your email, and corporate security did not give me an option to share with anybody.

Is it possible for you guys to upstream your side changes?

Our old test runner has a similar feature, which only erases the last line to achieve a similar effect. I plan to upload a video record for that also.