search

nextest-rs / nextest

A next-generation test runner for Rust.
https://nexte.st
Apache License 2.0
2.2k stars 97 forks source link

[feature] progress in testing log #526

Open jiangying000 opened 2 years ago

jiangying000 commented 2 years ago

currently nextest supports progress bar, but I did not find a way to enable a log like below, it is an example made by hand:

Finished test [unoptimized + debuginfo] target(s) in 6m 01s
Starting 500 tests across 161 binaries (17 skipped)
[1/500]    PASS [   0.009s]                                        bcs-ext tests::test_bcs_enum_compact
[2/500]     PASS [   0.009s]                                        bcs-ext tests::test_serialize
  ....many lines
[400/500]    PASS [   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown
 [401/500]     PASS [   0.234s]                starcoin-service-registry::basc test_basic

so that when a open a part of CI log ( like the one on github, github give us tail of log when testing is running) we can see current testing progress

sunshowers commented 2 years ago

I agree, this would be valuable. Let me see what I can do here.

sunshowers commented 2 years ago

I'm going to call this a progress counter, as distinct from the progress bar.

When should the counter be displayed?

I think it would make sense to display it whenever the progress bar isn't displayed. Specifically, we'd want to add a CLI flag called --show-progress, with values bar, counter, auto (default) or none. If --show-progress is auto, then:

  1. If this is a CI environment, set it to counter.
  2. If standard error is a terminal, set it to bar.
  3. Otherwise, set it to counter.

If --show-progress bar is specified and standard error is not a terminal, then this implies none.

We'll also want to deprecate --hide-progress-bar, make it mean --show-progress none, and make it conflict with --show-progress so both of those can't be specified at the same time.

Where should the progress counter be shown?

I think the best place to show the progress would be to the left of the timer: e.g.

        PASS [400/500   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown

One thing to be careful of here is padding. This is what it should look like:

        PASS [ 99/500   0.009s]                                        bcs-ext tests::test_serialize
        PASS [100/500   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown

What should be displayed?

I think the progress counter should be kept really simple for now. Only the number of completed tests and the total number of tests to run should be displayed. We could potentially show failed tests here but that'll be too noisy I think.

Would you like to implement this, @jiangying000? I think it should maybe be 1.5 days of work or so.

jiangying000 commented 2 years ago

I added this to my schedule, would like to finish this in Mid Oct.

sunshowers commented 2 years ago

Perfect, thanks!

jiangying000 commented 2 years ago

Rain, sorry for being late, I have been deeling with some complex bugs in my full time work for some time and it might take longer, but I will definitely finish the feature

and please feel free to unassign me if you want to make it step forward faster

sunshowers commented 2 years ago

No worries! I don't plan to work on this any time soon so I'm happy to wait for your contribution.

jiangying000 commented 1 year ago

I'm going to call this a progress counter, as distinct from the progress bar.

When should the counter be displayed?

I think it would make sense to display it whenever the progress bar isn't displayed. Specifically, we'd want to add a CLI flag called --show-progress, with values bar, counter, auto (default) or none. If --show-progress is auto, then:

  1. If this is a CI environment, set it to counter.
  2. If standard error is a terminal, set it to bar.
  3. Otherwise, set it to counter.

If --show-progress bar is specified and standard error is not a terminal, then this implies none.

We'll also want to deprecate --hide-progress-bar, make it mean --show-progress none, and make it conflict with --show-progress so both of those can't be specified at the same time.

Where should the progress counter be shown?

I think the best place to show the progress would be to the left of the timer: e.g.

        PASS [400/500   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown

One thing to be careful of here is padding. This is what it should look like:

        PASS [ 99/500   0.009s]                                        bcs-ext tests::test_serialize
        PASS [100/500   0.232s]                starcoin-service-registry::basc test_launch_and_shutdown

What should be displayed?

I think the progress counter should be kept really simple for now. Only the number of completed tests and the total number of tests to run should be displayed. We could potentially show failed tests here but that'll be too noisy I think.

Would you like to implement this, @jiangying000? I think it should maybe be 1.5 days of work or so.

@sunshowers hello, rain, could you take a look, is this spec still valid a year later. I am going to impl this.

sunshowers commented 1 year ago

That's awesome @jiangying000! Yes, it looks right.