search

LarsDenBakker / web-test-runner

This package has moved to: https://github.com/modernweb-dev/web/tree/master/packages/test-runner
6 stars 1 forks source link

Create a terminal logging design #25

Closed LarsDenBakker closed 4 years ago

LarsDenBakker commented 4 years ago

When we have all the information for the terminal available, we should do a complete review of the logging:

TODO: https://github.com/LarsDenBakker/web-test-runner/issues/37

LarsDenBakker commented 4 years ago

Aside from test coverage, we now have all the logging info we need. The focus has been on getting all this info, we definitely need to think about how we want to format it and which things we might not want to display.

This is the general format we have right now:

<test file a>:
  <test name>: (failed on <browser name>)
    at: <test file a>:<line>:<column>
    debug in browser: <link to debug in browser>
    error: <error msg>
    <error rich diff or error stack>

  <another test name>: (failed on <browser name>)
    at: <test file a>:<line>:<column>
    debug in browser: <link to debug in browser>
    error: <error msg>
    <error rich diff or error stack>

  Browser logs:
    <Log A>
    <Log B> 

  <Browser name> logs:
    <Log A>

  Request 404s:
    <path a>
    <path b>

  <Browser name> request 404s:
    <path a>
    <path b>

<test file b>:
  Failed to load test file
    <error stack>

<test file c>:
  Unknown errors:
    <browser name a>
      <error stack>
    <browser name b>
      <error stack>

Test coverage: <total> % 
  Lines: xx.xx %, Statements: xx.xx %, Branches: xx.xx %, Functions: xx.xx %

Finished running tests!

Total:    |██████████████████████████████| 34/34 test files | 212 passed, 7 failed
  Chromium: |██████████████████████████████| 34/34 test files | 212 passed, 4 failed
  Firefox:  |██████████████████████████████| 34/34 test files | 210 passed, 6 failed
  Webkit:   |██████████████████████████████| 34/34 test files | 209 passed, 7 failed

Duration: 40s
LarsDenBakker commented 4 years ago

Some general thoughts about the structure:

We show the filename multiple times, once as the "category" at the root and once per test with the column + line so that you can jump directly to a test. Along with potentially a link to debug in the browser it feels like quite a lot of clutter.

Should we just show one total progress bar, and no progress bar per browser? I do like being able to see which browser is being slower, so that I know the impact of running tests on multiple browser during development for example.

Jest shows a code frame with the error location, for examle the expect where it went wrong. Do we think this helps? It adds another 7 or so lines in the terminal for an error.

Should we have two separate error reports, a summary overview when there are a lot of errors and a detailed view when there are few errors or when you are focusing on a test?

How to display syntax errors? Right now we just let the logs from es-dev-server come in, but we need to place it somewhere probably.

daKmoR commented 4 years ago

imho I like it 👍

the debug in browser we could display only if --debug is supplied?

for the summary I would prefer a view like this

Total:    |██████████████████████████████| 34/34 test files | 212 passed, 7 failed
--- Browsers? e.g. some separator, indentation seem strange --- 
Chromium: |██████████████████████████████| 4 failed
Firefox:  |██████████████████████████████| 6 failed
Webkit:   |██████████████████████████████| 7 failed

for the browser I am only interested in how many failed... all the other details I have in Total

LarsDenBakker commented 4 years ago

@daKmoR --debug opens the browser already, this link would be for when you're just noticing a failed test and want to see what's going on. I think I'd use that a lot personally.

LarsDenBakker commented 4 years ago

Test coverage was added now, I updated the format a few comments above. What I'm doing now is if your tests pass the coverage threshold, we just show the total coverage %. If you're failing one of the coverage markers, it displays the second line with the separate markers.

daKmoR commented 4 years ago

how will that work? 🤔 e.g. I do npm run test and it finishes (e.g. server is no longer running) and then I click the link?

or will the link only be added while in watch mode?

LarsDenBakker commented 4 years ago

Mhm yes it would be watch only

LarsDenBakker commented 4 years ago

@thepassle and I improved the terminal logging a lot today. Let's go with this, and improve it over time. I'll close this issue, and we can create issues for separate topics.

One thing to improve still is https://github.com/LarsDenBakker/web-test-runner/issues/52