Upload failing integration tests to asciinema

[jesseduffield]

Is your feature request related to a problem? Please describe. When an integration test fails, it can be hard to see what went wrong. Although the output will tell at exactly which step the test failed, you still can't really know the cause of failure. Running the same test locally works fine most of the time, but if CI fails and local doesn't, you'll have to do a lot of guesswork.

Asciinema lets you record terminal sessions and save them to .cast files, or to upload them to the asciinema website. We could asciinema to capture test sessions so that it's easy to see what went wrong if a test fails. Here's an example recording

Describe the solution you'd like One possible approach would be the following: any failing integration tests on CI upload their recordings to asciinema and links to the recordings are added somewhere that's easy to find (e.g. as a comment on the PR or somewhere in the CI output).

How this could be done:

• we'd need CI to have asciinema installed (perhaps via a docker image that also contains Go)
• when running the test, we would no longer be running it in headless mode but instead running it through asciinema rec. For example as a proof-of-concept in pkg/integration/components/runner.go we can go:

  lazygitCmd := fmt.Sprintf("%s -debug --use-config-dir=%s --path=%s %s", tempLazygitPath(), paths.Config(), paths.ActualRepo(), test.ExtraCmdArgs())
  cmdStr := fmt.Sprintf("asciinema rec -y -t \"%s\" -c \"%s\"", test.Name(), lazygitCmd)

• we can store all the recordings locally, and then once the tests have been run, get the recordings of all failed tests and upload them.
• Then we would need to capture the URLs of the recordings and post them somewhere on the PR or in the CI output.

Additional context

• asciinema rec has a --yes arg which will automatically upload without prompting the user, but doesn't seem to have an arg for never uploading and always storing locally.
• I don't know if asciinema throttles based on uploads
• We would only want to upload the final run of the test if it fails (currently we retry failing tests once)
• If we don't want a dependence on the actual asciinema site we could upload the recordings to somewhere else (not sure if github actions lets us upload artifacts) and require that the dev downloads the recording and plays them with asciinema themselves.

[mark2185]

It does seem useful.

Going a step further, could we maybe pack it up into a docker snapshot and give that to the devs as some sort of core dump?

I mean, this is not for the end user so we can go a little bit the Evil Scientist route here.