search

sourcegraph / cody

AI that knows your entire codebase
https://cody.dev
Apache License 2.0
2.22k stars 213 forks source link

Playwright V2 E2E Framework #4706

Open RXminuS opened 3 days ago

RXminuS commented 3 days ago

Completely revamped E2E testing framework for E2E tests, Issue Replications, and faster TDD UI development.

  1. eliminates all flake
  2. fully isolated tests + fully configurable per-test overrides (vscode version, installed extensions, config, etc)
  3. runs faster, fully parallel and headless
  4. leverages Playwright tracing UI instead of video recordings
  5. allows for easy recording and mocking of network traffic
  6. allows for automatic recording of tests by just "using VSCode"
  7. "hot-reloadable" extensions & tests (so watch mode works properly)

Test plan

keegancsmith commented 2 days ago

@RXminuS I just pushed some commits to make this work on my linux machine. Please take a look. Busy reviewing the code as well.

RXminuS commented 2 days ago

I think the BFG & Symf asset fetching is a problem in the extension in general, not just for these tests. I first noticed it while trying to fix the original E2E tests. The way to trigger it in VSCode is if for some reason you have two extension hosts and you reload the window with an updated extension it would break as two are concurrently trying to unzip to the same location.

I can definetly clean it up a bit into a cleaner and more isolated abstraction

RXminuS commented 2 days ago

When the fixture fails due to timeout it is really hard to debug. For some reason it doesn't blame to the actual line that was running:

I'll add it to the todo list because it's really annoying. I'm not sure if it's just something in Playwright though. You can actually see the original location further up in the logs and in the trace it does show correctly as well

keegancsmith commented 2 days ago

Ok the complexity is worth it then. I think it is probably worth avoiding concurrent downloads. I did some research cause I dig systems programming and wondered whats best. Everyone seems to do it differently, sqlite relies on fcntl locks and git relies on lock files but then asks the user to clean up if they are left lying around. Given we are in node, I quite liked the creativity of relying on mkdir as a locking primitive. It also has the same issue around assuming mtime is never in the future, but from reading the code I don't think it has a lot of the other potential pitfalls. https://www.npmjs.com/package/proper-lockfile

Edit looking through there open issues I only saw one proper legit one and it ended up being from playwright (who use this package!). I would also use there version of the package since that seems like a very annoying bug https://github.com/moxystudio/node-proper-lockfile/issues/111