Set up end to end Send/Receive testing with LexBox

sillsdev / LfMerge

Send/Receive for languageforge.org

MIT License

2 stars 4 forks source link

Set up end to end Send/Receive testing with LexBox #342

Closed rmunn closed 2 months ago

rmunn commented 3 months ago

Fixes https://github.com/sillsdev/LfMerge/issues/339.

This PR uses the LexBox API to set up end-to-end testing of Send/Receive scenarios.

Work completed:

Reset project on LexBox to empty
Upload new state of LexBox project
Remove the last N commits from a LexBox project
LcmTestHelper class full of static methods
SRTestBase class that sets up and tears down temp folders appropriately, and can pass appropriate temp folders to LcmTestHelper methods
Commit .fwdata changes and push result to LexBox
First real end-to-end S/R test

Future plans:

In a new PR, I will port most of the SynchronizeActionTests tests and convert them to real end-to-end tests. Then I'll add some more tests testing things like round-tripping of comments, various merge conflict scenarios, and so on.

github-actions[bot] commented 3 months ago

Test Results

    2 files ±0   21 suites +1 5m 37s :stopwatch: -2s 315 tests +2 293 :heavy_check_mark: ±0 22 :zzz: +2 0 :x: ±0 318 runs +2 296 :heavy_check_mark: ±0 22 :zzz: +2 0 :x: ±0

Results for commit 9f09fc02. ± Comparison against base commit f518d8d9.

:recycle: This comment has been updated with latest results.

rmunn commented 2 months ago

First end-to-end test is written and working. Doing self-review now, then will ask for code review.

rmunn commented 2 months ago

The one failing test is the one this PR adds, with HttpRequestException : Connection refused (localhost:80), from a previous SocketException : Connection refused. This is quite simply because the LfMerge GitHub workflow doesn't set up a copy of LexBox before running the tests. Before merging this PR, we might want to set up a local copy of LexBox, similarly to how https://github.com/sillsdev/languageforge-lexbox/pull/873 did it, so that the end-to-end tests can run.

Might want to split the test runs in two, actually; one runner will run the non-E2E tests with the current setup, and the other runner will set up a Kubernetes environment in GHA to run the E2E tests. That way the non-E2E tests can be run without paying the cost of setting up a k8s environment.

rmunn commented 2 months ago

Okay, commit 779358e demonstrates the race condition with hgweb's dir cache in local LexBox, and commit 80d45bba adjusts these tests to expect newly-created projects to sometimes not be available right away, avoiding the race condition. To test a potential fix in LexBox, use commit 779358e and run dotnet test --filter E2E_ repeatedly until you get test failures (shouldn't take long).