search

fediverse-devnet / feditest

A testing framework for distributed, heterogeneous systems communicating with complex protocols, such as the Fediverse
https://feditest.org/
MIT License
23 stars 4 forks source link

Consider using asyncio for performance. #98

Open steve-bate opened 3 weeks ago

steve-bate commented 3 weeks ago

The test durations are dominated by HTTP response delays. Making the tests asynchronous could dramatically speed up the test plan execution.

jernst commented 3 weeks ago

IMHO let's optimize for speed when people complain. Chances are that they will complain about having to do too much setup work and manually perform actions on applications through a browser long before they worry about network-related speeds.

steve-bate commented 3 weeks ago

It's ok for now with ~10 tests (when the rel tests are stripped down). When we get to 100+ tests across 20+ server implementations, I'll be complaining. ;-) That will be especially true if we start doing tests across pair-wise combinations of servers or cluster combinations of size > 2 (for multiple-level delete tests, etc.).

jernst commented 7 hours ago

It seems to me we need to execute tests in strict sequence, including waiting for the first HTTP request to return before making the second one within one test. We could possibly run to tests against the same server in parallel, but that would require those tests to use different actors at the minimum, and problems would be hard to debug. Perhaps we could run several sessions in parallel, however, if they involve different Nodes.

steve-bate commented 9 minutes ago

The WebFinger tests don't need to be executed in sequence for a given session and the sessions don't require sequential execution.

With the WebFinger tests, we'd see approximately a 40x decrease in the test run duration since most of the time is from waiting for HTTP requests to complete or time out. I haven't looked closely at the early AP tests yet, so I can't comment on those.

It seems to me we need to execute tests in strict sequence

In general, this implies test coupling and that's not a good thing because it makes test failure diagnosis more difficult.

including waiting for the first HTTP request to return before making the second one within one test.

That's a different issue and is test-specific. Using asyncio doesn't prevent that behavior. In some cases, we'll need to send async requests in a sequence. In other cases, it would be acceptable to send them concurrently.

On the other hand, since we are using real servers for WebFinger testing, concurrent test execution might increase the number of errors we see from server request throttling (CloudFlare DDoS protection, for example) since the request rate would be higher.

We could possibly run to tests against the same server in parallel, but that would require those tests to use different actors at the minimum, and problems would be hard to debug. Perhaps we could run several sessions in parallel, however, if they involve different Nodes.

I was thinking more about session-level concurrency than for tests within a session. However, to the extent that the session tests are coupled, that will make problems harder to debug ("did this test fail because some other previous test mutated server state in an unexpected way?"). A typical real server handles many concurrent requests for the same actor so there might not be a need for different actors for different tests if the tests are designed to run independently. Even if there is an issue with that, we could create multiple actors (an actor pool) when the server is launched and allocate those to tests in a managed way.