search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.82k stars 2.13k forks source link

Richer tests for Synapse modules #11084

Open reivilibre opened 3 years ago

reivilibre commented 3 years ago

It would be useful to have richer tests for Synapse modules.

The idea of making Complement extensible (if it's not already) and doing tests that way has been suggested but it seems like quite a chunk of work.

One simple-seeming way of doing better tests for Synapse modules might be to do tests in the style of actual Synapse features. For example, using HomeserverTestCase and checking that various REST requests all give the expected results.

(We seem happy to acknowledge for now that this could be a bit brittle if we wind up refactoring Synapse's test helpers, but that's infrequent enough)

This is not as simple as I first thought; Synapse's test helpers are contained in the tests package which is explicitly excluded by setup.py: https://github.com/matrix-org/synapse/blob/b3698f945cabb44649df03617db3c51d9fc62674/setup.py#L141 .

The effect of this seems to be that it's not possible (at least, without imagining some nasty hacks) to import those test helpers into a module test.

What should we do about this?

Naïvely, I would perhaps like to see these pulled out into a separate package which is installable as e.g. matrix-synapse-test-helpers, but really not sure what is ergonomic or typical here in Python-land (it also seems like we'd have a circular dependency if we did that, synapse <---> test helpers).

clokep commented 3 years ago

@reivilibre The way I've usually seen this done is by having the test helpers as a sub-module of the package itself, e.g. synapse.tests or something could include the helpers we wish to publicly expose to modules.