MITM servers for fun tests

[kegsay]

There are concrete test cases which require us to fudge connectivity to the server or fudge responses the server sends back. We cannot currently do this using complement-crypto, because we lack the necessary tooling. This issue tracks the addition of such tooling.

NB: This is different to mock federation servers which Complement natively supports well.

Option A: Mock Client Server

Extend mock federation server support in Complement to support mock client server support. This would allow us to mock up registration, message sending, etc.

Pros:

• Test server code written in-line in the test.

Cons:

• Requires a LOT of mock code. E.g always need the ability to register, login, create rooms, send messages...
• At what point is this basically an HS in Go ala Dendrite?
• Bugs in mock code.

Option B: Reverse Proxy MITM

Shove nginx/apache/traefik in front of the real HS, and write reverse proxy configuration to send back mocked responses, reject endpoints, etc.

Pros:

• No need to mock registration, login, send messages, etc.

Cons:

• Test server code written in a different language.
• Increased learning curve as now requires knowledge of reverse proxy configuration.
• Potentially not in-line with the test, certainly won't have the syntax highlighting in the IDE.
• Limited flexibility, you can't write arbitrary responses back.

Option C: Go reverse proxy MITM

Use Go's httputil package to write a reverse proxy in the test.

Pros:

• No need to mock registration, login, send messages, etc.
• Test server code written in-line in the test.

Cons:

• Unclear how Go code is then shipped to the reverse proxy for the test, short of rebuilding the reverse proxy when it needs new code.
• Likely slow if rebuilding. Can maybe just run one and then ship instructions to execute?
• Any data serialisation format to ship the operations as data will end up looking like a reverse proxy configuration format and all the negatives it entails. Oh and it's homegrown so even less portable. Needs to be able to actually just handle Go code.
• Plugin?

Option D: Transparent(ish) RPC

Use something like https://github.com/hashicorp/go-plugin/ to write a plugin which can then communicate over some kind of RPC channel that we don't care about.

Pros:

• No need to mock registration, login, send messages, etc.
• Test server code written in-line in the test.

Cons:

• Pulls in a new library, with uncertainties around reliability, though it seems to be in use.

[dkasak]

Option E: mitmproxy DSL module for writing Matrix attack MITM proxies

Write a module for mitmproxy exposing a DSL for easily modifying:

a. Requests flowing into the homeserver, simulating malicious or faulty clients. b. Responses flowing out of the homeserver, simulating malicious or faulty homeservers.

Place this in front of either Synapse, Dendrite or Conduit.

Pros:

• No need to mock registration, login, send messages, etc.
• You get a nice TUI and web UI for inspecting and manipulating requests for free, as well as full scripting capability.
• Since modules are just Python modules, no need for recompilation. mitmproxy supports hot reloading of code on a change.
• You get Security to contribute since this is already on our long-term plan. :)

Cons:

• Limited introspection into the server's internal state, so for any information we'd require for which there is no API to ask the server about, we'd have to calculate it ourselves. Or just teach the module to connect to the server's database to get what it needs?

[kegsay]

Additional cons for option E:

• It's python, so language context switch problems the same as with reverse proxy configuration files.

[kegsay]

I ended up doing a blend of option C and D.

• it uses httputil to implement a reverse proxy.
• tests expose a "controller" server which can be manipulated in tests to alter both outgoing requests and incoming responses.
• the reverse proxy hits out to the controller for each request/response.
• HTTP requests and responses are serialised into request/response bodies. So the HTTP body of the request to the controller is itself an HTTP request, in standard HTTP format.

The pros/cons end up being:

Pros:

• No need to mock registration, login, send messages, etc.
• Test server code written in-line in the test.
• Can manipulate the request/response in any arbitrary way; not limited to returning static hard-coded data.

Cons:

• Handrolled special snowflake code. Reverse proxy is ~130loc, with ~300loc tests, controller code on top is ? loc.

[poljar]

Additional cons for option E:

• It's python, so language context switch problems the same as with reverse proxy configuration files.

But you can write Python using Rust. Actually, people already do that https://github.com/mitmproxy/mitmproxy_rs.

[kegsay]

Hmm, I do like the ability to have the web UI, and if it's a thing other people already use / have familiarity with then there is benefit to trying to make use of that.

In terms of configuration, these look to be equivalent (handrolled vs mitmproxy):

$ mitmdump --mode reverse:http://hs1@3000 --mode reverse:http://hs2@3001
$ REVERSE_PROXY_HOSTS=http://hs1,3000;http://hs2,3001 ./reverseproxy

Controlling what happens during the MITM process is the tricky bit I'm currently fleshing out. I want the API to look something like:

// stop /keys/query requests from working for alice, one time.
cancelInterception := tc.Deployment.ReverseProxyController.InterceptResponses(
    t, rp.RespondWithError(504, "Gateway Timeout"),
    rp.WithUserInfo(t, tc.Alice.BaseURL, tc.Alice.UserID, tc.Alice.DeviceID),
    rp.WithPathSuffix("/keys/query"),
    rp.WithRepititions(1),
)
defer cancelInterception()

so provided mitmproxy has the flexibility to do this then even if it is a custom DSL, it would be abstracted from the test. That way, those who do like mitmproxy can then make use of it.

[kegsay]

So it sounds like I want a custom add-on, which seems to be able to manipulate the request/response e.g https://docs.mitmproxy.org/stable/addons-examples/#http-add-header

However, I need this to be modifiable at runtime so each test can tweak what things get set. This probably means I need to have a port exposed on the mitmproxy container to accept arbitrary(?) addons.

[poljar]

So it sounds like I want a custom add-on, which seems to be able to manipulate the request/response e.g https://docs.mitmproxy.org/stable/addons-examples/#http-add-header

Yes, you will most certainly want to write an extension, or multiple ones.

This probably means I need to have a port exposed on the mitmproxy container to accept arbitrary(?) addons.

Would it perhaps work if have a master addon/extension which then picks which extension to actually execute and having it listen on a UNIX socket so you can control this behavior?

[kegsay]

That would compose pretty well if people then want to roll their own addons, it would just require the mitmproxy container to have all the addons preloaded in the image. It's also more secure than arbitrary code execution :D

[dkasak]

Also gives a nice foundation for reimplementing the Polyjuice tests in a potentially more robust way, due to not requiring a fake homeserver reimplementation.

[kegsay]

This now exists, but is missing contextual information to say "only do X for alice/device requests".

[kegsay]

We can go a long way just with mimproxy filters so using that for now. Will expand this when needed. Proof of concept: https://github.com/matrix-org/complement-crypto/blob/main/tests/client_connectivity_test.go#L34