Open richvdh opened 5 years ago
IMHO there would be need to:
Remark: Tor is actually the only stable solution for real federation. There are alot of systems able to run synapse out there, but people have connectivity problems. IPv6 will not be available to the masses within the next 10 years, so Tor is the only chance for all people to run their own Synapse Homeserver.
I was just about to open an issue for this. There are a few things that would be important to implement:
Support using a proxy for all outgoing connections as a homeserver. This would be required for the .onion
Matrix server to have any hope of being anonymous -- otherwise all federation out of the Matrix server (or through the Matrix server) would trivially reveal its IP address.
There is an argument that it should be possible to set (in a room's options) that only users which are on .onion
homeservers are allowed to join (sort of like the "this room is not federated" option today) and for the server to have a configuration which only allows federation with .onion
homeservers (but I think you can do that already with the federation whitelist). This way you can set up conversations that explicitly only will be shared by anonymous homeserver users.
Get most (if not all) homeservers to route .onion
addresses. While I believe messages transit through federation as a broadcast, you don't want there to be only a few hot servers which route .onion
traffic (not just because it becomes a pretty serious single point of failure but also for performance reasons). However, we should probably allow for .onion
routing to be disabled since some users might not feel comfortable running a Tor client (even though there should be no reason to worry about this).
And it would be nice for clients to indicate when they are in a .onion
room or if a .onion
user is talking. IRC clients have in the past had issue with Tor users (effectively) social-engineering conversations to go to certain topics, but marking users as being anonymous would be useful. Clients should also tell users to use Tor if they're trying to log into a .onion
homeserver.
Is there any news on this topic?
I'm also interested in running a .onion homeserver - am happy to contribute on this topic in any way I can.
I'm also looking into how to do this. Main features necessary for implementation seem to be adding SOCKS5 proxy support and disabling SSL for .onion urls. I'm not a huge python guy so I'm not sure, but it looks like changes for this could be confined to this file. Help from anyone with a better orientation of this codebase would be greatly appreciated with respect to integrating this.
This would de-anonymize each server that is set-up to do this (bridge traffic between .onion and clearnet) as well as expose metadata of all the clients of each of these servers.
Anyone who is bridging traffic between .onion and clearnet I'm assuming doesn't care about anonymity and probably is running their homeserver on clearnet. It would not reveal the IP addresses of .onion homeservers that route all traffic over Tor.
The simplest solution would be to just add Tor (as a client) to every synapse server which would avoid any privacy issues, but this might not be practical (some people may have concerns about running Tor on their homeservers -- in some countries, running Tor at all may result in a visit from the authorities). And if not all servers run Tor, then we should avoid fragmenting the federation (it probably isn't good for the Matrix ecosystem if you end up in a situation where publicly-federatable servers can only be accessed by users on some homeservers).
it probably isn't good for the Matrix ecosystem if you end up in a situation where publicly-federatable servers can only be accessed by users on some homeservers
why would this be an issue? especially since the alternative is those homeservers just not existing? Pretty sure matrix will handle this pretty gracefully. Ideally, big homeservers like matrix.org would have tor support, and if a user is running against a small homeserver that isn't running tor, and can't connect to rooms hosted on a .onion, the error would be pretty expected. Perhaps there is something about how matrix federation works that I'm missing.
as mentioned in may 2019 above i still think matrix has massiv need for Tor routing. Especially where Dendrite will be able to run a homeserver as single binary on all OS soon, Tor is THE ONLY solution which allows connects to the HS without port forwarding, NAT and dynamic IP problems.
The is NO practicable soltuion for normal users to run their HS without Tor. And please do not name strange UDP something P2P stuff which is not even fully understood by profis. For non techies this will never be a replacement and these ones are also to easy blockable by providers aso. . Especially where Tor Libs exists for Go and others. So it would be possible to create a single static binary without extra Tor setup needed.
Federation as I see it is the possibility for EVERYBODY to run his own HS for himself or the familiy/friends. This needs Tor otherwise it will not work for non techies.
And in that case, anonymisation in not priority 1, it is the possibility to get a jfkslkjflks.onion address which is reachable without any extra setup. And it works everywhere, kinda unblockable compared to UDP stuff which is blocked in probably 90% of the firewalls anyway.
Would love to see this would be built in.
I would be interested as well :)
Interested a lot. We need that feature
Not sure if this is the right place to ask.
Is it currently possible to setup a (non-anonymous) Tor onion service for Synapse? The location of the server doesn't need to be protected and server-to-server traffic doesn't need to be routed over Tor. Only the Matrix clients need to be able access the server via a .onion service.
Hey is there's a fix for this issue element is not working on Tor socks proxy and i use cloudflare on my private server i noticed matrix.org is using cloudflare as well and tor socks proxy work very well on matrix.org so maybe there's a config missing in cloudflare to activate
Not sure if this is the right place to ask.
Is it currently possible to setup a (non-anonymous) Tor onion service for Synapse? The location of the server doesn't need to be protected and server-to-server traffic doesn't need to be routed over Tor. Only the Matrix clients need to be able access the server via a .onion service.
Except cloudflares global onion service, which is enabled for matrix.org, I haven't seen any implementation in the wild, where you have an additional onion service next to your public matrix service.
Let's make this happen.
It is pretty pointless having a secret server for people to chat on, without the server being anonymous.
TOR is the ideal option.
Is there any progress on this? Are SOCKS5 proxies already supported in either Synapse or Dendrite? This would be the only missing piece right now for federating .onion homeservers without clearnet federation. TLS checks can be disabled already on both.
Still looking for this in 2022. Any news/updates about supporting routing .onion over tor?
Support for I2P would be even better!
For those of you who are happy to tinker, you could use an HTTP to SOCKS proxy wrapper like polipo (no longer maintained) for a proof of concept. Haven't tried it myself but there are some guides out there. Both Synapse and Dendrite support outgoing HTTP proxies. https://www.marcus-povey.co.uk/2016/03/24/using-tor-as-a-http-proxy/
Please add tor support :+1:
EmbassyOS has been supporting the Synapse Matrix server over Tor for quite some time now. I tested it a bit and it works fine. Their GitHub page: https://github.com/Start9Labs/embassy-os
I give you a federation-over-Tor proof-of-concept. It wasn't easy, takes three times more work than a standard instance, but let's try to enumerate the steps and propose further improvements to make the process easier for others.
tor
client and privoxy
. For rev-proxy I used caddy
, for resolving DNS I simply fiddled with Tor and resolv.conf
, but you can set up bind or even a whole separate machine.
server_name: "X.onion"
listeners:
- port: 8008
tls: false
type: http
x_forwarded: true
bind_addresses: ['::1', '127.0.0.1']
resources:
- names: [client, federation]
compress: false
trusted_key_servers:
- server_name: "matrix.org"
accept_keys_insecurely: true
allow_public_rooms_over_federation: true use_insecure_ssl_client_just_for_testing_do_not_use: true federation_verify_certificates: false tls_certificate_path: /etc/.../cert.pem tls_private_key_path: /etc/.../key.pem
The domain name (`X`) comes from `tor` client, it's a well-documented process, I'll skip that here. The server backend runs on unencrypted and uncompressed at `127.0.0.1:8080` as it usually does with Reverse Proxy, perhaps you can expose it directly to Tor, good luck. I had to turn on some generally insecure options, namely `accept_keys_insecurely`, `use_insecure_ssl_client_just_for_testing_do_not_use` and `federation_verify_certificates` as explained below.
3. The certificates I had to generate myself:
`openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout key.pem -out cert.pem -subj "/CN=example.com" -addext "subjectAltName=DNS:X.onion"`
using, of course, the `X` domain name returned by Tor. Why use these at all? Well, it's a good question. The first Synapse suggestion would be to stop relying on https in federation. There is not much point in it since I use Tor addresses anyways and certs being self-signed yield various complications:
a) To connect to my node with Element (or any electron-based client), I need to run it with `--ignore-certificate-errors`. I can, however, run the client-facing site on plain http, but it would be silly since I have to encrypt `:8448` traffic anyway.
b) Caddy won't just let me use any self-signed certificate, the SAN list needs to match my domain. It also won't auto-create it with zerossl, as they don't support `.onion` domains.
c) Synapse also checks certificates and I had to flip those unsafe options. For non-over-Tor federations that would be a risk, so at minimum, Synapse should recognize `.onion` addresses and allow unencrypted/unverified-certificate connections with them.
4. Tor configuration seems pretty straightforward:
HiddenServiceDir /somewhere HiddenServicePort 80 127.0.0.1:80 HiddenServicePort 443 127.0.0.1:443 HiddenServicePort 8448 127.0.0.1:8448 SOCKSPort 127.0.0.1:9050 AutomapHostsOnResolve 1 DNSPort 127.0.0.2:53
Obviously, after first run it creates the `hostname` file in `/somewhere` which you can use to configure the rest. I open three ports here: `80` for normal Tor traffic, `443` so that Element can connect to my site without prepending the address with "http://" and `8448` for federations. Socks is necessary for `privoxy`, and thus, for federations. I'm not sure if `AutomapHostsOnResolve` is now required, there were some DNS resolution errors from Synapse when I experimented, but at least I won't forget it now. And the most interesting one: `DNSPort 127.0.0.2:53` - you can ask: why not `127.0.0.1:9053`? Well, because you cannot use `127.0.0.1#9053` as a parameter to one of the python libraries used in Synapse, that's why.
5. Privoxy is easy:
listen-address 127.0.0.1:8118 forward-socks5t / 127.0.0.1:9050 .
What it does is using `tor`s SOCKS proxy and wraps it into a nice http proxy that Synapse can understand. Just remember to export:
export http_proxy=http://127.0.0.1:8118 export https_proxy=http://127.0.0.1:8118
in your system service definition when running Synapse so that it can connect to your overseas homeservers. Normal, non-over-Tor federations are easy without it, but the receiving party needs to understand that the inbound connection is coming from `.onion` server, otherwise you will get a `401` error in Synapse logs.
6. Finally, `Caddyfile`:
X.onion:8448 { tls /etc/.../cert.pem /etc/.../key.pem reverse_proxy http://localhost:8008 }
X.onion:80 { redir https://X.onion }
X.onion:443 {
tls /etc/.../cert.pem /etc/.../key.pem
encode gzip
reverse_proxy /_matrix/ http://localhost:8008
reverse_proxy /_synapse/ http://localhost:8008
handle_path /.well-known/matrix/* {
respond /client { "m.homeserver": { "base_url": "https://X.onion" } }
respond /server { "m.server": "X.onion:443" }
respond /support { ... }
}
}
It handles a few things. Since we are forced to run at port `443` (Element will let you register/login to a non-https homeserver if you prepend the address with `http://` but it will malform a `http://X.onion` address into `matrix:httpX.onion` when you try to discover rooms on a homeserver with such address), let's just redirect all `:80` connections to `:443`. On `:8448` we handle federation traffic and it's encrypted since Synapse requires it. The main configuration needs to include `.well-known` delegation since we don't have SRV records in Tor. And since federation failed on this lack of SRV support in Tor, I added a small patch to Synapse to make it not fail the connection over tor: https://github.com/matrix-org/synapse/pull/15523
7. To allow Synapse resolving `.onion` domains, the DNS must support it. Since we enabled it in `torrc`, just let the system resolver know it by putting `nameserver 127.0.0.2` in `/etc/resolv.conf` (works with Windows too). There is some redundancy of this step with step 5 but I didn't really get deep into that, it seems plausible to assume that privoxy should provide some mechanism for this.
9. What works, what doesn't? I was able to create a room, join it from another homeserver, chat a little. I didn't try VoIP (no TURN server) but I expect it to be possible over Tor in the foreseeable future. I didn't try integrations and bots, as I don't expect it to be any different. The performance is not great, not terrible. Obviously, double traffic encryption doesn't help. I didn't profile CPU usage neither.
I'm not saying this guide is the state-of-art thing. Perhaps I've been working with Synapse instances suboptimally, maybe some config is redundat or plainly wrong. More ideas are welcome, I will be monitoring this thread as I expect this to be a game-changing feature. Maybe someone can now create a Dockerfile based on my hints and publish it here, I will be happy to connect with more instances from my machine.
Edit: changed `SOCKSPort` and privoxy socks target to `127.0.0.1` so it's not exposed to the network.
EmbassyOS has been supporting the Synapse Matrix server over Tor for quite some time now. I tested it a bit and it works fine. Their GitHub page: https://github.com/Start9Labs/embassy-os
That’s correct, the Synapse over Tor wrapper repository is available at https://github.com/Start9Labs/synapse-wrapper
I give you a federation-over-Tor proof-of-concept.
@zdzichu-rks great work!
We have a Start9 - Tor Party Room that you can use to test federation over tor with your instance:
#tor-party:matrix.privacy34kn4ez3y3nijweec6w4g54i3g54sdv7r5mr6soma3w4begyd.onion
We have a Start9 - Tor Party Room that you can use to test federation over tor with your instance:
Yup, it does work, however the amount of Synapse Errors is worrysome:
4 synapse.federation.sender.per_destination_queue
10 synapse.handlers.federation
22 synapse.replication.tcp.handler
34 synapse.http.matrixfederationclient
55 synapse.media.media_repository
93 twisted
131 synapse.handlers.federation_event
139 synapse.crypto.keyring
334 synapse.federation.federation_client
For all I know, most homeservers expect inbound SSL connections from the address they can resolve and fetch server keys. Privoxy connection limit gets exhausted pretty fast and it took over 10 minutes to load room history. I suppose Synapse should at least gracefully acknowledge some other server can't understand Tor and ignore it.
My patch is now live in https://github.com/matrix-org/synapse/releases/tag/v1.84.0, perhaps anyone could now give feedback if this is working for them without hiccups.
Not saying this ticket should be closed yet, but perhaps a definitive TODO list could be now compiled and tracked.
For a not-just-Tor instance, which doesn't care about its own anonymity, the following differences make sense:
privoxy.conf
:
forward-socks5t .onion 127.0.0.1:9050 .
so that non-.onion
traffic is not routed trough Tor, just making it faster.
torrc
we skip all the HiddenService...
lines, as we don't host any hidden site. Obviously, we still need some DNS domain, certificates and such, just as regular instance should already have set.homeserver.yaml
, still need:
trusted_key_servers:
accept_keys_insecurely: true
use_insecure_ssl_client_just_for_testing_do_not_use: true
federation_verify_certificates: false
suppress_key_server_warning: true
Which is the main issue here - as we break SSL so that MITM attacks are now possible. It's still encrypted and will work just fine until someone wants to impersonate another instance, but Synapse should allow finer configuration, e.g. just limiting this to .onion
connections.
/etc/systemd/system/multi-user.target.wants/matrix-synapse.service
in [Service]
section:
Environment="export http_proxy=http://127.0.0.1:8118"
Environment="export https_proxy=http://127.0.0.1:8118"
/etc/resolv.conf
is in fact frequently overwritten. Just run chattr +i /etc/resolv.conf
after your changes. This could also be handled better by Synapse - if run with Tor, just use it for name resolution instead of system-wide DNS settings.127.x.y.z
are being blacklisted by Synapse, perhaps under some assumption it is not an actual network connection. No worries, in homeserver.yaml
:
ip_range_whitelist:
- '127.0.0.0/8'
- 'fe80::/10'
X.onion:8448/
: An error occurred while connecting: 22: Invalid argument.
. It comes from synapse.http.federation.matrix_federation_agent
, the _do_connect
function, which might be another candidate for my patches, e.g. if it's not respecting ssl certificate ignorance flags set above. But after all, my T2T (tor2tor) node is working fine, so I'm not sure why it only occurs now. The only real difference is privoxy setting and I already checked if using the /
as target pattern helps, it doesn't.Scratch 4. above, it should be (no export
s):
Environment="http_proxy=http://127.0.0.1:8118"
Environment="https_proxy=http://127.0.0.1:8118"
Other than this, pay attention to what your rev-proxy (Caddy in my case) certs configuration is on your .onion
service. In general, I'm not saying you can't run with http-only, something like:
http://X.onion:8448 {
reverse_proxy http://localhost:8008
}
X.onion:443 {
redir http://X.onion
}
X.onion:80 {
...
But this probably will fail on the redir
part, as Synapse's federation agent will simply fail when presented with Caddy's default or no certificate. Instead, my self-signed config from the first post above seems to do just fine.
There is also a possibility to configure client
and federation
listeners separately, e.g. by letting clients to use non-encrypted site and avoid running Element with --ignore-certificate-errors
but I haven't got that much free time.
It seems that both use cases (T2T and hybrid) work fine now and now I really see why don't you try and run it yourselves. It really does bring Matrix experience to the new level, especially if you own a vanity .onion
domain.
My further experiments led to two new points:
Synapse documentation is useful. One can use:
federation_verify_certificates: true
federation_certificate_verification_whitelist:
- "*.onion"
Seems pretty self-explanatory. Useful both for T2T and hybrid instances. Completely fixes my headaches regarding federation MITM attacks.
Synapse documentation is inaccurrate. https://matrix-org.github.io/synapse/latest/setup/forward_proxy.html doesn't mention, that we can also add:
Environment="matrix_proxy=http://127.0.0.1:8118"
To actually use proxy (and Tor!) while collecting keys from other instances via :8448
. It seems to use urllib.request.getproxies_environment
underneath which will collect any env var and try to match it with schemes like http
, https
, and matrix
. Since I didn't have matrix
proxy set up, my federations failed in synapse.http.federation.matrix_federation_agent
, now they're fine.
Seems I can now join, browse public rooms, pull media in both directions and none of my instances yields any warnings related to the other one. Will monitor these further.
There is some issue with joining some spaces based on hybrid instance from my T2T instance if there are users from another not-tor-aware instances already in that space, effectively blocking me from joining it.
On hybrid instance the log shows:
2023-06-07 XX:XX:XX,XXX - synapse.federation.sender.transaction_manager - 178 - WARNING - federation_transaction_transmission_loop-6 - TX [foreign.nontor.instance] {X} Remote returned error for $some_event_id: {'error': "ERROR 403: unable to verify signature for sender domain mytorinstance.onion: 401: Failed to find any key to satisfy: _FetchKeyRequest(server_name='mytorinstance.onion', minimum_valid_until_ts=X, key_ids=['ed25519:a_bRKp'])"}
And then after a timeout, my T2T client gets:
MatrixError: [403] Auth events could not be found (https://mytorinstance.onion/_matrix/client/r0/join/%23some-room%3Amy.hybrid.instance?server_name=my.hybrid.instance)
I suppose there is a bug in spaces synapse code that fails the join if some foreign host doesn't acknowledge the event, while with joining regular rooms there is no such issue.
It might help if the foreign instance had my hybrid instance set as a trusted key server, but nobody is going to do that.
Following the steps above, it seems largish media (6mb+) shared on one TOR server is not able to be retrieved by another federated tor server. Smaller media federates just fine between the two. The larger file can be successfully downloaded by users on the same HS that the uploader used.
Are there any tweaks that could be implemented to get around this?
Environment: Synapse 1.86 Debian TOR Privoxy
Clients used: Element (Web) Schildi (Web) Element (Desktop)
Tweaks so far:
Homeserver.yaml:
federation:
client_timeout: 10m
max_short_retry_delay: 5m
max_long_retry_delay: 10m
max_short_retries: 5
max_long_retries: 20
Nginx conf:
proxy_request_buffering off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
client_body_timeout 600;
send_timeout 600;
proxy_read_timeout 600;
Error received
Either 62b file download - or -
{"errcode":"M_UNKNOWN","error":"Failed to fetch remote media"}
Anyone who's backing a TOR only solution remember that TOR has the possibility of being compromised. We should be embracing a network agnostic approach. It seems like there are several distinct but nonexclusive use cases that need to be addressed individually.
The above POC shows potential.
I'd like an option that allows administrators to:
And
An option to
And
An option to block all traffic not coming from certain domains/protocols (TOR/I2P) to enforce anonymous only homeservers. Maybe this should be left to the server admin to configure on their network and not a consideration of the Synapse server at all?
It would also be really nice to be able to make the implementation agnostic to the anonymization network and even be able to run both TOR and I2P for larger shared homeservers. There are valid criticisms of all of the available anonymization networks and the end users should be allowed to choose which works best for them based on their requirements and threat model.
Since this issue is specifically about TOR there is no reason to discuss I2P specifically but please take other anonymization networks into consideration when coming up with an approach.
It would be nice to be able to configure synapse so that it can federate both with servers which are on .onion addresses, and those which are on regular clearnet addresses. Currently this is tricky, because there is no way to tell synapse to route some hostnames via a tor proxy whilst routing others as normal