Open MatthieuBarthel opened 2 years ago
MatthieuBarthel
commented
2 years ago I just tried to make a request from python inside the synapse container and it works :thinking:
root@6151c1073bd8:/# python
Python 3.9.10 (main, Mar 2 2022, 04:31:58)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://matrix.org')
<Response [200]>
DMRobertson
commented
2 years ago Hi @MatthieuBarthel, have you changed any TLS-related settings in synapse's configuration?
MatthieuBarthel
commented
2 years ago Hi @DMRobertson, thanks your help. I don't see anything special in homeserver.yaml. Maybe it could be related to SSL_CERT_FILE environment variable which points to my CA public key ? Here my docker-compose.yml :
services:
postgres:
image: postgres:14
environment:
- POSTGRES_DB=synapse
- POSTGRES_USER=synapse
- POSTGRES_PASSWORD=xxxxx
- POSTGRES_INITDB_ARGS=--locale C --encoding UTF8
volumes:
- ./postgresdata:/var/lib/postgresql/data
synapse:
image: matrixdotorg/synapse:v1.54.0
environment:
- SYNAPSE_REPORT_STATS=no
- SYNAPSE_SERVER_NAME=imatt.ch
- SSL_CERT_FILE=/data/iMattRootCA.crt
deploy:
labels:
- traefik.enable=true
- traefik.http.routers.matrix.rule=Host(`matrix.imatt.ch`)
- traefik.http.routers.matrix.entrypoints=websecure
- traefik.http.routers.matrix.middlewares=securityHeaders@file
- traefik.http.services.matrix.loadbalancer.server.port=8008
networks:
- default
- traefik
volumes:
- ./synapse:/data
networks:
traefik:
external: trueIf I remove this environment variable synapse won't start, so I cannot test easily with my current setup. I'll try to make a test installation in the next days. Thanks again
DMRobertson
commented
2 years ago I wonder if setting that environment var is somehow preventing openssl from using the system CA certs in /etc/ssl/certs. I don't think that should be the case from my reading of https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html ---but perhaps I'm mistaken.
If I remove this environment variable synapse won't start
What error message(s) do you get if you do this?
MatthieuBarthel
commented
2 years ago What error message(s) do you get if you do this?
I says it cannot initialize the OIDC provider (which has the self signed certificate) :
matrix_synapse.1.wpn85n6vbswn@web9 | Traceback (most recent call last):
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/app/_base.py", line 227, in wrapper
matrix_synapse.1.wpn85n6vbswn@web9 | await cb(*args, **kwargs)
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 371, in start
matrix_synapse.1.wpn85n6vbswn@web9 | await oidc.load_metadata()
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/handlers/oidc.py", line 123, in load_metadata
matrix_synapse.1.wpn85n6vbswn@web9 | raise Exception(
matrix_synapse.1.wpn85n6vbswn@web9 | Exception: Error while initialising OIDC provider 'oidc-keycloak'
matrix_synapse.1.wpn85n6vbswn@web9 | Error during startup:
matrix_synapse.1.wpn85n6vbswn@web9 | Traceback (most recent call last):
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/handlers/oidc.py", line 119, in load_metadata
matrix_synapse.1.wpn85n6vbswn@web9 | await p.load_metadata()
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/handlers/oidc.py", line 436, in load_metadata
matrix_synapse.1.wpn85n6vbswn@web9 | return await self._provider_metadata.get()
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/util/caches/cached_call.py", line 136, in get
matrix_synapse.1.wpn85n6vbswn@web9 | return await self._cachedcall.get()
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/util/caches/cached_call.py", line 106, in get
matrix_synapse.1.wpn85n6vbswn@web9 | self._result.raiseException()
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/twisted/python/failure.py", line 475, in raiseException
matrix_synapse.1.wpn85n6vbswn@web9 | raise self.value.with_traceback(self.tb)
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
matrix_synapse.1.wpn85n6vbswn@web9 | result = current_context.run(
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
matrix_synapse.1.wpn85n6vbswn@web9 | return g.throw(self.type, self.value, self.tb)
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/util/caches/cached_call.py", line 126, in _wrapper
matrix_synapse.1.wpn85n6vbswn@web9 | return await f()
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/handlers/oidc.py", line 448, in _load_metadata
matrix_synapse.1.wpn85n6vbswn@web9 | metadata_response = await self._http_client.get_json(url)
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/http/client.py", line 591, in get_json
matrix_synapse.1.wpn85n6vbswn@web9 | body = await self.get_raw(uri, args, headers=actual_headers)
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/http/client.py", line 676, in get_raw
matrix_synapse.1.wpn85n6vbswn@web9 | response = await self.request("GET", uri, headers=Headers(actual_headers))
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/http/client.py", line 443, in request
matrix_synapse.1.wpn85n6vbswn@web9 | response = await make_deferred_yieldable(request_deferred)
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
matrix_synapse.1.wpn85n6vbswn@web9 | current.result = callback( # type: ignore[misc]
matrix_synapse.1.wpn85n6vbswn@web9 | File "/usr/local/lib/python3.9/site-packages/synapse/http/client.py", line 789, in _timeout_to_request_timed_out_error
matrix_synapse.1.wpn85n6vbswn@web9 | raise RequestTimedOutError("Timeout waiting for response from remote server")
matrix_synapse.1.wpn85n6vbswn@web9 | synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote serverI wonder if setting that environment var is somehow preventing openssl from using the system CA certs in /etc/ssl/certs. I don't think that should be the case from my reading of https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html ---but perhaps I'm mistaken.
Yes I can now confirm that, I removed OIDC config, removed SSL_CERT_FILE and created a local user, the SSL verification issue is gone.
I also think this not the normal behavior, SSL_CERT_FILE usage is documented here: https://github.com/matrix-org/synapse/blob/develop/docs/openid.md
MatthieuBarthel
commented
2 years ago I chained my CA with this list of certificates, which fixes the issue: https://raw.githubusercontent.com/certifi/python-certifi/master/certifi/cacert.pem
I still believe there is some bug because openssl verification and even requests from python were working in my tests with SSL_CERT_FILE set my CA.
DMRobertson
commented
2 years ago Can we double-check your homeserver.yaml config again? Do you have any of the following set?
MatthieuBarthel
commented
2 years ago No I have none of them set, here is my homeserver.yaml with comments/blank lines removed :
modules:
server_name: "imatt.ch"
pid_file: /data/homeserver.pid
public_baseurl: https://matrix.imatt.ch/
presence:
listeners:
- port: 8008
tls: false
type: http
x_forwarded: true
resources:
- names: [client, federation]
compress: false
manhole_settings:
limit_remote_rooms:
templates:
retention:
caches:
per_cache_factors:
database:
name: psycopg2
txn_limit: 10000
args:
host: postgres
user: synapse
password: xxx
database: synapse
cp_min: 5
cp_max: 10
log_config: "/data/imatt.ch.log.config"
media_store_path: "/data/media_store"
url_preview_accept_language:
oembed:
turn_uris: [ "turns:turn.imatt.ch?transport=udp", "turns:turn.imatt.ch?transport=tcp" ]
turn_shared_secret: "xxx"
turn_user_lifetime: 86400000
turn_allow_guests: True
registration_shared_secret: "xxx"
account_threepid_delegates:
metrics_flags:
report_stats: false
room_prejoin_state:
macaroon_secret_key: "xxx"
form_secret: "xxx"
signing_key_path: "/data/imatt.ch.signing.key"
old_signing_keys:
trusted_key_servers:
- server_name: "matrix.org"
saml2_config:
sp_config:
user_mapping_provider:
config:
oidc_providers:
- idp_id: keycloak
idp_name: "Keycloak Server"
issuer: "https://keycloak.imatt.ch/auth/realms/iMatt"
client_id: "synapse"
client_secret: "xxx"
scopes: ["openid", "profile"]
user_mapping_provider:
config:
localpart_template: "{{ user.preferred_username }}"
display_name_template: "{{ user.name }}"
cas_config:
sso:
password_config:
policy:
ui_auth:
email:
email:
smtp_host: smtp.example.com
smtp_port: 465
smtp_user: username
smtp_pass: xxxx
notif_from: "Matrix <xxx@imatt.ch>"
require_transport_security: true
push:
user_directory:
stats:
opentracing:
redis:Many thanks :)
DMRobertson
commented
2 years ago I still believe there is some bug because openssl verification and even requests from python were working in my tests with SSL_CERT_FILE set my CA.
Note that the requests module trusts the certificates in certifi. One has to set the environment variable REQUESTS_CA_BUNDLE to use a different set of CA certs.
DMRobertson
commented
2 years ago I'm not fully sure what's going on here. My reading of the OpenSSL docs and source code suggests that the SSL_CERT_FILE works in addition to the SSL_CERT_PATH.
Frustratingly, I can't see anything in Synapse or Twisted that would explain the behaviour you're seeing.
A few desperate suggestions:
/etc/ssl/certs and not set SSL_CERT_FILE. I'm not sure if that's easy to do with your Docker setup.SSL_CERT_PATH=/etc/ssl/certs in addition to SSL_CERT_FILE=/data/....MatthieuBarthel
commented
2 years ago Thanks for the suggestions, I tried to dig but without success unfortunately.
You could try to inject your certificate into the container in
/etc/ssl/certsand not set SSL_CERT_FILE. I'm not sure if that's easy to do with your Docker setup.
I did it by mounting my CA as a docker volume, it didn't work (keycloak SSL verification failed on start). Then I did it by copying my CA in a custom docker image (cloned the repo and made the change in docker/Dockerfile in both build stages before "pip install"), the result is the same. I also tried to build it with multiple python versions.
You could try explicitly setting
SSL_CERT_PATH=/etc/ssl/certsin addition toSSL_CERT_FILE=/data/....
I tried but it didn't change anything.
I made a few more tests from the container, with just SSL_CERT_FILE pointing to my CA :
root@35b0f204a09a:/# echo $SSL_CERT_FILE
/data/iMattRootCA.crtOpenssl seems to verify well both matrix.org and keycloak.imatt.ch (signed by my CA) :
root@35b0f204a09a:/# openssl s_client -showcerts -connect keycloak.imatt.ch:443
...
---
SSL handshake has read 16575 bytes and written 414 bytes
Verification: OK
---
root@35b0f204a09a:/# openssl s_client -showcerts -connect matrix.org:443
---
SSL handshake has read 5218 bytes and written 382 bytes
Verification: OK
---From a python console, only Matrix gets verified :
root@35b0f204a09a:/# python
Python 3.9.10 (main, Mar 2 2022, 04:31:58)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://matrix.org', verify=True)
<Response [200]>
>>> requests.get('https://keycloak.imatt.ch', verify=True)
...
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='keycloak.imatt.ch', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
# if I specify my CA path explicitly, only my domain gets verified :
>>> requests.get('https://keycloak.imatt.ch',verify='/data/iMattRootCA.crt')
<Response [200]>
>>> requests.get('https://matrix.org',verify='/data/iMattRootCA.crt')
...
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)With SSL_CERT_PATH=/etc/ssl/certs the results are the same.
I saw in the synapse source code that a library called grequests is also used in addition of requests, I tried to import it but it fails (ModuleNotFoundError), I don't know if it is possible to open an python console "scoped" to the synapse environment.
MatthieuBarthel
commented
2 years ago oops my bad, I forgot to run update-ca-certificates after copying the CA, it's working with the custom docker image
anoadragon453
commented
2 years ago It is possible to open a python interpreter inside a live, running Synapse process through use of Synapse's manhole, which is probably the most accurate environment. Do be careful if you're experimenting inside your production instance though :)
I saw in the synapse source code that a library called grequests is also used in addition of requests, I tried to import it but it fails (ModuleNotFoundError)
As far as I can see, this only used for a script (jitsimeetbridge.py) in the contrib directory. Synapse won't be using it for federation requests.
MatthieuBarthel
commented
2 years ago Thanks @anoadragon453, I restored my previous configuration (using SSL_CERT_FILE), so federation was broken again, opened an interpreter with manhole and I tried to request matrix.org and keycloak.imatt.ch, the results were the same that I had with the python interpreter (only matrix.org is verified), this is strange because in fact matrix boots fine with my custom certificate and fails to connect to matrix.org.
A simple way to fix the issue would be to update the doc to remove SSL_CERT_FILE usage, people would simply have to copy custom CA(s) into /usr/local/share/ca-certificates/, and to run update-ca-certificates before starting synapse.
In the docker image, it would need to be run in entrypoint.sh before the start command, it could be triggered only if a specific environment variable exists.
If this solution seems fine I believe I can submit a MR.
erikjohnston
commented
2 years ago So I think the issue here is related to Twisted doing something odd with the default trust stores: https://twistedmatrix.com/trac/ticket/9209#ticket. It notes that it should work fine if we install ca-certificates, which the docker image doesn't currently. @MatthieuBarthel could you try installing ca-certificates in the docker image and seeing if that fixes things? If it does then that feels like the "right" fix to me
MatthieuBarthel
commented
2 years ago I tried it, unfortunately it still gives the same result: my CA works but not the rest.
FYI, I've just set SSL_CERT_FILE and used this custom image:
FROM matrixdotorg/synapse:v1.57.0
RUN apt-get update && \
apt-get install -y ca-certificates && \
rm -rf /var/lib/apt/lists/*Do not hesitate if you want me to do others tests. Thanks :)
nakoo
commented
2 years ago I tried it, unfortunately it still gives the same result: my CA works but not the rest. FYI, I've just set
SSL_CERT_FILEand used this custom image:FROM matrixdotorg/synapse:v1.57.0 RUN apt-get update && \ apt-get install -y ca-certificates && \ rm -rf /var/lib/apt/lists/*Do not hesitate if you want me to do others tests. Thanks :)
I have similar issue with Dendrite (the new matrix server). My current guess is matrix doesn't work very well with docker overlay network. I haven't faced any issue when running bridge mode.
Description
I am using latest synapse docker image (v1.54), everything seems to work fine except I cannot list remote public rooms from app.element.io, it says "Failed to fetch room list" (all external homeserver fails: matrix.org, gitter.im, libera.chat). I am having this issue since I installed synapse (v1.52).
Federation seems ok: https://federationtester.matrix.org/#imatt.ch
In the synapse logs we can see it fails to verify the remote server SSL certificate (here matrix.org) :
The SSL verification seems fine from the docker host and the synapse container :
I believe the SSL verification issue is somewhere related to python in the docker image.
Steps to reproduce
My synapse installation is probably pretty common, I use keycloak as authentication provider with a self signed certificate but I don't see any relation.
I get the error from Element when I try to list public rooms :
Version information
If not matrix.org:
Version: Synapse 1.54 / Python 3.9.10
Install method: Docker
Platform: Docker Swarm on Ubuntu 20.04, Traefik as reverse proxy