search

matrix-org / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!
https://matrix-org.github.io/dendrite/
Apache License 2.0
5.67k stars 664 forks source link

Unable to join room due to invalid signature verification #1418

Closed llacqie closed 3 years ago

llacqie commented 4 years ago

Problems with the certificate of the home server of one of the participants in the room, does not allow you to join the room!

ERROR listing:

time="2020-09-10T11:15:21.069528376Z" level=error msg="Failed to fetch key for server" func="github.com/matrix-org/gomatrixserverlib.(*DirectKeyFetcher).FetchKeys.func1" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/keyring.go:482" context=missing error="Get matrix://netwhood.online/_matrix/key/v2/server: x509: certificate is valid for matrix.netwhood.online, not netwhood.online" fetcher=DirectKeyFetcher

llacqie commented 4 years ago

image

llacqie commented 4 years ago

When "disable_tls_validation: true"

ERRO[2020-09-10T11:54:37.138123421Z] [/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/keyring.go:482] func1 Failed to fetch key for server context=missing error="gomatrixserverlib: key response direct from \"netwhood.online\" failed checks" fetcher=DirectKeyFetcher

kegsay commented 4 years ago

disable_tls_validation just allows self-signed certificates, not incorrect domain names. This isn't a bug as far as I can tell, more of a misconfigured server. We can't really allow this check to be skipped otherwise anyone can pretend to be matrix.org for example (with a certificate signed for somewhere.com)

llacqie commented 4 years ago

How then can I configure the server to connect to rooms with similar participants? P.S. With synapse I can join these rooms without problems

kegsay commented 4 years ago

Have you tried setting matrix.org as a perspective server? https://github.com/matrix-org/dendrite/blob/master/dendrite-config.yaml#L264

llacqie commented 4 years ago

It seems to be set by default. If not, how do I set it?

kegsay commented 4 years ago

It is set by default, you need to make sure that this section is not commented out.

This really sounds like a misconfiguration though of netwhood.online - see https://federationtester.matrix.org/#netwhood.online

kegsay commented 4 years ago

Worth noting that https://federationtester.matrix.org/#matrix.netwhood.online works correctly.

neilalexander commented 4 years ago

Can you please tell us which commit you are running that reveals the join problem? #1407 was supposed to fix this.

llacqie commented 4 years ago

I didn't comment out this section when I set up the config, so everything should work, but it doesn't ... and it's not a problem for this particular room, I can't join most of the rooms

neilalexander commented 4 years ago

Need to know which commit you are running please, or at least if you built the server before this week.

If possible, please also make sure that you are running the latest master commit and see if you can reproduce the problem.

kegsay commented 4 years ago

@wirke we're aware of the problem and are working on a fix. Thanks for reporting it!

neilalexander commented 4 years ago

Can you please supply a full set of logs from between when you start the join up until you see the join error in your client? Do you see any other log lines that are level=error?

llacqie commented 4 years ago

logpart.txt

neilalexander commented 4 years ago

Thanks — it looks like the real issue is this:

time="2020-09-11T12:01:51.027673007Z" level=error msg="failed to join user \"@kitlhut0r:khtrx.tk\" to room \"!DMaQHGcdfYEnDLVaeM:matrix.org\" through 1 server(s): last error joinCtx.CheckSendJoinResponse: respSendJoin: gomatrixserverlib: missing auth event with ID $15963424341DaOlB:khtrx.tk for event $15964499083ECygk:khtrx.tk" func="github.com/matrix-org/dendrite/federationsender/internal.(*FederationSenderInternalAPI).PerformJoin" file="/root/dendrite/federationsender/internal/perform.go:105"

The event $15963424341DaOlB:khtrx.tk is marked as failed due to the fact that we couldn't validate the event with an existing key:

time="2020-09-11T12:01:50.795135236Z" level=error msg="Signature validation failed for event \"$15963424341DaOlB:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:a_veEX\""

... therefore it gets excluded from the auth chain/room state.

I wonder if this event is actually important to the join, either because it's a create/join rules/membership auth event, or if there's something else going on here.

You said that Synapse doesn't have any trouble joining the room?

neilalexander commented 4 years ago

Room ID in question is !DMaQHGcdfYEnDLVaeM:matrix.org, and I was able to join it successfully from dendrite.neilalexander.dev.

Have you checked that the time is synced correctly on your server with a reliable source? I wonder if this is a clock drift problem.

llacqie commented 4 years ago

With synapse running on this server, I successfully joined this room... Сorrect time(UTC) and date.

llacqie commented 4 years ago

Room's public address is #ru_distributed:matrix.org, but I also cant connect to general:mozilla.org. I think it's important that I use reverse proxy server(nginx) with letsencrypt certs in front of dendrite.

llacqie commented 4 years ago

Yes, from your server all works fine, how you configured server?

neilalexander commented 4 years ago

My server is a standard monolith + Postgres deployment, sitting behind Cloudflare and an nginx reverse proxy (configured with a Cloudflare origin TLS certificate). matrix.org is the only configured perspective key server.

Although I don't think incoming traffic is the problem necessarily here - the problem is seemingly with the ed25519:a_veEX key and why your server thinks it is invalid.

llacqie commented 4 years ago

I can attach an archive with the entire working directory of the server for further research, perhaps this will help you...

llacqie commented 4 years ago

I only deleted all .db files) https://github.com/wirke/dendrite_work_dir

llacqie commented 4 years ago

I understand that fixing this problem will take quite a long time, could you suggest a stable commit?!

llacqie commented 4 years ago

I want to start the server already...)

neilalexander commented 4 years ago 
time="2020-09-11T11:56:49.838026936Z" level=error msg="Signature validation failed for event \"$g6KLp6ahuWgchk3y:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:YDTq\""
time="2020-09-11T11:56:49.838775447Z" level=error msg="Signature validation failed for event \"$15913745480DGUzZ:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:a_HOdC\""
time="2020-09-11T11:56:49.838990141Z" level=error msg="Signature validation failed for event \"$15978624871LVDIz:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:a_hBut\""
time="2020-09-11T11:56:49.839469234Z" level=error msg="Signature validation failed for event \"$15978624120zcPXF:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:a_hBut\""
time="2020-09-11T11:56:49.839756271Z" level=error msg="Signature validation failed for event \"$8xdCUc42CJmr9dkI:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:YDTq\""
time="2020-09-11T11:56:49.840036489Z" level=error msg="Signature validation failed for event \"$15941370081hCrWR:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:a_IjKo\""
time="2020-09-11T11:56:49.840447675Z" level=error msg="Signature validation failed for event \"$lCk4zTo1tZKAEUMa:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:YDTq\""
time="2020-09-11T11:56:49.840578895Z" level=error msg="Signature validation failed for event \"$15913601570NKyfY:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:a_HOdC\""
time="2020-09-11T11:56:49.840699187Z" level=error msg="Signature validation failed for event \"$15964499083ECygk:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:a_veEX\""
time="2020-09-11T11:56:49.840814987Z" level=error msg="Signature validation failed for event \"$15963424341DaOlB:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:a_veEX\""
time="2020-09-11T11:56:49.840926721Z" level=error msg="Signature validation failed for event \"$Hr4y7vOAhgNurdTz:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:YDTq\""
time="2020-09-11T11:56:49.841047124Z" level=error msg="Signature validation failed for event \"$15940634282JLZxj:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:a_IjKo\""
time="2020-09-11T11:56:49.841159386Z" level=error msg="Signature validation failed for event \"$nnXt7eDHgT6B2tsh:khtrx.tk\"" func="github.com/matrix-org/gomatrixserverlib.(*RespState).Check" file="/root/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200907151926-38f437f2b2a6/federationtypes.go:422" error="Bad signature from \"khtrx.tk\" with ID \"ed25519:YDTq\""
time="2020-09-11T11:56:50.082045026Z" level=warning msg="Failed to join room through server" func="github.com/matrix-org/dendrite/federationsender/internal.(*FederationSenderInternalAPI).PerformJoin" file="/root/dendrite/federationsender/internal/perform.go:81" error="joinCtx.CheckSendJoinResponse: respSendJoin: gomatrixserverlib: missing auth event with ID $Hr4y7vOAhgNurdTz:khtrx.tk for event $g6KLp6ahuWgchk3y:khtrx.tk" room_id="!DMaQHGcdfYEnDLVaeM:matrix.org" server_name=matrix.org
time="2020-09-11T11:56:50.082485817Z" level=error msg="failed to join user \"@kitlhut0r:khtrx.tk\" to room \"!DMaQHGcdfYEnDLVaeM:matrix.org\" through 1 server(s): last error joinCtx.CheckSendJoinResponse: respSendJoin: gomatrixserverlib: missing auth event with ID $Hr4y7vOAhgNurdTz:khtrx.tk for event $g6KLp6ahuWgchk3y:khtrx.tk" func="github.com/matrix-org/dendrite/federationsender/internal.(*FederationSenderInternalAPI).PerformJoin" file="/root/dendrite/federationsender/internal/perform.go:105"

If @kitlhut0r:khtrx.tk is you, then it appears as if you have some problems with your keys.

Have you been reusing the same Matrix signing key (matrix_key.pem probably) from your previous installations, or have you been generating new keys?

Possibly related: matrix-org/synapse#4012.

llacqie commented 4 years ago

After the last commit, the user enters the room.

llacqie commented 4 years ago

However, this is most likely due to the fact that netwood.online is disabled.... I installed dendrite on a clean VPS server and can't connect to matrix:matrix.org...

llacqie commented 4 years ago

ERRO[2020-09-26T17:17:42.117626227Z] [/home/ubuntu/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200925165243-b9780a852681/keyring.go:497] func1 Failed to fetch key for server context=missing error="Post \"matrix://synapse.leprette.fr/_matrix/key/v2/query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" fetcher=DirectKeyFetcher

ERRO[2020-09-26T17:22:41.493510797Z] [/home/ubuntu/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200925165243-b9780a852681/keyring.go:497] func1 Failed to fetch key for server context=missing error="Post \"matrix://redpillrebel.com/_matrix/key/v2/query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" fetcher=DirectKeyFetcher

ERRO[2020-09-26T17:23:02.454563937Z] [/home/ubuntu/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200925165243-b9780a852681/keyring.go:497] func1 Failed to fetch key for server context=missing error="Post \"matrix://matrix.dapor.net/_matrix/key/v2/query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" fetcher=DirectKeyFetcher

ERRO[2020-09-26T17:23:01.164941438Z] [/home/ubuntu/go/pkg/mod/github.com/matrix-org/gomatrixserverlib@v0.0.0-20200925165243-b9780a852681/keyring.go:497] func1 Failed to fetch key for server context=missing error="Post \"matrix://parker.jetzt/_matrix/key/v2/query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" fetcher=DirectKeyFetcher

ptman commented 3 years ago

Dendrite seem to die in a panic for me with this:

time="2020-10-10T05:22:59.812947122Z" level=warning msg="Outgoing request failed" func="DoHTTPRequest\n\t" file=" [github.com/matrix-org/gomatrixserverlib@v0.0.0-20201006143701-222e7423a5e3/client.go:412]" context=missing error="Put matrix://kieg.science/_matrix/federation/v1/send/1602307379102-0: x509: certificate is valid for jitsi.kieg.science, matrix.kieg.science, not kieg.science" out.req.ID=CJRpThiya62f out.req.method=PUT out.        crypto/tls/conn.go:1255 +0x161
net/http.(*persistConn).Read(0xc0090f6360, 0xc008e42000, 0x1000, 0x1000, 0x40c69d, 0x60, 0x0)
        net/http/transport.go:1753 +0x75
bufio.(*Reader).fill(0xc011892b40)
        bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc011892b40, 0x1, 0xc00e2d9860, 0x0, 0x6d5e29, 0xc001e8cc98, 0x450b67)
        bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0090f6360)
        net/http/transport.go:1906 +0x1d6
created by net/http.(*Transport).dialConn
        net/http/transport.go:1575 +0xb0d

goroutine 2054410 [select, 14 minutes]:
github.com/matrix-org/dendrite/federationsender/statistics.(*ServerStatistics).Failure.func1(0xc00071cbe0)
        github.com/matrix-org/dendrite@/federationsender/statistics/statistics.go:130 +0x12b
created by github.com/matrix-org/dendrite/federationsender/statistics.(*ServerStatistics).Failure
        github.com/matrix-org/dendrite@/federationsender/statistics/statistics.go:127 +0x1bd

goroutine 2634733 [select]:
github.com/matrix-org/dendrite/federationsender/queue.(*destinationQueue).backgroundSend(0xc000da9680)
        github.com/matrix-org/dendrite@/federationsender/queue/destinationqueue.go:214 +0x190
created by github.com/matrix-org/dendrite/federationsender/queue.(*destinationQueue).wakeQueueIfNeeded
        github.com/matrix-org/dendrite@/federationsender/queue/destinationqueue.go:151 +0x64

goroutine 1793651 [select, 20 minutes]:
github.com/matrix-org/dendrite/federationsender/statistics.(*ServerStatistics).Failure.func1(0xc0007d6550)
        github.com/matrix-org/dendrite@/federationsender/statistics/statistics.go:130 +0x12b
created by github.com/matrix-org/dendrite/federationsender/statistics.(*ServerStatistics).Failure
        github.com/matrix-org/dendrite@/federationsender/statistics/statistics.go:127 +0x1bd

goroutine 1791696 [select, 20 minutes]:
github.com/matrix-org/dendrite/federationsender/statistics.(*ServerStatistics).Failure.func1(0xc0011ac8c0)
        github.com/matrix-org/dendrite@/federationsender/statistics/statistics.go:130 +0x12b
created by github.com/matrix-org/dendrite/federationsender/statistics.(*ServerStatistics).Failure
        github.com/matrix-org/dendrite@/federationsender/statistics/statistics.go:127 +0x1bd

goroutine 740846 [IO wait, 77 minutes]:
internal/poll.runtime_pollWait(0x7fc5df1a18e0, 0x72, 0xffffffffffffffff)
        runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc000f17798, 0x72, 0x1000, 0x1001, 0xffffffffffffffff)
        internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000f17780, 0xc002691300, 0x1001, 0x1001, 0x0, 0x0, 0x0)
        internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc000f17780, 0xc002691300, 0x1001, 0x1001, 0x203007, 0x0, 0xffc)
        net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc008eed388, 0xc002691300, 0x1001, 0x1001, 0x0, 0x0, 0x0)
        net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc01d394fa0, 0xc002691300, 0x1001, 0x1001, 0x3, 0x8, 0xc004fd78a0)
        crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc01cd9b758, 0x1290060, 0xc01d394fa0, 0x411be5, 0xf7e280, 0x1090ac0)
        bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc01cd9b500, 0x1290f80, 0xc008eed388, 0x5, 0xc008eed388, 0xff3ca0)
        crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc01cd9b500, 0x0, 0x0, 0x3)
        crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
        crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc01cd9b500, 0xc01b81a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        crypto/tls/conn.go:1255 +0x161
net/http.(*persistConn).Read(0xc001a81c20, 0xc01b81a000, 0x1000, 0x1000, 0xc01e97db60, 0xc004fd7c20, 0x40c765)
        net/http/transport.go:1753 +0x75
bufio.(*Reader).fill(0xc001fd3f20)
        bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc001fd3f20, 0x1, 0x0, 0x0, 0x1, 0xc01e97da00, 0x0)
        bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc001a81c20)
        net/http/transport.go:1906 +0x1d6
created by net/http.(*Transport).dialConn
        net/http/transport.go:1575 +0xb0d

goroutine 1788892 [select, 21 minutes]:
github.com/matrix-org/dendrite/federationsender/statistics.(*ServerStatistics).Failure.func1(0xc000726d70)
        github.com/matrix-org/dendrite@/federationsender/statistics/statistics.go:130 +0x12b
created by github.com/matrix-org/dendrite/federationsender/statistics.(*ServerStatistics).Failure
        github.com/matrix-org/dendrite@/federationsender/statistics/statistics.go:127 +0x1bd