[bug] can't follow mastodon accounts: Unauthorized

[igalic]

trying to follow @meena@glitch.social from @me@soc.eena.me:

Apr 14 20:02:20 social gotosocial[28943]: time=2022-04-14T20:02:20Z level=info msg=authentication not passed for public key owner https://glitch.social/actor; signature value was 'keyId="https://glitch.social/actor#main-key",algorithm="rsa-sha256",headers="(request-target) host date accept",signature="I3v2a1AuO0hVB3fib83VZARhUL3BO9HXbdvtHaq8L2vqu5gYWUy1z64Y7JNJ2CX1WQauZ+J+49fx3eK5BnU3O+bEJo1B6r4VDW7mLVzRemzHvQOLMgySLXtncn16P2PriimqQ1Ejn/q1B8O//POcs0n7Ot7z2pc4Uu3BMTY9eek7u6zv1hfXEoMavROwdyvJ2vBTxc6+1If185OVIJeVkOMivtufYRNkl1w29EaQVVsB0aRvL+oHm16+a5cCHV7MHnuEtyvOk0Ognt1n4/LtRPfp/VKpDKznzqmrGRA45UdzQncB8DpbB6OEQtl+Cd+kigbIQ6Ja+9NzAZyFR86Abw=="' func=AuthenticateFederatedRequest
Apr 14 20:02:20 social gotosocial[28943]: time=2022-04-14T20:02:20Z level=info msg=not authorized func=UsersGETHandler url=/users/me
Apr 14 20:02:20 social gotosocial[28943]: time=2022-04-14T20:02:20Z level=info msg=[8.08787ms] Unauthorized: wrote 54 bytes clientIP=192.168.17.10 latency=8.08787ms method=GET path=/users/me statusCode=401 userAgent=http.rb/4.4.1 (Mastodon/3.4.6; +https://glitch.social/)
Apr 14 20:02:20 websrv2-hel1 /usr/local/bin/gotosocial[28944]: time=2022-04-14T20:02:20Z level=error msg=batch deliver had at least one failure: POST request to https://glitch.social/users/meena/inbox failed (401): 401 Unauthorized 
Apr 14 20:02:20 social gotosocial[28943]: time=2022-04-14T20:02:20Z level=error msg=batch deliver had at least one failure: POST request to https://glitch.social/users/meena/inbox failed (401): 401 Unauthorized

similar failures occur when i try to follow an account on a newer codebase, with the federation fixes:

Apr 14 21:09:58 social gotosocial[28943]: time=2022-04-14T21:09:58Z level=debug msg=entering NewID func=NewID newID={"@context":"https://www.w3.org/ns/activitystreams","actor":"https://soc.eena.me/users/me","id":"https://soc.eena.me/users/me/follow/012X9043BMGPWHXEY6W3QZ6VWR","object":"https://cathode.church/users/meena","to":"https://cathode.church/users/meena","type":"Follow"}
Apr 14 21:09:58 social gotosocial[28943]: time=2022-04-14T21:09:58Z level=debug msg=entering Create create={"@context":"https://www.w3.org/ns/activitystreams","actor":"https://soc.eena.me/users/me","id":"https://soc.eena.me/users/me/follow/012X9043BMGPWHXEY6W3QZ6VWR","object":"https://cathode.church/users/meena","to":"https://cathode.church/users/meena","type":"Follow"} func=Create
Apr 14 21:09:58 social gotosocial[28943]: time=2022-04-14T21:09:58Z level=debug msg=entering Get func=Get id=https://soc.eena.me/users/me
Apr 14 21:09:58 social gotosocial[28943]: time=2022-04-14T21:09:58Z level=info msg=[22.09331ms] OK: wrote 176 bytes clientIP=192.168.17.10 latency=22.09331ms method=POST path=/api/v1/accounts/01FXX82TB2C3H026FKG18T9ZBY/follow statusCode=200 userAgent=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0
Apr 14 21:09:59 social gotosocial[28943]: time=2022-04-14T21:09:59Z level=info msg=[2.991909ms] OK: wrote 554 bytes clientIP=192.168.17.10 latency=2.991909ms method=GET path=/users/me/main-key statusCode=200 userAgent=http.rb/5.0.4 (Mastodon/3.5.1+glitch+cathode; +https://cathode.church/)
Apr 14 21:10:00 social gotosocial[28943]: time=2022-04-14T21:10:00Z level=debug msg=performing GET to https://cathode.church/actor#main-key func=Dereference
Apr 14 21:10:00 social gotosocial[28943]: time=2022-04-14T21:10:00Z level=info msg=authentication not passed for public key owner https://cathode.church/actor; signature value was 'keyId="https://cathode.church/actor#main-key",algorithm="rsa-sha256",headers="(request-target) host date accept",signature="TnNY5jJVteujED5P5Zy7oNvJUeiKqahpM5+B+3rnLSWQEzSwR+zh5asanc3oSWqtYcHMfUa38PVeYS4LnEKpEXvarYYOyFMQ2sEaJfMevODyCOv8+Z0py2e2USdu3gf+lT808mAESaE2OLIK5lG/EH19Cp4wJVWK60/P+p2TzCgYmCBW4hd78PXCNIvyg1LDAiCqpH90YQSFVM7nbzdwI8w2T62ckLX9jCk5zgvcFikD5F3OPl3ELo2e9fZCkOHZ580NJXy2d4G9J2cDNWBnu9AQFFWBI5UAUBqWVkSfG/YyZs0qN9vyANmlFpqMcUDnvR+XsZ1Ir4oLhdjBm2a/mg=="' func=AuthenticateFederatedRequest
Apr 14 21:10:00 social gotosocial[28943]: time=2022-04-14T21:10:00Z level=info msg=not authorized func=UsersGETHandler url=/users/me
Apr 14 21:10:00 social gotosocial[28943]: time=2022-04-14T21:10:00Z level=info msg=[287.51058ms] Unauthorized: wrote 54 bytes clientIP=192.168.17.10 latency=287.51058ms method=GET path=/users/me statusCode=401 userAgent=http.rb/5.0.4 (Mastodon/3.5.1+glitch+cathode; +https://cathode.church/)
Apr 14 21:10:01 websrv2-hel1 /usr/local/bin/gotosocial[28944]: time=2022-04-14T21:10:01Z level=error msg=batch deliver had at least one failure: POST request to https://cathode.church/users/meena/inbox failed (401): 401 Unauthorized 
Apr 14 21:10:01 social gotosocial[28943]: time=2022-04-14T21:10:01Z level=error msg=batch deliver had at least one failure: POST request to https://cathode.church/users/meena/inbox failed (401): 401 Unauthorized

here's my httpd.conf

```apache MDomain soc.eena.me auto ServerName soc.eena.me DocumentRoot "/var/empty" SSLEngine On ServerName soc.eena.me RewriteEngine on RewriteCond %{HTTP:Connection} Upgrade [NC] RewriteCond %{HTTP:Upgrade} websocket [NC] RewriteRule / wss://social:8000//$1 [P,L] ProxyPassReverse / wss://social:8000/ ProxyPass / http://social:8080/ ProxyPassReverse / http://social:8080/ RequestHeader set "X-Forwarded-Proto" expr=https ```

and my gotosocial config

```yaml log-level: "debug" application-name: "Meena's Social" host: "soc.eena.me" account-domain: "soc.eena.me" protocol: "https" bind-address: "[::]" port: 8080 trusted-proxies: - "192.162.17.1/24" db-type: "sqlite" db-address: "/var/db/gotosocial/db/sqlite.db" web-template-base-dir: "/usr/local/www/gotosocial/template/" web-asset-base-dir: "/usr/local/www/gotosocial/assets/" accounts-registration-open: false accounts-approval-required: true accounts-reason-required: true media-image-max-size: 2097152 media-video-max-size: 10485760 media-description-min-chars: 0 media-description-max-chars: 500 storage-backend: "local" storage-local-base-path: "/var/db/gotosocial/storage" statuses-max-chars: 5000 statuses-cw-max-chars: 100 statuses-poll-max-options: 6 statuses-poll-option-max-chars: 50 statuses-media-max-files: 6 letsencrypt-enabled: false oidc-enabled: false smtp-host: "" syslog-enabled: true syslog-protocol: "" syslog-address: "" ```

my gotosocial version is: gotosocial version 0.2.3 d350087 2022-04-13T20:40:15Z [go1.18] (in fact it's https://github.com/superseriousbusiness/gotosocial/pull/449) running in 13.0-RELEASE-p11 on amd64

[tsmethurst]

Alright, I'm playing around with this a bit... it looks like I can't dereference your account from GoToSocial either, which suggests something is up with your instance but I'm really not sure what :thinking: I'll keep peeking around and see if I can figure it out

[edit] would it be possible to enable trace logging and try to just capture the logs from during the request/http signature handshake? trace logging is very verbose and hard to read but i'm used to parsing it by now and it might tell us something useful. As far as I can tell from here, your instance doesn't seem to think any http signatures are valid, so I'm curious why that is

[igalic]

will do!

[tsmethurst]

Thanks! :) I very much wanna get this sorted

[igalic]

here we go: https://gist.github.com/65343e512bbc12628e6f169859bbc04d

[tsmethurst]

Thanks! This helps narrow it down :) I'll keep looking, I have some ideas

[tsmethurst]

Huh... my intuition was wrong... I tried to narrow it down by creating a new account on testingtesting123.xyz with the same GtS version, but I can dereference that one just fine :thinking:

[igalic]

could something be wrong with my Proxy?

[tsmethurst]

could something be wrong with my Proxy?

i guess.... the settings look fine to me but I don't know httpd

could it be the case that when a remote instance does a request to https://soc.meena.me/users/me, the url gets rewritten to something else from GoToSocial's perspective? because that might explain why the signature validation fails (it validates on request-target)

[igalic]

the request is, or should be for: https://soc.eena.me/users/me and httpd.conf and gotosocial.conf agree in that regard, as far as i can read

[tsmethurst]

perhaps it's worth running it without httpd but using nginx or just running it raw on 443 and 80 with letsencrypt enabled, and then we can try and narrow it down to see if it's a proxy issue? i'm also going to look at golang's crypto libraries and see how it decides on valid rsa256 etc

[igalic]

i have no idea how to configure nginx. plus, nginx doesn't have a mod_md aequivalent. so aside from having to learn how to configure a new web server, that doesn't speak proper HTTP half the time, I'd also have to find a new way to configure LetsEncrypt.

this server is currently serving:

• the lounge: https://im.eena.me/
• hededoc: https://scratchpad.pkgbase.live/
• and my (static) website: https://igalic.co/

the proxy configuration is mostly taken from The Lounge Docs. Does gotosocial have a WSS component? because it certainly looks like pinafore is trying to connect to one.

hmmm… i'm also running all of this behind mod_security — i wonder if that's doing anything?!

[igalic]

hmmm… i'm also running all of this behind mod_security — i wonder if that's doing anything?!

taken mod_security out of the equation and i'm still getting the exact same failures. but i just realized, you're setting 'Host' in your [nginx config]():

    proxy_set_header Host $host;

That means i need ProxyPreserveHost.

Let's try.

[igalic]

yupp, this seems to have done the trick:

<VirtualHost *:80>
  ServerName soc.eena.me
  DocumentRoot "/var/empty"
</VirtualHost>

<VirtualHost *:443>
  SSLEngine On
  ServerName soc.eena.me
  ProxyPreserveHost On
  RewriteEngine on
  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule / wss://social:8000/$1 [P,L]
  ProxyPassReverse / wss://social:8000/
  ProxyPass / http://social:8080/
  ProxyPassReverse / http://social:8080/
  RequestHeader set "X-Forwarded-Proto" expr=https

</VirtualHost>

of course firefox is still saying that pinafore connect to wss:

Firefox can’t establish a connection to the server at wss://soc.eena.me/api/v1/streaming?stream=user&access_token=YJG3ZMM4Y2ETNTHHNC0ZNZG5LTHKNDCTY2ZJZMJMYMQZZMVL.

but it's a start.!

[igalic]

I'd amend the documentation to say something like:

Note: proxy_set_header Host $host; is essential: It guarantees that the proxy and the gotosocial speak of the same Server name. If not, gotosocial will build the wrong authentication headers, and all attempts at federation will be rejected with 401.

[phikal]

Hi,

despite explicitly adding the proxy_set_header Host $host; option, I appear to have the same issue:

timestamp="14/11/2022 12:42:07.588" func=concurrency.(*WorkerPool).Queue.func1 level=ERROR type=worker.Worker[messages.FromClientAPI] error="BatchDeliver: at least one failure: POST request to https://fosstodon.org/inbox failed (401): 401 Unauthorized" msg="message processing error"

Has something changed since this issue?

[tsmethurst]

That means you're getting a 401 returned back to your instance from fosstodon, not that your instance is giving fosstodon a 401.

[phikal]

Oh, right. Is this a known issue or should I open a new one?

[tsmethurst]

Is this a known issue or should I open a new one?

If you see it occasionally for various instances when your instance is under load (say you just got boosted or mentioned by a big account) then it's sort-of known. Otherwise, probably something else is going wrong, but just one 401 isn't worth opening an issue for :P

[phikal]

tobi @.***> writes:

Is this a known issue or should I open a new one?

If you see it occasionally for various instances when your instance is under load (say you just got boosted or mentioned by a big account) then it's sort-of known. Otherwise, probably something else is going wrong, but just one 401 isn't worth opening an issue for :P

No, this isn't just one 401. Any and every instance I try to connect to gives me 401. Interestingly, search works and I can find accounts. Some have profile pictures, others don't. But all feeds are empty, I can only request to follow and I don't get any other information like follower lists. The instance is certainly not under load, because nobody knows about it.

[phikal]

It seems that #974 better describes the issue.