[bug] Federation issue with custom API client

jorijn commented 1 week ago

Describe the bug with a clear and concise description of what the bug is.

I made custom API client code for a bot I'm writing. It's supposed to generate small toots about news articles, based off a RSS feed.

The GotoSocial account is fine, but once I start sending statuses, I'll see messages like this in the logs:

timestamp="08/11/2024 22:54:29.458" func=server.init.func1.Logger.13.1 level=INFO latency="5.949305ms" userAgent="Symfony HttpClient (Curl)" method=POST statusCode=200 path=/api/v1/statuses clientIP=REDACTED requestID=sj3wa3ck04001922kbn0 msg="OK: wrote 1.33kiB"
timestamp="08/11/2024 22:54:29.466" func=federation.(*federatingActor).Send level=INFO msg="send activity Create via outbox https://social.jorijn.com/users/loki/outbox"
timestamp="08/11/2024 22:54:29.466" func=workers.(*clientAPI).CreateStatus level=ERROR msg="error federating status: CreateStatus: error sending Create activity via outbox https://social.jorijn.com/users/loki/outbox:  https://social.jorijn.com/users/loki/followers resp was not ActivityPub media type: "

For full context, this is the HTTP request:

Request

* Host social.jorijn.com:443 was resolved.
* IPv6: REDACTED
* IPv4: REDACTED
*   Trying [REDACTED]:443...
* ALPN: curl offers h2,http/1.1
*  CAfile: /opt/homebrew/etc/openssl@3/cert.pem
*  CApath: none
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=social.jorijn.com
*  start date: Nov  7 07:56:19 2024 GMT
*  expire date: Feb  5 07:56:18 2025 GMT
*  subjectAltName: host "social.jorijn.com" matched cert's "social.jorijn.com"
*  issuer: C=US; O=Let's Encrypt; CN=E5
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to social.jorijn.com (2a02:a467:568b:0:9209:d0ff:fe1c:c38b) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://social.jorijn.com/api/v1/statuses
* [HTTP/2] [1] [:method: POST]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: social.jorijn.com]
* [HTTP/2] [1] [:path: /api/v1/statuses]
* [HTTP/2] [1] [authorization: Bearer REDACTED]
* [HTTP/2] [1] [content-type: application/json]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [user-agent: Symfony HttpClient (Curl)]
* [HTTP/2] [1] [accept-encoding: gzip]
* [HTTP/2] [1] [content-length: 533]
> POST /api/v1/statuses HTTP/2
Host: social.jorijn.com
Authorization: Bearer REDACTED
Content-Type: application/json
Accept: */*
User-Agent: Symfony HttpClient (Curl)
Accept-Encoding: gzip
Content-Length: 533

Content that's being sent (formatted):

Array
(
            [status] => 🐱 Breaking Mews: AT&T just spent $1B on spectrum licenses! Amateurs... We cats trade prime windowsill territories for just a few treats. Though I must say, their "C-band frequency" sounds purr-fect for broadcasting nap schedules. Maybe they consulted some feline experts? 😼 #itsmyparty

https://mobile.slashdot.org/story/24/11/07/2230235/us-cellular-to-sell-some-spectrum-licenses-to-att-for-1-billion
            [language] => en
            [media_ids] => Array
                (
                )

            [sensitive] => (false)
            [visibility] => public
)

Response

< HTTP/2 200
< server: nginx
< date: Fri, 08 Nov 2024 21:54:29 GMT
< content-type: application/json; charset=utf-8
< content-length: 1366
< cache-control: no-store
< content-encoding: gzip
< content-security-policy: default-src 'self'; object-src 'none'; img-src 'self' blob:; media-src 'self'
< permissions-policy: browsing-topics=()
< referrer-policy: same-origin
< vary: Accept-Encoding
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-ratelimit-limit: 300
< x-ratelimit-remaining: 259
< x-ratelimit-reset: 2024-11-08T21:56:53.000Z
< x-request-id: sj3wa3ck04001922kbn0
< x-robots-tag: noimageai
< x-robots-tag: noai
< strict-transport-security: max-age=15768000

The status is posted, and is visible on GotoSocial: https://social.jorijn.com/@loki/statuses/01JC6WB1YDSBAYTC5G5PC4P7FY

However, it won't federate. Update actions to the profile, Status deletes etc. all show the same error.

I haven't been able to inspect the /users/loki/followers path since it requires request signing.

Any help is appreciated.

What's your GoToSocial Version?

v0.17.3

GoToSocial Arch

amd64 Docker

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Anything else we need to know?

No response

tsmethurst commented 1 week ago

I've seen something similar to this before, but never got to the bottom of it fully. I'd say this is most likely an issue with configuration, networking, or with your split domain setup, not anything to do with the particular messages you're sending. Could you describe how you configured your split domain setup? Did you change the account domain or host or protocol values after already running for a little while by any chance?

jorijn commented 6 days ago

I have set up an entirely new test instance on test.jorijn.com, using the NGINX configuration provided in the documentation. The new test setup doesn't have a split domain setup.

The following logs appear when I send out a message using the client API.

timestamp="09/11/2024 08:07:09.829" func=workers.(*Processor).ProcessFromClientAPI level=INFO activityType=Create objectType=Note fromAccount=test msg="processing from client API"
timestamp="09/11/2024 08:07:09.832" func=server.init.func1.Logger.13.1 level=INFO latency="7.617851ms" userAgent="Symfony HttpClient (Curl)" method=POST statusCode=200 path=/api/v1/statuses clientIP=172.18.0.1 requestID=g22by3wk04001er5a7s0 msg="OK: wrote 1.10kiB"
timestamp="09/11/2024 08:07:09.847" func=federation.(*federatingActor).Send level=INFO msg="send activity Create via outbox https://test.jorijn.com/users/test/outbox"
timestamp="09/11/2024 08:07:09.849" func=workers.(*clientAPI).CreateStatus level=ERROR msg="error federating status: CreateStatus: error sending Create activity via outbox https://test.jorijn.com/users/test/outbox:  https://test.jorijn.com/users/test/followers resp was not ActivityPub media type: "

NGINX logging from that time:

REDACTED - - [09/Nov/2024:07:07:09 +0000] "POST /api/v1/statuses HTTP/2.0" 200 1129 "-" "Symfony HttpClient (Curl)"

To answer all the questions:

Could you describe how you configured your split domain setup?: The .well-known part is handled by an NGINX instance on jorijn.com, it permanently redirects to social.jorijn.com.
Did you change the account domain or host or protocol values after already running for a little while by any chance?: No

I'd be more than happy to submit one of these toots to any of your test instances if you provide me with an account, so we can rule out infrastructure.

jorijn commented 6 days ago

I have since been tinkering some more, and it started working all of a sudden. I don't have any real clue what changed. The setup didn't change, the content didn't change.. Sorry I can't be of more help. If you do need me to check on anything, let me know.

tsmethurst commented 6 days ago

Ah, so weird! I wish I knew where this bug came from 🤔 But nevermind, it's working now, investigation can come later. Is it alright with you if I rename the issue to more specifically address the "resp was not ActivityPub media type" error message?

jorijn commented 6 days ago

Yes, and feel free to close it. :)

superseriousbusiness / gotosocial