bug: Relays Broken

[moyitpro]

💡 Summary

I noticed in the latest release that relays are no longer working. Messages from server 1 (sharkeytest1.tamaki-shimai.moe) does not appear on server 2 (sharkeytest2.tamaki-shimai.moe), which is connected to the relay that is locally accessible only, relaytest.tamaki-shimai.moe. It seems the Sharkey isn't pulling new posts from the relay in the latest dev release compared to the stable release (see expected behavior)

Note: A Relay allows ActivityPub servers that are joined to it to exchange all posts between the servers that are in the relay

🥰 Expected Behavior

Server 1 messages should appear on Server 2, which is joined by the same relay and vice versa.

Pic in the screenshot below, which is from the publicly accessible test server, sharkey.tamaki-shimai.moe running the stable release of Sharkey, it's receiving posts I don't follow from the relay connected at relay.sakurajima.moe

🤬 Actual Behavior

I can reproduce this by creating two Sharkey servers in a local environment running on the latest Ubuntu 22.04.3 LTS release running on the dev branch. Messages made on server 1 isn't showing up on the federated timeline of server 2 and vice versa.

📝 Steps to Reproduce

1. Configure relays between two test servers (I used Activity-Relay, which is running on the first test Sharkey server at relaytest.tamaki-shimai.moe for a test)
2. Add the relay (e.g. https://relaytest.tamaki-shimai.moe/inbox) to both test servers. Refresh to make sure it says Accepted.
3. Type a public test message on the first and second test server.
4. Check the federated timelines to see if messages from both servers appear on the timeline without following each other (the relay receives posts from servers joined to the relay and sends it to all the servers joined in the relay).

💻 Frontend Environment

* Model and OS of the device(s): Windows 11
* Browser: Chromium
* Server URL: localhost (test server is not publicly accessible)
* Misskey: 2023.11.0.beta3

🛰 Backend Environment (for server admin)

* Installation Method or Hosting Service: shell script (test enviroment)
* Misskey:2023.11.0.beta3
* Node:v20.9.0
* PostgreSQL: 9.6
* Redis:6:7.2.3-1rl1~jammy1
* OS and Architecture: 22.04.3 LTS, x86-64

Relay shows both servers joined in the relay. sharkey@sharkeytest1:~/Activity-Relay$ ./relay control domain list WARN[0000] RELAY_ICON: INVALID OR EMPTY. THIS COLUMN IS DISABLED. WARN[0000] RELAY_IMAGE: INVALID OR EMPTY. THIS COLUMN IS DISABLED.

• Subscriber list : [] sharkeytest2.tamaki-shimai.moe [] sharkeytest1.tamaki-shimai.moe
• Follower list : Total : 2

[moyitpro]

I tried other Relay software in the local environment.

AodeRelay - Couldn't have the relay approve the request after adding it, don't think it's supported by Misskey Fedi.buzz - it does kind of work, but posts do not show up automatically, had to refresh to get them to show.

Activity-Relay is the only real one that had support for Misskey, which was working before the recent changes and I can get working and it was working with 2023.10.2.

Of course, it took while to make a response to testing other software as I am busy, but this should be enough information to investigate the issue.

[wont-work]

relay.fedi.buzz is working just fine* on my end, following 5 instances (2 of them being sharkey!) and 2 tags through it so far. not sure on aoderelay though, haven't joined any as to not bloat my instance with too many unnecessary notes

*: except the misskey-wide bug where everything it relays shows up as renotes

[Mar0xy]

For my instance which I setup a few days ago relay.fedi.buzz and rel.re(running aoderelay v0.3.104-HEAD-a0f9827e) work just fine.

[moyitpro]

@Mar0xy Thanks, Maybe I should look into switch relay software when I have the time to. The private relay I set up was there to join the Mastodon server I run with the Firefish server, and we also had one other Mastodon instance whom I am friends with the admin. The relay is private so only those I approve can join it.

Then again, I'm not sure if it's a bug with Activity-Relay as it was running on an older version, but it looks like the previous releases only updated the dependencies.

[Mar0xy]

Just tried out activity-relay and for some reason it only proceeds this far before not doing anything

[Mar0xy]

Just tried it on a local environment and it resulted in a warn log about a bad request So there must be something going on in the background that causes it to mess up though this could be due to running it locally with no domain

[Mar0xy]

The issue with activity-relay seems like after having the follow request submitted successfully from sharkey it just proceeds to ignore it and and doesn't accept the follow resulting in sharkey also not getting updates from it nor being able to set it to approved so either a misskey update broke compatibility with it or an recent update to activity-relay broke it.

AodeRelay still worked fine after all my attempts on a dev instance.

I will leave this open for now so others can see it and chime in with more information if they got any

[moyitpro]

@Mar0xy I tried installing aoderelay (nrelay.sakurajima.moe) and I can't seem to get it accepted by the relay. Am I even using the right version, I'm using the one from the official repo.

Can you provide the .env you use to set it up. Might be something with my relay config.

[Mar0xy]

they show up as connected though

[moyitpro]

Yes, it's strange since none of the messages are getting passed through the sharkey instance.

Here is the .env file, I have the relay behind a proxy on a web server.

HOSTNAME=nrelay.sakurajima.moe PORT=8082 HTTPS=false DEBUG=false RESTRICTED_MODE=true VALIDATE_SIGNATURES=true API_TOKEN=(redacted) FOOTER_BLURB="Sakurajima Relay" LOCAL_DOMAINS="sakurajima.moe" LOCAL_BLURB="" PROMETHEUS_ADDR=127.0.0.1 PROMETHEUS_PORT=9000

[Mar0xy]

I never really hosted a relay as I was always just looking at the output of the server running sharkey to see if it delivered the message and received an accepted message since nrelay.sakurajima.moe is restricted I am also not able to check it on my end with my rant.lol instance to see what happens exactly.

[moyitpro]

@Mar0xy I temporarily added this instance so you can test. Remove it after getting the output.

[Mar0xy]

Alright thanks I have now gotten my answer

[Insert5StarName]

Alright thanks I have now gotten my answer

Didn't we have a similar issue with relays before?

[Mar0xy]

Alright thanks I have now gotten my answer

Didn't we have a similar issue with relays before?

Not quite what is going on here is the fact that while the relay redirects to https for some reason the actual actor URL is not https causing it to get confused

[Mar0xy]

how it is suppose to be:

How it actually is:

[Mar0xy]

So I wonder if simpy toggling HTTPS in the env of the relay and readding the relay fixes it

[moyitpro]

Yes, that did the trick, but the instance images on the relay page and the relay command is broken when HTTPS is turned on with aoderelay, it takes the hostname of the VPS instead of the one from the .env file. Might need to change the env file of the user that runs the relay.

What I also notice is that the timeline doesn't automatically refresh when it gets new posts from a relay, and you have to refresh it. Is that normal behavior?

[Mar0xy]

Is that normal behavior?

It is suppose to refresh unless the websocket disconnected or fan out timeline is disabled(which is a MK bug afaik)

[Mar0xy]

I will close this as it mainly turned out to just be an issue related to configuration of the relay. The HTTPS redirection not being accepted when resolving actors for relays is a good thing in itself as HTTP to HTTPS redirections can include MITM attack if not properly handled by using HSTS.

And if the relay already redirects HTTP to HTTPS then the relay info itself should at the very least be set to HTTPS