Trying to comment in a video goes to "The page you are looking for isn't here" on e.g. mastodon.social

rptb1 commented 2 days ago

Describe the current behavior

Attempting to comment on a video reaches a 404 error "The page you are looking for isn't here" on mastodon.social, mastodon.online, etc.

Steps to reproduce

Click on a comment on https://tube.bogus.agency/w/iRQnAiash3LvJbPojtgdED
Enter a working handle. I used "@rptb1@mastodon.social". Click "Remote Interact".
Log in to the Mastodon site if necessary. It breaks whether you are logged in or not.
Observe 404 page at https://mastodon.online/authorize_interaction?uri=https%3A%2F%2Ftube.bogus.agency%2Fw%2FiRQnAiash3LvJbPojtgdED

Describe the expected behavior

Trying the same thing at e.g. https://v.basspistol.org/w/ddpeu5mmt6BzfDyYWGrEyC opens a draft remote comment. (That site is also PeerTube version 6.3.3.)

Additional information

This is a newly created instance on a fresh Ubuntu 24 LXD container, set up by exactly following the installation instructions, except that the reverse proxy is in another container. The reverse proxy has exactly the Nginx configuration file from the instructions, but with the "optimizations" section removed (since that requires local access to files). Otherwise I made no changes other than setting up minimum configuration.

PeerTube instance:
- URL: https://tube.bogus.agency/
- Version: 6.3.3
- NodeJS version: 23.1.0
- Ffmpeg version: 6.1.1-3ubuntu5
Browser name, version and platforms on which you could reproduce the bug: Firefox 132.0.1 on Ubuntu 22.04 with fresh blank Firefox profile. Also tried: LibreWolf 132.0.1-1, Fennec F-Droid on Android.
Link to browser console log if relevant:
Link to server log if relevant (journalctl or /var/www/peertube/storage/logs/):

Here are the logs generated from reproduction steps above.

{"tags":["http"],"level":"info","message":"fd42:10e8:9fef:5b94:d8cb::b657 - - [11/Nov/2024:10:28:12 +0000] \"GET /w/iRQnAiash3LvJbPojtgdED HTTP/1.0\" 200 4952 \"-\" \"Mastodon/4.4.0-nightly.2024-11-06 (http.rb/5.2.0; +https://mastodon.online/)\"","label":"tube.bogus.agency:443","timestamp":"2024-11-11T10:28:12.120Z"}
{"tags":["http"],"level":"info","message":"fd42:10e8:9fef:5b94:d8cb::b657 - - [11/Nov/2024:10:28:12 +0000] \"GET /videos/watch/90a09cc7-566a-4595-9b32-b971130ccdb9 HTTP/1.0\" 200 4952 \"-\" \"Mastodon/4.4.0-nightly.2024-11-06 (http.rb/5.2.0; +https://mastodon.online/)\"","label":"tube.bogus.agency:443","timestamp":"2024-11-11T10:28:12.295Z"}

Edit: I just noticed these logs are from trying @rptb1@mastodon.online, but the effect is the same in each case.

rptb1 commented 2 days ago

This looks like the same issue as #4799 but that was closed and never debugged, so here is a chance to fix it!

It might be a setup problem, in which case the installation instructions could be improved, or the log messaging. But I don't know what's happening yet.

rptb1 commented 2 days ago

I noticed that the log messages above appeared to be coming from my reverse proxy. In production.yaml the comment on trust_proxy says:

# If you run PeerTube behind a remote proxy, add the proxy IP address (or subnet)

And I added the external address of the proxy. But I realise this probably means the internal address that connects to the backend. So I added the whole lxdbr0 subnet like this:

  - '10.238.181.1/24'
  - 'fd42:10e8:9fef:5b94::1/64'

and now when I try the reproduction steps above, I get IP real IP addresses:

{"tags":["http"],"level":"info","message":"65.21.180.28 - - [11/Nov/2024:11:25:35 +0000] \"GET /w/iRQnAiash3LvJbPojtgdED HTTP/1.0\" 200 4952 \"-\" \"Mastodon/4.4.0-nightly.2024-11-06 (http.rb/5.2.0; +https://mastodon.online/)\"","label":"tube.bogus.agency:443","timestamp":"2024-11-11T11:25:35.538Z"}
{"tags":["http"],"level":"info","message":"65.21.180.28 - - [11/Nov/2024:11:25:35 +0000] \"GET /videos/watch/90a09cc7-566a-4595-9b32-b971130ccdb9 HTTP/1.0\" 200 4952 \"-\" \"Mastodon/4.4.0-nightly.2024-11-06 (http.rb/5.2.0; +https://mastodon.online/)\"","label":"tube.bogus.agency:443","timestamp":"2024-11-11T11:25:35.704Z"}

This has not immediately fixed the problem, but it might be related, if Mastodon is doing some sort of validation.

rptb1 commented 2 days ago

You can get mastodon.online etc. to attempt to fetch any URL by attempting to load the "authorize_interaction" URL. For example loading https://mastodon.online/authorize_interaction?uri=https%3A%2F%2Ftube.bogus.agency%2Fthis-is-a-test causes this log entry:

{"tags":["http"],"level":"info","message":"95.217.165.198 - - [11/Nov/2024:14:12:41 +0000] \"GET /this-is-a-test HTTP/1.0\" 406 - \"-\" \"Mastodon/4.4.0-nightly.2024-11-06 (http.rb/5.2.0; +https://mastodon.online/)\"","label":"tube.bogus.agency:443","timestamp":"2024-11-11T14:12:41.485Z"}

That suggests that Mastodon is not getting the a valid reply to this query.

The "federation" section of production.yaml has this comment:

  # Some federated software such as Mastodon may require an HTTP signature to access content
  sign_federated_fetches: true

Could the signatures be incorrect? How do I check?

rptb1 commented 2 days ago

Issue #692 may be related but:

I can find nothing different in the example working setup.
That issue is not specifically about federation, just making the site work. Mine works already. Mostly.

rptb1 commented 2 days ago

Just trying to track this down. You can hand-craft a URL to Mastodon's "authorize_interaction" and get the same results.

So the problem does not seem to be in the first few steps, but in whatever Mastodon is doing to validate my PeerTube with those GET requests.

Could the signatures be incorrect?

I can't see any evidence of header signing on any of the working URLs using curl -i.

rptb1 commented 1 day ago

So I set up my own Mastodon at https://mastodon.bogus.agency/ and tried commenting on my test video with @admin@mastodon.bogus.agency , and it worked as far as making a public reply https://mastodon.bogus.agency/@admin/113465899458798333 .

So perhaps there is an incompatibility with mastodon.online and mastodon.social . I will try to investigate.

Chocobozzz commented 1 day ago

Hi,

~~I can't access your instance. Is it down?~~

I managed to fetch your video using framapiaf.org. There may be a cache on mastodon.social, did you try using another video?

rptb1 commented 1 day ago

~I can't access your instance. Is it down?~

It might go up and down as I'm trying to debug transcoding. :)

I managed to fetch your video using framapiaf.org. There may be a cache on mastodon.social, did you try using another video?

Yes. This fresh one also won't go to mastodon.online https://tube.bogus.agency/w/iwdxA62535C6SHh3iRZmFQ .

Perhaps they have some sort of anti-spam measures that require a server to be up for a while. That might explain why #692 fixed itself mysteriously. And there might be nothing you can do. I'll experiment each day for a while.

Chocobozzz / PeerTube