[Bug]: Lemmy 0.18.0 no longer federates outbound with non-Lemmy instances

JupiterRowland commented 1 year ago

Requirements

[X] Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
[X] Did you check to see if this issue already exists?
[X] Is this only a single bug? Do not put multiple bugs in one issue.
[X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

Communities on Lemmy instances can no longer be found or followed from the Fediverse outside of Lemmy. Apparently, this started with 0.18.0.

It is no longer possible to find them by searching for them, it is no longer possible to subscribe to them, it is no longer possible to see posts from them. It is still possible to post to them from outside, though.

Instances I know to be hit by the bug

Steps to Reproduce

Search for communities

Example on Lemmy's side: the main community on sh.itjust.works which recently upgraded to 0.18.0, at which point outbound federation ceased to work. Other communities on sh.itjust.works behave the same. It has been reported that lemmy.ca has the same issue.

Example on the non-Lemmy side: Mastodon 4.1.0. The same thing has been proven to happen on Hubzilla 8.4.2, including with a community that has already been subscribed to about a week ago, prior to the Lemmy 0.18.0 upgrade on sh.itjust.works. Before Lemmy 0.18.0, searching for Lemmy communities worked on both Mastodon 4.1.0 and Hubzilla 8.4.2.

Copy the URL of a community on a Lemmy 0.18.0 instance; in this example, https://sh.itjust.works/c/main.
Go to any Mastodon instance. Being logged in is not required.
Paste the copied URL into the search field.
Hit Enter.

Expected result/result before the 0.18.0 upgrade: The main community on sh.itjust.works appears as the only search result. If logged into this Mastodon instance, the community can be subscribed to with one click.

Actual result: No search results; "Could not find anything for these search terms" instead.

Directly subscribe to communites without searching for them first

Disclaimer: This requires a Hubzilla channel and being logged into it.

The same Lemmy-side example as above will be used; the same explanations are still valid.

On the non-Lemmy side, the example is Hubzilla which does not require to search for a community to subscribe to it, just to illustrate that this doesn't work either.

Copy the URL of a community on a Lemmy 0.18.0 instance; in this example, https://sh.itjust.works/c/main.
Go to a Hubzilla instance on which you have your channel. Log in if necessary.
Navigate to the Connections page.
Click the green Add button.
In the field that says, "Enter channel address", paste the copied URL.
Click the green + button.

Expected result/actual result before the 0.18.0 upgrade: The connection is created. The connection configuring window opens so you can adjust it. The community appears amongst your connections.

Actual result: The page flicks briefly. The connection configuring window does not open. The community does not appear amongst your connections. The connection is not created.

Receive and read posts from subscribed communities

This can only be reproduced with communities which have been subscribed to prior to the 0.18.0 upgrade, but which run on 0.18.0 now. The requirement for a step-by-step reproduction would be to already have subscribed to a community which is on 0.18.0 now from an account/channel on an instance of a project that isn't Lemmy (e.g. Mastodon). For reasons mentioned above, this requirement cannot be established currently.

Technical Details

I'm not an admin, so no access to logs here, only external observations by a Hubzilla user.

Version

0.18.0

Lemmy Instance URL

No response

perillamint commented 1 year ago

wakest commented 1 year ago

I have been making some posts about this too, glad to see it was reported here. this is a pretty big deal and really needs to be solved!

It seems the federation is still working but whats not working anymore is the webfinger lookup...

itszednotzee commented 1 year ago

I think this issue affects kbin as well, posts/comments made by 0.18.0 users don't appear on kbin
lemmy comment chain same chain on kbin kbin
This is a big issue for lemmy as a large amount of users are federated from kbin as it is the largest lemmy-compatible instances right now (ignoring the enormous bot issue on lemmy anyway)

smorks commented 1 year ago

i made the changes to my nginx config based on the above linked lemmy-ansible issue, and tested with the curl commands listed, and it looks like it's now returning the correct json response. but it still doesn't appear to be working from mastodon.

i've looked at/tested some other places, such as friendica and kbin, and those appear to be working? at least for lemmy.ca?

@JupiterRowland are you able to test lemmy.ca with hubzilla to see if it's working now?

edit: this used to work fine with 0.17.4, and I'm using docker based installation with nginx not in docker, so it would seem to me that something changed in the lemmy backend to cause this.

JupiterRowland commented 1 year ago

@smorks At least community discovery still doesn't work on both Mastodon and Hubzilla.

Nutomic commented 1 year ago

It would be good if an admin of a Mastodon or Hubzilla instance could check the server logs at the time of fetching a Lemmy object, and see what kind of error is being logged. There have been some changes to federation, but I cant think of anything that would cause this type of breakage.

dfyx commented 1 year ago

I'm seeing something similar even between lemmy 0.18 instances:

my instance contains a comment I made
programming.dev is missing my comment
lemmy.fmhy.ml is missing my comment
lemmy.one is missing my comment

This seems to have happened to all my comments from the last few days, even before I updated to 0.18

Edit: reported as a separate bug because it's probably similar but not the same: https://github.com/LemmyNet/lemmy/issues/3380

seansica commented 1 year ago

It's quite possible this is user error, but I just spun up a new server, https://lemmynewengland.com, and it doesn't appear to be connecting to the federation at all. Federation is enabled of course. It was deployed via Ansible, running 0.18.0.

seansica commented 1 year ago

Here is what I am observing:

1. Other servers are not searchable

Federation is enabled. Both the allow-list and deny-list are left empty. Yet other servers are not searchable.

I tried explicitly adding lemmy.ml to the allow-list but that had no effect.

2. Visit other servers not working

Add lemmy.ml to the federation allow list
Navigate to Instances
Click lemmy.ml
Redirects to lemmy.ml successfully. All requests return 200.
The page on lemmy.ml displays a message:

Error! There was an error on the server. Try refreshing your browser. If that doesn't work, come back at a later time. If the problem persists, you can seek help in the Lemmy support community or Lemmy Matrix room.

Nothing happens when you click anywhere on lemmy.ml. It's not responsive.
Open an Incognito window or different browser and navigate to lemmy.ml. It works fine.

The only clue I have so far is that whenever I perform a search (any search) on my lemmy instance, the following warning is displayed in the lemmy_1 container:

2023-07-01T14:42:54.720841Z  WARN lemmy_server::root_span_builder: couldnt_find_object: invalid query
   0: lemmy_apub::fetcher::search::search_query_to_object_id
             at crates/apub/src/fetcher/search.rs:19
   1: lemmy_apub::api::resolve_object::perform
           with self=ResolveObject { q: "hello world", auth: Sensitive }
             at crates/apub/src/api/resolve_object.rs:21
   2: lemmy_server::root_span_builder::HTTP request
           with http.method=GET http.scheme="http" http.host=lemmynewengland.com http.target=/api/v3/resolve_object otel.kind="server" request_id=2fb9ff97-e5fd-4996-b914-9d789b8c5c48
             at src/root_span_builder.rs:16
LemmyError { message: Some("couldnt_find_object"), inner: invalid query, context: SpanTrace [{ target: "lemmy_apub::fetcher::search", name: "search_query_to_object_id", file: "crates/apub/src/fetcher/search.rs", line: 19 }, { target: "lemmy_apub::api::resolve_object", name: "perform", fields: "self=ResolveObject { q: \"hello world\", auth: Sensitive }", file: "crates/apub/src/api/resolve_object.rs", line: 21 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=GET http.scheme=\"http\" http.host=lemmynewengland.com http.target=/api/v3/resolve_object otel.kind=\"server\" request_id=2fb9ff97-e5fd-4996-b914-9d789b8c5c48", file: "src/root_span_builder.rs", line: 16 }] }

ross-spencer commented 1 year ago

I think this may be connected to some of what folks are saying above.

Trying to see a Lemmy post on Mastodon.social, or digipres.club, my Lemmy instance digipres.cafe is showing the following issue in the logs.

Federation issue.

Searching for https://digipres.cafe/post/8 via digipres.club:

lemmy_1     | 2023-07-02T14:21:29.262899Z  WARN lemmy_server::root_span_builder: Object not local, fetch it from original instance: Object not local, fetch it from original instance
lemmy_1     |    0: lemmy_apub::http::post::get_apub_post
lemmy_1     |              at crates/apub/src/http/post.rs:18
lemmy_1     |    1: lemmy_server::root_span_builder::HTTP request
lemmy_1     |            with http.method=GET http.scheme="http" http.host=digipres.cafe http.target=/post/8 otel.kind="server" request_id=baea9535-378b-47e9-ae50-b0f3264aa94d
lemmy_1     |              at src/root_span_builder.rs:16
lemmy_1     | LemmyError { message: Some("Object not local, fetch it from original instance"), inner: Object not local, fetch it from original instance, context: SpanTrace [{ target: "lemmy_apub::http::post", name: "get_apub_post", file: "crates/apub/src/http/post.rs", line: 18 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=GET http.scheme=\"http\" http.host=digipres.cafe http.target=/post/8 otel.kind=\"server\" request_id=baea9535-378b-47e9-ae50-b0f3264aa94d", file: "src/root_span_builder.rs", line: 16 }] }
lemmy_1     | 2023-07-02T14:21:29.771921Z  WARN lemmy_server::root_span_builder: Object not local, fetch it from original instance: Object not local, fetch it from original instance
lemmy_1     |    0: lemmy_apub::http::post::get_apub_post
lemmy_1     |              at crates/apub/src/http/post.rs:18
lemmy_1     |    1: lemmy_server::root_span_builder::HTTP request
lemmy_1     |            with http.method=GET http.scheme="http" http.host=digipres.cafe http.target=/post/8 otel.kind="server" request_id=901749a9-a8e5-4003-ac62-760ed3ccb537
lemmy_1     |              at src/root_span_builder.rs:16
lemmy_1     | LemmyError { message: Some("Object not local, fetch it from original instance"), inner: Object not local, fetch it from original instance, context: SpanTrace [{ target: "lemmy_apub::http::post", name: "get_apub_post", file: "crates/apub/src/http/post.rs", line: 18 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=GET http.scheme=\"http\" http.host=digipres.cafe http.target=/post/8 otel.kind=\"server\" request_id=901749a9-a8e5-4003-ac62-760ed3ccb537", file: "src/root_span_builder.rs", line: 16 }] }

The pertinent message here seems to be: "Object not local, fetch it from original instance"

Nb. Before upgrading to 0.18 via lemmy-ansuble this worked. Federation is on for the instance. I have already applied the fixes for: https://github.com/LemmyNet/lemmy-ansible/issues/106 and so this is the next bug on the list from the upgrade to solve.

smorks commented 1 year ago

this issue appears to be fixed in v0.18.1, at least for lemmy.ca

ghost commented 1 year ago

I can confirm this, federation with Mastodon and kbin seems to work again on 0.18.1 Please also make sure you updated your nginx config to fix issue https://github.com/LemmyNet/lemmy-ansible/issues/106

LemmyNet / lemmy