search

LemmyNet / lemmy

🐀 A link aggregator and forum for the fediverse
https://join-lemmy.org
GNU Affero General Public License v3.0
13.25k stars 879 forks source link

Some issues with federation #1416

Closed realkinetix closed 3 years ago

realkinetix commented 3 years ago

As per some of our Matrix chat, there seems to be some issues with federation recently. Things from lemmy.ca aren't making it to lemmy.ml:

Compare: https://lemmy.ml/community/18192 vs https://lemmy.ca/c/kamloopsbc or https://lemmy.ca/community/331 and https://lemmy.ml/c/economics

Tried responding to a comment from @dessalines at https://lemmy.ca/post/2131 and it didn't make it through.

Attaching requested files:

docker-compose.yml.txt lemmyca-config.hjson.txt

The docker-compose is stock, and unused on my instance.

dessalines commented 3 years ago 
  federation: {
    enabled: true
    allowed_instances: ""
    blocked_instances: ""
  }

This is incorrect, allowed_instances and blocked_instances should both be commented or absent (if you want open federation.) If you want explicit federation, only one of them are allowed.

dessalines commented 3 years ago

@Nutomic This page needs to be explicit about this: https://join.lemmy.ml/docs/en/federation/administration.html#instance-allowlist-and-blocklist , could you do a PR with that block in there.

Or at the very least link here: https://yerbamate.ml/LemmyNet/lemmy/src/branch/main/config/defaults.hjson#L62

edit: we also need to change that file, its incorrect. We should have them both commented by default, and explicitly say that only one is allowed at a time.

realkinetix commented 3 years ago

Huh... ok, I have commented both out and restarted lemmy backend. I only have backups back as far as Dec. 28 and it appears I've had the config set that way for some time (I don't recall when I had to rebuild my config.hjson). Did lemmy's federating logic around these settings change along the way here?

Also, it appears that federation still worked in one direction - if it's going to break, shouldn't it break in both directions?

Nutomic commented 3 years ago

@dessalines We explicitly allow all federation if both allowed_instances and blocked_instances are empty. And if you comment them out, you will get the default values which are also empty strings for each. So that wouldnt cause this problem.

@realkinetix Please check your nginx config as well, and make sure it is like ours. In particular these three if statements are important, as well as the backend location below.

If that doesnt help, check the Lemmy logs while you are commenting in a lemmy.ml community, and post them here.

Edit: we should add a field "installation method" to the issue template.

realkinetix commented 3 years ago

@Nutomic Thanks for the suggestions. I have checked out the nginx config - it's setup as per the stock nginx config, except I've used 127.0.0.1 instead of 0.0.0.0 for a couple of reasons. Also, lemmy.ca's nginx shouldn't be in the way of posts going from lemmy.ca to lemmy.ml, correct?

Lemmy is being very quiet - is there a config setting to increase logging verbosity?

Nutomic commented 3 years ago

The receiving instance has to be able to fetch users and communities from your instance, but I tested that manually with curl and it worked fine.

For more logs, set env var RUST_LOG=info or debug. But our logging is pretty terrible at the moment :(

realkinetix commented 3 years ago

Hey guys, it appears we're receiving 500 Internal Server Errors when trying to federate posts over - I did a test post in lemmy.ca's /c/kamloopsbc and this showed up in the logs:

Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::client::connect::http] connected to 5.196.14.162:443 
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z INFO  actix_web::middleware::logger] 127.0.0.1:54428 "POST /inbox HTTP/1.0" 200 0 "-" "-" 0.075396
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::proto::h1::io] flushed 722 bytes 
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::proto::h1::io] flushed 1601 bytes 
Feb 09 15:47:38 [pictrs] _[2mFeb 09 15:47:38.753_[0m _[32m INFO_[0m actix_web::middleware::logger: 127.0.0.1:51182 "GET /image/original/X3Y8NnIl1z.jpg HTTP/1.1" 200 30462 "https://lemmy.ca/" "pict-rs-frontend, v0.1.0" 0.001078    
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z INFO  actix_web::middleware::logger] 127.0.0.1:54430 "GET /pictrs/image/X3Y8NnIl1z.jpg HTTP/1.1" 200 30462 "https://lemmy.ca/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0" 0.002032
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::proto::h1::io] flushed 721 bytes 
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::proto::h1::io] flushed 959 bytes 
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::proto::h1::io] read 192 bytes 
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::proto::h1::io] parsed 5 headers 
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::proto::h1::conn] incoming body is content-length (17 bytes) 
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::proto::h1::conn] incoming body completed 
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG hyper::client::pool] pooling idle connection for ("https", lemmy.ml) 
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z DEBUG reqwest::async_impl::client] response '500 Internal Server Error' for https://lemmy.ml/inbox
Feb 09 15:47:38 [lemmy] [2021-02-09T15:47:38Z INFO  background_jobs_core::processor_map] Job 8e890802-4561-41ce-8f13-fe11e2ff41a6 SendActivityTask completed 0.617523
Feb 09 15:47:39 [lemmy] [2021-02-09T15:47:39Z DEBUG hyper::proto::h1::io] read 192 bytes 
Feb 09 15:47:39 [lemmy] [2021-02-09T15:47:39Z DEBUG hyper::proto::h1::io] parsed 5 headers 
Feb 09 15:47:39 [lemmy] [2021-02-09T15:47:39Z DEBUG hyper::proto::h1::conn] incoming body is content-length (17 bytes) 
Feb 09 15:47:39 [lemmy] [2021-02-09T15:47:39Z DEBUG hyper::proto::h1::conn] incoming body completed 
Feb 09 15:47:39 [lemmy] [2021-02-09T15:47:39Z DEBUG hyper::client::pool] pooling idle connection for ("https", lemmy.ml) 
Feb 09 15:47:39 [lemmy] [2021-02-09T15:47:39Z DEBUG reqwest::async_impl::client] response '500 Internal Server Error' for https://lemmy.ml/inbox
Feb 09 15:47:39 [lemmy] [2021-02-09T15:47:39Z INFO  background_jobs_core::processor_map] Job e6238d16-1e6d-4feb-a15c-db9d373117c8 SendActivityTask completed 0.730765
dessalines commented 3 years ago

Could you post your nginx here too?

realkinetix commented 3 years ago

Clock being off by 13 seconds on lemmy.ca seems to have been the issue. Time synced (and ntp'ed to prevent future skew), and federation seems to have been resolved.

Nutomic commented 3 years ago

I opened a docs issue about this: https://github.com/LemmyNet/lemmy-docs/issues/22