Bug: Instance is not listed on lemmyverse.net

tgxn / lemmy-explorer

Instance and Community Explorer for Lemmy

https://lemmyverse.net/

105 stars 9 forks source link

Bug: Instance is not listed on lemmyverse.net #130

Closed Security-Chief-Odo closed 12 months ago

Security-Chief-Odo commented 1 year ago

Describe the bug Beehaw.org is not listed on https://lemmyverse.net/

To Reproduce Go to https://lemmyverse.net/ Search for 'beehaw.org' Get nothing found

Expected behavior Go to https://lemmyverse.net/ Search for 'beehaw.org' Beehaw.org is shown

Additional context Would like to determine why this is the case. I am a systems administrator at beehaw.org, and when I search our logs, I see successful HTTP 200's for the requested endpoints /api/v3/community/list(params), /.well-known/nodeinfo, /nodeinfo/2.0.json and /api/v3/site for the user-agent lemmy-explorer-crawler/**

If it has been successful, I'd like to understand why beehaw.org is not listed on lemmyverse.net as expected.

tgxn commented 12 months ago

Hmm, yeah there's something up, there's a bug that's not setting the redis ttl properly -.-

So the error that I got 5 days ago "Failed to get url https://beehaw.org/api/v3/site after 4 attempts: Request failed with status code 502", is still stopping the crawler from trying agian.

I've manually expired and fixed the ttl's so it'll be scanned shortly. I'll see what's going on with these TTL's since they should be getting set when the error is written to redis.

more info error distribution, should be no errors older than ~8 hours