search

tgxn / lemmy-explorer

Instance and Community Explorer for Lemmy
https://lemmyverse.net/
115 stars 9 forks source link

Bug: Lemmy instances on 0.19.4 are no longer included #184

Closed freamon closed 5 months ago

freamon commented 5 months ago

Possible cause:

From what I can understand of this crawler's code, it looks for "http://nodeinfo.diaspora.software/ns/schema/2.0" in the response to /.well-known/nodeinfo

Lemmy 0.19.4 included a change to only provide "http://nodeinfo.diaspora.software/ns/schema/2.1".

Usually, fediverse platforms that provide 2.1 also provide 2.0, so it wouldn't be a problem, but they've apparently decided to make problems (including for themselves) by not doing so.

BentiGorlich commented 5 months ago

This is easily circumventable, as most software supporting the nodeinfo schema also supports the /.well-known/nodeinfo endpoint to indicate which urls have to be looked at

freamon commented 5 months ago

Just realised: there's already a PR at #183 for this.

tgxn commented 5 months ago

Sorry about this, I've been away for a little while. :) Let me get on top of this :D

tgxn commented 5 months ago

Merged the PR. Gotta deploy it our to the crawler infra, which is currently performing a backup, will probably take another hour. :)

tgxn commented 5 months ago

Ok, It's deployed and running. Should have an update in the next update in ~3-4 hours :)

image

tgxn commented 5 months ago

Hmm, it's scanned the backlog again but not making a huge dent in the un-scannable instances. Will have to investigate further.

tgxn commented 5 months ago

Ok, it's ripping through them now, should be good shortly :D

tgxn commented 5 months ago

And yeah it's pretty obvious 0.19.4 has broken it :D

https://lemmyverse.net/inspect/versions

tgxn commented 5 months ago

OK, I'm goign to close this. They have started rolling in, though may take another ~12 hours before they all flow through :)