[0.5.0]Search across all instances

[krestenlaust]

Is your feature request related to a problem? Please describe. When an instance has been selected, I can't remove the selection

Describe the solution you'd like I'd like to be able to search across all instances

[marsara9]

Sorry can you elaborate more on this one? I'm not sure I understand what the ask / issue is.

[krestenlaust]

Reproduction:

1. Open search-lemmy.com
2. Search something, I'll search "denmark"
3. I get this query: "https://search-lemmy.com/results?query=denmark&preferred_instance=lemmy.towards.vision&page=1", that only searches on the instance "lemmy.towards.vision"

I would like the search to be across all instances, not just a random instance.

[marsara9]

There's a drop-down where you can select your preferred-instance or "home" instance. Search results are already filtered by that instance as not all content is federated to all instances.

For example, this post: https://lemmy.ml/post/1694451 does not exist on lemmy.world yet, but it is on lemmy.ml. There would be no way to open that link in lemmy.world as it doesn't exist.

There exists an issue on Lemmy itself though: https://github.com/LemmyNet/lemmy/issues/3259. Once the Lemmy devs decide on if and how they want to support this, then there's a lot of improvements I can make, mainly:

1. I will no longer need to index EVERY lemmy instance. Currently I gather post data from the "seed instance" and then lookup all of those actor_ids on every instance that federates with that seed_instance. This way I can determine the internal post-id that has to be used to provide a link.
2. I currently have to filter the search results by your preferred / home instance, as I must be able to return the internal post-id back to the browser to create that link. If these ids become universal, then I'll no longer need to do this and I can even save an entire JOIN in the search query.

I am looking at fixing the drop-down to make it easier to select your home instance, but until that linked issue is resolved, I don't have anyway to let users search EVERYTHING.

[marsara9]

Adding a possible duplicates of the prior mentioned issue:
https://github.com/LemmyNet/lemmy/issues/1101 https://github.com/LemmyNet/lemmy/issues/2987

I'm going to leave this issue open until I see clear direction from the Lemmy devs on how they plan on addressing the post-id issue, if at all. Once that happens, then I'll either being work on removing the extra crawling step as outlined above or closing this as-is, if Lemmy chooses not to provide universal ids.

[krestenlaust]

I'm not sure if we understand each other. Isn't this supposed to be the lemmy alternative to adding "reddit" to your search in google? If I just wanted to search a single instance, I would just use the search option on the instance, or add "instance-name" or "site:instance-url" to my search on google.

Can't you simply aggregate search results for every instance, and then eliminate duplicates?

I've only read a little about activitypub, so I'm not sure how you'd identify duplicates, but in any case, eliminating duplicates by title would probably be fine in most cases.

🤷‍♀️

[phoenixeliot]

I would love to see search results across all instances regardless of where I can open it, and open it in the source instance or a preferred instance, whatever is available. I care more about finding it than being able to open it in a specific instance.

Right now, it seems you can basically only search one instance at a time for the most part unless federation has gotten lucky and sucked up the post you're looking for into the instance you've selected.

For example, this search only finds results if I provide lemmy.world, and not any of the other instances I've tried: https://www.search-lemmy.com/results?query=estradiol&preferred_instance=lemmy.world&page=1

[phoenixeliot]

If duplicate results are a big problem for things that have been widely federated and it's hard to de-dupe them, for now it might also be sufficient to just give an option for "search all instances" instead of searching just one, so the user can choose.

[hhu94]

I agree with @krestenlaust and @phoenixeliot. I am looking for a way to search through posts and comments in every existing Lemmy instance, not my home instance or any other particular single instance.

Essentially, I'm looking for an alternative to searching "search text site:reddit.com" on google.

If this use case is not supported, I don't understand what the value of the project is, since you can already search instances (and their corresponding federated instances) through the lemmy web UI's search function.

Of particular note is that I don't care which instance the search result links to. I don't even need results to link to my home instance. The user will always be able to navigate to that search result in their own instance if they want to.

[marsara9]

Ideally I'd love to just let you search everything, but the linked issues from Leemy itself are preventing me. If you're OK with dead links, then I can easily just let you search the entire fediverse, but if I do that I cannot guarantee that the links provided will work.

As for how this is different than using the site's search itself... Lemmy does an implicit AND between all of your search terms, essentially "Foo Bar Baz" (were * can be anything) and it doesn't appear to do any sort of ranking, whereas this search engine 1) does an implicit OR and 2) it tries to rank the search results based on the score and the the number of matched terms.

I could look at having an option to disable the automatic link redirection. And just have everything open on either the 'seed instance' or the 'source instance', but how to present this option to the UI?

Technically speaking you're only searching lemmy.ml right now anyway, the results are just filtered down by what posts with the same actor_id can be found on other instances that federate with lemmy.ml. On those other instances I don't do anything more than lookup what the internal-id is and match it up with the actor_id I found from the seed instance (lemmy.ml). If the issues mentioned above are resolved, I can eventually skip this step entirely.

The reason behind this is, it took 3 days to index everything on lemmy.ml (due to the frequency of errors) and it's still trying to just cross-reference those posts with all of the federated instances. If I do a full pull of posts on the federated instances, it'll end up taking months if not years just to index everything. With just doing a cross-reference lookup I can shorten the amount of time required to index a federated site to just an hour or so.

A lot of this is also based on how ActivityPub is suppose to work... the idea is that in theory all content from all instances should be shared with all other instances in roughly near real time. Now due to queuing issues there is a delay, but in theory the longer an instance has been online the closer it will be to having a complete copy of all of the content from the entire Fediverse. Any content created before an instance came online won't be present however. It's then just a matter of converting the Ids used by one instance to the Ids used by the target instance and you're good to go. (This appears to be the step that you're complaining about).

---

Slight tangent, but I'm exploring the option of replacing the preferred-instance drop-down with a settings button that just takes you to another page. Then on that page you can choose your preferred-instance (or maybe none at all), and include things like a NSFW filter, etc...

[krestenlaust]

I'll have to read up on activitypub before I can contribute any more to the technical aspect of this conversation :D

Slight tangent, but I'm exploring the option of replacing the preferred-instance drop-down with a settings button that just takes you to another page. Then on that page you can choose your preferred-instance (or maybe none at all), and include things like a NSFW filter, etc...

Generally, if you can cram everything into the single page you have right now, I think it would be preferred. I believe it's better UX to have your settings just at hand, and it's easier maintenance to have a single page. :)

[marsara9]

True but I'm looking a:

• home instance selection
• (turning off home instance selection entirely?)
• adding a NSFW filter
• others...?

As it stands, using this on mobile, doesn't leave much room for your actual search results as-is.

[krestenlaust]

Not because I want you to copy the UI, but https://lemmyverse.net/ has got plenty of space for those options:

Community tab:

[marsara9]

Just adding another screenshot for my own reference.