search

marsara9 / lemmy-search

An enhanced search engine just for Lemmy/Fediverse
https://www.search-lemmy.com
GNU Affero General Public License v3.0
81 stars 4 forks source link

Still active? #63

Open bendangelo opened 11 months ago

bendangelo commented 11 months ago

Is this still active? I have interest in taking the lead on this project.

marsara9 commented 11 months ago

I haven't had the time to focus on this in quite awhile. I'd be more than happy to explain my thoughts on how to push this project forward and ultimately add you as a owner.

(I'll update this comment tomorrow with more details).

bendangelo commented 11 months ago

Ok. I was also wondering how does this improve search over the default search system? Because it's already quite good.

marsara9 commented 11 months ago

The original idea was to aggregate the search results across ALL of the instances in the fediverse. Initially only searching all of lemmy and then eventually including Mastodon, Kbin, etc...

This way you had a single location / site you could go to and search everything, even across sites that may no longer be federated.

Also, and it may have changed, but the built-in search of Lemmy, in particular, is actually pretty bad. For example searching for foo bar and bar foo would return completely different results. The search was also horribly il-performant as well. Rather than update the code within Lemmy, which would have required a database rewrite, I just created a separate project that had a database specifically built for searching.

Long story short though. Since the APIs that allowed this project to function, no longer work. A new solution to indexing the required data needs to be developed. Ultimately this means making this project its own ActivityPub server. So it can participate in the fediverse just like anything else and will receive pushes with the new data rather than trying to periodically pull the data via the official APIs.

bendangelo commented 11 months ago

Ok that all makes sense to me. Going forward as an ActivityPub server is the right move. I'll dig more into this soon.

slrgt commented 10 months ago

@marsara9 Any update? I would love to be able to set lemmysearch as my default search engine

marsara9 commented 10 months ago

The project is all but dead sadly, as I don't have the time to dedicate to a rewrite at the moment. If anyone wants to take up the mantle I'll more than happily explain my ideas going forward and transfer ownership.

slrgt commented 10 months ago

Could you write your ideas and make a post on mastodon that mentions a lemmy community about it? So its on mastodon and lemmy. Maybe it will get enough eyes on it there that someone will fork it to continue it. There’s a lot of interest in an open source search engine right now especially after Kagi started paying Brave and people dont like that. A search engine for the fediverse could be as useful as google considering most people just google a question and add reddit to the search and now lemmy is getting enough threads to compete with reddit as a source of useful up to date information.

On Sun, Jan 7, 2024 at 12:42 PM, marsara9 @.***(mailto:On Sun, Jan 7, 2024 at 12:42 PM, marsara9 < wrote:

The project is all but dead sadly, as I don't have the time to dedicate to a rewrite at the moment. If anyone wants to take up the mantle I'll more than happily explain my ideas going forward and transfer ownership.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

marsara9 commented 10 months ago

There's only a few posts on Lemmy, no community. But regardless, one thing to keep in mind is that this wasn't meant to be a general search engine. The original idea behind this was to have a way to search the Fediverse as a whole.

Prior to Lemmy really gaining traction, and even today it's still rather hard to search for relevant topics on Google (or any other search engine) and find posts or even communities for your interests / questions. This project was thus born to solve the problem of adding "Reddit" to the end of all your Google search queries (but for the Fediverse).

Originally this worked by using the public APIs that were built into Lemmy in order to index new and old posts alike. But eventually attacks on Lemmy required that those same APIs be shutdown or at least modified to prevent abuse.

The alternative then becomes to make this project follow the same ActivityPub standards that all other fediverse projects use. This won't allow historical indexing but it will allow all future posts to be searched. This should in theory also allow more easy integration into other fediverse platforms like Mastodon, Kbin, etc... as well.

But this constitutes a significant rewrite of this existing server which I don't currently have the time or resources for. But I'll be more than happy to hop on Discord, Lemmy, etc... in order to explain details about ActivityPub, or the existing server structure, etc... as needed in order for someone else to develop this.

slrgt commented 10 months ago

Yeah I meant as a fediverse search engine it would be enough to replace google for me and most people.

Could you make a thread announcing you can’t continue working on it snd explaining your ideas and how someone else could take the project over? A post specifically about it shutting down could spark enough interest for someone to tske the mantle

On Sun, Jan 7, 2024 at 1:02 PM, marsara9 @.***(mailto:On Sun, Jan 7, 2024 at 1:02 PM, marsara9 < wrote:

There's only a few posts on Lemmy, no community. But regardless, one thing to keep in mind is that this wasn't meant to be a general search engine. The original idea behind this was to have a way to search the Fediverse as a whole.

Prior to Lemmy really gaining traction, and even today it's still rather hard to search for relevant topics on Google (or any other search engine) and find posts or even communities for your interests / questions. This project was thus born to solve the problem of adding "Reddit" to the end of all your Google search queries (but for the Fediverse).

Originally this worked by using the public APIs that were built into Lemmy in order to index new and old posts alike. But eventually attacks on Lemmy required that those same APIs be shutdown or at least modified to prevent abuse.

The alternative then becomes to make this project follow the same ActivityPub standards that all other fediverse projects use. This won't allow historical indexing but it will allow all future posts to be searched. This should in theory also allow more easy integration into other fediverse platforms like Mastodon, Kbin, etc... as well.

But this constitutes a significant rewrite of this existing server which I don't currently have the time or resources for. But I'll be more than happy to hop on Discord, Lemmy, etc... in order to explain details about ActivityPub, or the existing server structure, etc... as needed in order for someone else to develop this.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

rcmaehl commented 10 months ago

Hi @slrgt, if you continue this project, create a spin off, or find someone to take it over; please let me know! I run a project that replaces Windows Search with alternatives and I'd love to have a Fediverse search engine in the list.

paulo-roger commented 3 months ago

Hello, any news regarding the project?

bendangelo commented 3 months ago

I've made no progress in lemmy search. But going forward I believe creating a common crawl and integrating it with https://github.com/spyglass-search/spyglass is the best solution.