Feature: integrate meilisearch for searching

Codycody31 commented 2 months ago

Adds support for the use of meilisearch for quick indexing and searching of discuit communities.

Required

[x] Move code for updating a communities index used before the API returns info to separate function
[x] Rather than placing search code in the core move to internal/search so we can have a package for it and not have to worry about names being the same
[x] Integration into the UI for searching (only if it is enabled)
[x] When indexing, break up the community name on each capital letter (to fix searching for Dev not matching DiscuitDev, etc)
[x] Allow searching on the communities page but default it when blank to the standard API

If Possible

[x] Index posts and users
[x] Sorting based on members, post creation time, user's account age, etc

General

After this is merged, someone who is better at UI stuff than me should probably make the searching be nice lol

Known Issues

When searching for a community while creating a new post, the picture is not used due to it not being included in the data set to meilisearch

Related Issues:

closes: #3
closes: #15
closes: #28

Codycody31 commented 2 months ago

A new issue has been found! batching of data, as currently, it would attempt to dump the entire posts table to the index, however, the posts table has over 27K records sooooo. I'll need to batch the data, good thing is I already have code for it, just need to convert it over.

Codycody31 commented 2 months ago

@previnder, the indexing logic along with everything else is finished. Now when pulling data to index it takes it in batches of 5000 records (I'll change it if you want it to be higher or lower, probably higher around 10,000 or 15,000). When pushing the data over to meilisearch we chunk it into batches of 50MiB (or less if we don't have that much data). By default, the searching ability is disabled and still makes use of the Google search to allow for the hoster to fully configure everything without having to worry about meilisearch till they are ready.

Codycody31 commented 2 weeks ago

Merged latest, and now configured to also work with environment variables

discuitnet / discuit