[NLP, Backend] Increase search performance

FHNW-IVGI / Geoharvester

NDGI Project Geoharvester

10 stars 1 forks source link

[NLP, Backend] Increase search performance #99

Open FStriewski opened 1 month ago

FStriewski commented 1 month ago

The search takes to long to return results, as we search across all languages.

Ideas:

Detect language from the searchstring (Searchfield). Maybe we want to limit this feature to a search with >X words (stopwords excluded), to make the detection more robust.
Add UI to set the search language. This might be different from the UI language and could be confusing. In this case a search in DE would only search DE fields, but not EN, FR, IT.

p1d1d1 commented 1 month ago

@FStriewski I could be wrong but as far as I can remember we said that "search language = ui language". Meaning: setting the UI language also sets the search language. And it makes sense to me. What do you think?

FStriewski commented 1 month ago

@FStriewski I could be wrong but as far as I can remember we said that "search language = ui language". Meaning: setting the UI language also sets the search language. And it makes sense to me. What do you think?

That would be an easy and quick fix, indeed. But it would prevent you from searching in another language (unless you change the UI to that language)

p1d1d1 commented 1 month ago

I don't see this as an issue. To search in another language one has to set the UI language to that language. If e.g. I go to https://www.digitec.ch/ and I want to search in IT, I have to set the UI language to IT.

The objective of "our" multilingual search is to enable e.g. me as italian speaking guy to get search in "my" language" but get results not only from Ticino but also from german and french sources.

FStriewski commented 1 month ago

I have limited the redis search to fields matching the UI language (or the lang parameter for the api search) and it seems that the performance is significantly better now.