search

Metro-Records / la-metro-councilmatic

:metro: An instance of councilmatic for LA Metro
MIT License
6 stars 2 forks source link

Reduce the size of search results #1151

Open hancush opened 1 month ago

hancush commented 1 month ago

One search result can be several MB in size. That can really add up! I suspect the biggest culprits are the full text fields. Figure out a way to index that information, but not return it in the results. Looks like ElasticSearch provides for this, if django-haystack does not: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html

hancush commented 1 month ago

Related to #1145, #718

hancush commented 1 month ago

Also related to efforts to reduce page load time.

antidipyramid commented 1 month ago

It doesn't look like there's a way to add a fields parameter to the Elastic query with Haystack. We can try extending Haystack's Elasticsearch7SearchBackend class.

antidipyramid commented 1 month ago

@hancush I tested #1152 on staging and found that it didn't lower the size of the index by very much. I've read that elastic uses a compression algorithm by default in indexes, so that may be why.

It seemed to reduce memory usage when searching, but not by much either.

hancush commented 1 month ago

Hmm, bummer, @antidipyramid.

antidipyramid commented 1 month ago

@hancush I was able to get Elastic to exclude the text fields of bills from search results by customizing Haystack's Elastic backend. I deployed that branch and found that, again, it reduced memory usage somewhat but not enough to be considered a fix.