search

Metro-Records / la-metro-councilmatic

:metro: An instance of councilmatic for LA Metro
MIT License
6 stars 2 forks source link

Search for Project results in error after clicking "tag search" #972

Closed camachoo closed 11 months ago

camachoo commented 1 year ago

Type "project" in the search bar on the home page. Click enter. Click "Tag Search" image

hancush commented 1 year ago

I think this term is associated with so many board reports that it's breaking Solr. Not sure how to fix, but can look into it.

shrayshray commented 1 year ago

I noticed the top tagging terms are all experiencing the same issue. See attached for report on top tagging terms (thanks to the reports Datamade is generating for us!). It does seem like an issue to have this occurring for the most popular terms. There is work to be done on our side to evaluate whether these terms are over-tagging, and on the Datamade side to see if this level of tagging can be accommodated. Even if over-tagging is occuring now, it's possible we'll have this amount of instances in the future. report-2023-05-08-board-report-tag-analytics.pdf

hancush commented 1 year ago

@antidipyramid is going to run the full scrape and index all board reports on the Heroku instance, then check to see whether the switch to Elasticsearch resolves this problem.

Let us know what you find out!

antidipyramid commented 1 year ago

@hancush I'm getting this error during a full scrape:

Screen Shot 2023-09-06 at 2 30 53 PM

Do we just have to increase storage on the AWS server?

hancush commented 1 year ago

Try sudo docker system prune first, then run df -ah to see how much space you have left.

antidipyramid commented 1 year ago

@hancush After docker system prune:

Screen Shot 2023-09-08 at 9 38 47 AM Screen Shot 2023-09-08 at 9 39 34 AM

Currently the instance has 20GB of storage. Our current staging instance also has 20GB but prod has 50GB.

Should we up the storage?

antidipyramid commented 1 year ago

@hancush Upped the storage on the EC2 instance. The full scrape now completes successfully.

After a tag search for Project, we get this error:

Screen Shot 2023-09-11 at 8 55 37 AM

Seems like we're not forming these search queries correctly. Or they're at least extremely inefficient.

hancush commented 1 year ago

@antidipyramid This method constructs and runs those queries: https://github.com/Metro-Records/la-metro-councilmatic/blob/70b6e171b9b0033511b55c43ddd8f87a417a8179/lametro/forms.py#L63

There's definitely a better way to do this. Are you able to try some different approaches this week?