datamade / nyc-council-councilmatic

NYC Council version of Councilmatic
MIT License
7 stars 3 forks source link

Solr updates an awful lot of bills when running `update_index` #140

Closed reginafcompton closed 5 years ago

reginafcompton commented 6 years ago

The Solr log indicates that Solr updates thousands of Bills, even with an age argument. This is a problem, because it delays data imports (i.e., data imports do not execute every 15 minutes as desired, because Flock politely waits for Solr to finish indexing and for the RTF conversion script to run).

Command run in cron: python manage.py update_index --batch-size=100 --age=1

Output in log (i.e., /var/log/councilmatic/nyc-updateindex.log):

Indexing 25967 nyc bills
INFO 2018-04-06 13:00:04,480 Starting new HTTP connection (1): 127.0.0.1
...

Does the age argument not work as we expect in Solr 7?

reginafcompton commented 5 years ago

Closed via https://github.com/datamade/django-councilmatic/pull/228

Explanation: https://github.com/datamade/nyc-council-councilmatic/issues/155#issuecomment-458246086