search

open-city / councilmatic

DEPRECATED, replaced by https://github.com/datamade/chi-councilmatic
16 stars 9 forks source link

Handle incremental updates #5

Closed derekeder closed 11 years ago

derekeder commented 11 years ago

Part 1

Currently updatelegfiles throws a UNIQUE CONSTRAINT error when trying to save a piece legislation already in the system. Should handle this gracefully and move on to the next.

Part 2

Come up with a smarter way of deciding what legislation to update. Current implementation will index the entire Legistar site, which takes several days to go through. Perhaps use datestamps? Is there a 'last updated' date we can search on?

derekeder commented 11 years ago

Part 1 is done: currently handling this with 'Update found or Insert'. Still doing a full scrape of the API, which pretty inefficient.

derekeder commented 11 years ago

This is working well enough with our weekly job. Closing.