search

datamade / django-councilmatic

:heartpulse: Django app providing core functions for *.councilmatic.org
http://councilmatic.org
MIT License
26 stars 16 forks source link

Clear downloads #228

Closed reginafcompton closed 5 years ago

reginafcompton commented 5 years ago

This PR implements an option for clearing the contents of the downloads directory.

It handles several related issues, mainly: https://github.com/datamade/nyc-council-councilmatic/issues/155 and https://github.com/datamade/django-councilmatic/issues/226

I emulated the pattern used for import_only and download_only, because I wanted to preserve the option of NOT clearing the downloads. Why? Sometimes, one must run a big import, and preserving the contents of the downloads for some interval of time can be helpful.

Does this make sense? If so, I can add the pattern to people, bills, and events (or think about refactoring for DRYer code).

@evz and @hancush - could you take a look at this before I go any further?

reginafcompton commented 5 years ago

I diverged slightly from my original solution. I implemented an optional keep_downloads argument.

The import for people expects downloaded data about organizations. (N.b., This has always been the case, but we did not have safeguards in place for handling the people endpoints in isolation). So, rather than deleting the downloads, folder-by-folder, as the import happens, the script just deletes everything at the end.

If someone is running a big import and wants to keep everything, then they can do that with the keep_downloads option.