Removes database objects that haven't been seen in recent scrapes
optional arguments:
-h, --help show this help message and exit
--window WINDOW objects not seen in this many days will be deleted from the database
--report only generate a report of what objects this command would delete without making any changes to the database
--noinput delete objects without getting user confirmation
Testing Instructions
Make sure tests pass
To test with a live database, you should use a local instance of opencivicdata/scrapers-us-municipal:
Make sure your database has been populated
Rebuild your scrapers containers with git by adding sudo apt-get install git to the Dockerfile
Install this branch of pupa locally by copying this directory into the root directory with cp -r /path/to/pupa . and running docker-compose run --rm scrapers pip install -e pupa
Make sure the new command works with docker-compose run --rm app pupa clean
Coverage: 94.806%. Remained the same when pulling 9f51bbf3f9c5ca8fd3ba048f61c02221ee31c135 on clean-cli into a9c9f80814e92df5ecd0c2525e77d7216ae51949 on master.
Overview
This PR adds a new pupa CLI command:
pupa clean
:Testing Instructions
To test with a live database, you should use a local instance of opencivicdata/scrapers-us-municipal:
git
by addingsudo apt-get install git
to theDockerfile
cp -r /path/to/pupa .
and runningdocker-compose run --rm scrapers pip install -e pupa
docker-compose run --rm app pupa clean