Deal with deleted bills

[fgregg]

The pdf for this bill http://chicouncilmatic.datamade.us/legislation/o-2015-8818/ has been deleted. https://chicago.legistar.com/Confirmation.aspx?M1=Gone

Let's deal with the "Gone" response in our pdfviewer

[derekeder]

Updated URL https://chicago.councilmatic.org/legislation/o-2015-8818/

City Clerk has deleted this bill from Legistar: https://chicago.legistar.com/LegislationDetail.aspx?ID=2533054&GUID=25F7BFE8-432A-4582-BE8A-79A0E12A541D

[fgregg]

Thought about this past couple of days.

There are two ways we can fix this, in data source (ocd) or in councilmatic.

If we fix in OCD, we would

• need a way to delete bills
• periodically check all bills
• in councilmatic, we would either need to check all bills periodically against ocd or add a way for ocd to communicate that bills have been deleted

If we fix in councilmatic,

• we don't have to worry about ocd. (stuff that's present in OCD but deleted in councilmatic will not be reloaded during a running instance of ocd, unless we fully reloaded the databased from ocd)
• we can be incremental in deleting

I propose we do something like this.

For every bill, we create a delete route that looks like

https://chicago.councilmatic.org/legislation/o-2015-8818/delete

This adds this bill to a delete queue that we can manage in the django admin. In the django admin we can confirm that we want to delete nominated bills. From that admin interface, we can also export a list of all deleted bills, if we, at some point, want to delete bills upstream in OCD?

Thoughts @evz

[derekeder]

related, we now have a last_seen property in python-opencivicdata for scraped items that we can use to periodically remove deleted bills: https://github.com/opencivicdata/python-opencivicdata/pull/145