Open Civic Data model migration

[fgregg]

We have long wanted to refactor councilmatic to use opencivicdata models. The main advantage is to reduce the ETL from a two stage process to a one stage process, thereby

• reducing the lag between a data item changing on legistar and showing up on a councilmatic site by both councilmatic and the scraper using the same database.
• avoiding all sorts bugs in our import_data script for bringing data into councilmatic since we won't have an import_data script
• reducing the friction of getting a new site off the ground since. All we need is a scraper and councilmatic app, not an OCD API service.
• speed up development of new features

We are finally doing that, with financial support from the City of Pittsburgh.

I see the work proceeding in three stages.

1. With https://github.com/datamade/django-councilmatic/pull/240 and https://github.com/datamade/chi-councilmatic/pull/252, get the Chicago Councilmatic running without notificiations.
2. Update the councilmatic-starter template to work with the changes in django-councilmatic so that Pittsburgh can start to develop
3. Adapt the notifications app.

Relates to #75, https://github.com/datamade/devops/issues/59.

[fgregg]

Of the changes we nee to make, the notifications app poses the most technical uncertainty to me.

We need some mechanisms for the notification app to keep track of what has changed with regards to the bills, events, and search results that a user has subscribed to.

• Django signals seem like an obvious choice but this would require that the notifications app be running be part of the scraping process, as you would want to have the signal to be attached to a save method of the scraped objects. This seems like too much brittleness.
• ActivityStreams seem like another possible solution, which would let notifications and the scrapers run in separate processes.
• Something else.

It would be good to get a sense of a general approach before we even start building notifications, as it has consequences for the entire architecture.

@jeancochrane and @hancush I would particularly like your thoughts here.

Upon review of the notifications code, it's not particularly elegant, but it looks like it should be straightforward to adapt it. I think there are some nicer patterns to think about interprocess communication, but they are not a requirement of this project.

[fgregg]

Meta development questions

• [ ] We'll need to maintain two branches of django-councilmatic until we bring along NYC an LA Metro. How do we want to do that?

[fgregg]

For the implementation, I think I'd like

1. Figure out the general architecture of notifications with @hancush and @jeancochrane
2. Have me get the outstanding django-councilmatic and chi-councilmatic PR's over the line
3. Hand off updating the template to someone else
4. Hand off development of the notification app to someone else.

[hancush]

re: https://github.com/datamade/django-councilmatic/issues/241#issuecomment-483392724, the plan is to maintain a 1.0 branch, but not a separate release series on pypi. we will revisit this when we're ready to transition metro to 1.0, or if we run into a compelling reason before then.

[hancush]

update: we've released 2.5.0 to pypi to make installation easier across repos, thanks @jeancochrane!