Use OCD Models for Councilmatic Models

[fgregg]

We assumed a development pattern like this.

1. Develop scraper and submit to DataMade. DataMade will run this scraper daily
2. Develop local site. Pull in information from DataMade API with load_data

It turns out that it's pretty hard to do these in two, cleanly separated steps. By looking at what's in your local councilmatic, you find things that need to be fixed in the scraper. It's pretty frustrating when the devlelopment cycle for those fixes in 24 hours.

I'm interested in looking at whether the we can use the OpenCivicData bill, person, event, etc models for Councilmatic. If so, then it might be possible to use the same local db for developing the scraper and for developing the site.

Thoughts @derekeder @mccc @cathydeng @evz @patcon @ErnieAtLYD

[patcon]

The only downside I can think of is that it was nice that the councilmatic db didn't make use of postgres-specific fields (uuid and json), and so it was easily possible to use an sqlite db to make dev onboarding simpler. BUT I would give that up in a heartbeat if we could figure out a nice way to use the pupa db directly.

Also, I've found it non-intuitive to extend the base bill model in the city-specific app without breaking everything(something about Proxy Models not being extensible) . This makes me particularly excited about your suggestion @fgregg, as it renders that concern irrelevant

And fwiw, I've personally sorted out my pain points (although perhaps not in the ideal way), but I genuinely think this would make it much simpler for the next person who walks the path :)

[fgregg]

@evz if we did this we could do table level replication and we could ditch import_data, right?

[ErnieAtLYD]

To be honest, it's been a couple of months, maybe even a year since I've last worked on this project. I remember being challenges with the scraper, but I also remember challenges with this particular repo, that the content here had hard-coded values that weren't applicable for Miami-Dade County. It got pretty intimidating as to which code base was our city base one, and which one was part of this.

Here's an example: https://github.com/datamade/django-councilmatic/blob/master/councilmatic_core/templates/councilmatic_core/council_members.html#L21

Miami-Dade County doesn't use a council member format; here, they are called commissioners. Stuff like that.

[fgregg]

I think we we really underestimated the effort in standing one of these these things up and did not provide adequate guidance and support. I'm sorry that you had a frustrating experience.

[ErnieAtLYD]

If I had any large suggestion - and I'm not sure if this is the same of this current task or not - is that I would move the "views and templates for bill/person/organization/event pages, search, and more" functionality out of this into the local repos so offer more control over nomenclature and structure.

It was also not obvious to me at first development that this repo was relied on for templating, so I would absolutely make a copy change to this on the city-councilmatic repos:

OPTIONAL: install django-councilmatic locally If you plan on making changes to core councilmatic features (as opposed to Miami-specific stuff), you'll want to install django-councilmatic locally instead of installing from pypi.

[evz]

@fgregg There's still going to be the need to bootstrap the database with some kind of data so I'm not sure we can ditch it altogether. What, specifically, were you thinking? Something like this?

pg_dump <ocd_database> <table> | pg_restore <councilmatic_database>

It seems like we'd want some kind of abstraction there to make it easier for others to prop up. For production purposes, it seems silly to run an instance of the scraper in parallel with the councilmatic instance when the same scraper is also running to populate the OCD API. But for development purposes, that seems like a nice option to have.

[fgregg]

WE DID IT!