Handling adding non-legislator data to Open States

[jamesturk]

This is a planning issue to figure out what needs to be done to support non-legislator data in Open States. Given the potential contribution of hundreds of city officials along with support to maintain them, I'm going to prioritize this & related features.

Adding this data has been an obstacle in the past due to maintenance concerns, but with the existing contribution workflow for openstates/people I think we're in a much better place to handle mayors, etc.

Here's what I think it'll take:

• [x] adjustments to openstates/people schema to support other types of roles
  • [x] schema changes (new role types & new fields)
  • [x] lint_yaml support for new data
  • [x] new_person support for new data
  • [x] retire support for new data
  • [x] to_database support for new data
  • [x] improve linting for new data (check jurisdiction ids, etc.)
  • [x] enforcement of start & end dates on these new people (similar to #43), see note below
  • [x] decision as to whether it'd make the most sense to store these in the same repo (openstates/people) or split this into openstates/state-people and openstates/city-people (or similar)
• [x] addition of their data to Open States' API & bulk data
  • [x] deciding if they'll be present in API v2, or if this will be a v3-only feature
  • [ ] addition to API v3
  • [ ] documentation

On start & end dates: One of the main objections historically has been that the data would get stale/unmaintainable. I think the way to mitigate this is to enforce that these new role types have start & end dates. The obstacle to adding start & end dates to legislators is that we have so many of them already, and that the average legislator is re-elected. Mayors/Governors serve fewer terms on average, and we can do this on the way in, which will be less burdensome. It also means that when a Mayor's term expires, if we don't have new data for them, it isn't a big deal as that city just stops being supported. The case of a mayor resigning or dying in office is of course still an issue, but small enough that I don't see it being a major burden.

(This will also handle the planning portion of #46.)

[jamesturk]

cc @chris-erickson @csnardi for thoughts on any of the above

I'll update soon w/ a proposed schema update for openstates/people

[chris-erickson]

Looks good so far to me. We have a lot of volunteers (esp. non-technical) so we're keenly interested in how we can have them help in keeping data correct and current. Resistbot would build core functionality on this, so we'd be highly motivated to do that.

[jamesturk]

Here's what I'm thinking regarding adjustments to openstates/people:

• for simplicity, keep all data in openstates/people but rename the data/people directory to data/legislature and add a new data/cities subdirectory
• the retired directory will remain as-is and people can be retired from the legislature or cities into that directory
• update scripts to handle these new paths (most will just need to glob from data/legislature and data/cities, some will require a bit more)
• the only schema change I think needs to be made is additional allowed values for role, mayor for now

Here's a PR with just the schema change for now: https://github.com/openstates/people/pull/304/files

We'll also need to agree on an ocd-jurisdiction identifier for each new city. I'm thinking we'll just keep a CSV of them somewhere in the openstates/people repo for now. That'd look something like:

state,name,jurisdiction_id
NC,Cary,ocd-jurisdiction/state:nc/place:cary/government

Where the jurisdiction ID (just like the existing state ones) are parallel to the relevant OCD Division Identifier (https://opencivicdata.readthedocs.io/en/latest/proposals/0002.html)

The issue of unified government across jurisdictions has come up, multiple names could in theory map to the same OCD ID, which could help with mapping if you all want to do it at that level. If that's the case, we'd probably also just need to designate one name as primary for display purposes. I don't think this is a great idea after all.

[showerst]

Couple quick comments, they may be way out of scope if you've already worked it out with @chris-erickson

1. Not every city has an elected mayor. I'd add position options for city managers (who are sometimes also a councilpersons) and commissioners, or just change it some kind of catchall local executive.
2. I think Some big unified governments only have councils (no executive), so i'd add roles for city council, county council, and county commissioner roles, but that might be overkill.

Here's a nice rundown w/ some examples from NLC.

[chris-erickson]

We're very much interested in getting this all right, no reason we wouldn't someday want city councils, etc. We just wanted to focus on mayors first since it covers a lot of citizens. We might implement a hacky mayor lookup first internally if the timeline is too stretched is all.

[jamesturk]

Agree, don't want to preclude adding others as needed. Most others would just be additional positions in the enum as needed, but if there's anything we're doing that looks like it'd need major changes for those please let me know!

[jamesturk]

the PR now includes mayoral data for AK & AL, a script to convert the Airtable data to the YAML format, and changes to the linter & other scripts to handle this new data

the next step will be to import some into the database, convert the remaining states mayors to YAML, then import them all

the other checkboxes above are important to ensure feature parity, but won't block us getting this data in

[chris-erickson]

Took me a bit to figure out what you were referring to 😃 https://github.com/openstates/people/pull/304

Looks like a good start to me.

[jamesturk]

update: I have mayoral data loading into Open States’ database, going to load a couple of states in production to make sure there aren’t any weird side-effects I didn’t anticipate

once that’s working done I just need to figure out how to actually surface the data via the API

[jamesturk]

A few cities are now available:

https://openstates.org/graphql#query=%7B%0A%20%20jurisdiction(id%3A%20%22ocd-jurisdiction%2Fcountry%3Aus%2Fstate%3Aal%2Fplace%3Amobile%2Fgovernment%22)%20%7B%0A%20%20%20%20name%0A%20%20%20%20id%0A%20%20%20%20organizations(first%3A%201)%20%7B%0A%20%20%20%20%20%20edges%20%7B%0A%20%20%20%20%20%20%20%20node%20%7B%0A%20%20%20%20%20%20%20%20%20%20currentMemberships%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20person%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20id%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20name%0A%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20role%0A%20%20%20%20%20%20%20%20%20%20%20%20post%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20role%0A%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A

The list of IDs that work:

https://github.com/openstates/people/blob/master/data/al/municipalities.yml https://github.com/openstates/people/blob/master/data/ak/municipalities.yml

Doing a bit more testing, then will load the remaining states.

Let me know if this looks about right w/ how you'd want to use the lookup.

[chris-erickson]

Looks right to me. We actually had errors popping up this AM not expecting this new "place" so we knew the moment it went live! Will be pretty straightforward to update our stuff for this.

🎉

[jamesturk]

Oops, can you elaborate on where the errors were? I know there was one site hiccup, but wasn't expecting these to show up anywhere yet & want to avoid breaking others.

On Thu, Jul 23, 2020 at 5:13 PM Chris Erickson notifications@github.com wrote:

Looks right to me. We actually had errors popping up this AM not expecting this new "place" so we knew the moment it went live! Will be pretty straightforward to update our stuff for this.

🎉

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/openstates/issues/issues/87#issuecomment-663235160, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAB6YUOWKPTKIAN4LWYPUDR5CRV3ANCNFSM4OZ2WARQ .

[chris-erickson]

We discussed this in Slack, but for the room, we had a regex that was a bit too clumsy. Data was fine.

[jamesturk]

the mayors are loaded, had a couple of duplicates to sort out & a few other minor issues now resolved- the import just started & if all goes well it'll be done by ~5:30pm Eastern

[jamesturk]

👍 mayoral data is now loaded, other related features will be opened in new tickets

[chris-erickson]

@jamesturk can you recommend an efficient way to get all known mayors in one query? Whether it's something like get all people where type is mayor or, jurisdictions that content a person with role mayor...

Perhaps that's yet to be addressed in https://github.com/openstates/issues/issues/92?

[jamesturk]

Yep, it'll take a few other modifications to make that kind of query easy. To preserve backwards compatibility w/ folks only expecting states the jurisdictions for cities are practically hidden right now. Direct access to the mayors is possible by having the jurisdiction ID, but that's about it.

On Wed, Jul 29, 2020 at 1:43 PM Chris Erickson notifications@github.com wrote:

@jamesturk https://github.com/jamesturk can you recommend an efficient way to get all known mayors in one query? Whether it's something like get all people where type is mayor or, jurisdictions that content a person with role mayor...

Perhaps that's yet to be addressed in #92 https://github.com/openstates/issues/issues/92?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstates/issues/issues/87#issuecomment-665805267, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAB6YWCLNP6JEXTTWCZUALR6BNUVANCNFSM4OZ2WARQ .

[chris-erickson]

@jamesturk do you think that we'd be able to do this kinda query soon, or should we consider a different way instead?

[jamesturk]

I hadn't planned on making this kind of query ready soon- are you all going to be using that & the lookup by ID version?

On Fri, Jul 31, 2020 at 5:14 PM Chris Erickson notifications@github.com wrote:

@jamesturk https://github.com/jamesturk do you think that we'd be able to do this kinda query soon, or should we consider a different way instead?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstates/issues/issues/87#issuecomment-667359271, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAB6YX6VUHPGQSYJVXQIPDR6MXZRANCNFSM4OZ2WARQ .

[jamesturk]

Should add, I might have a quick solution for you all that's possible- want to make sure I'm understanding the use case to get it right though.

On Mon, Aug 3, 2020 at 12:25 PM James Turk james@jamesturk.net wrote:

I hadn't planned on making this kind of query ready soon- are you all going to be using that & the lookup by ID version?

On Fri, Jul 31, 2020 at 5:14 PM Chris Erickson notifications@github.com wrote:

@jamesturk https://github.com/jamesturk do you think that we'd be able to do this kinda query soon, or should we consider a different way instead?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstates/issues/issues/87#issuecomment-667359271, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAB6YX6VUHPGQSYJVXQIPDR6MXZRANCNFSM4OZ2WARQ .

[chris-erickson]

Ultimately we're populating our rep data base with it, so we need a way to get "all mayors" preferably with a single API call if possible. We parse this as part of an import stage, populate the database, then handle individual queries for a user out of band on the result data. That make sense?

[jamesturk]

Yep that makes sense, i’ll see if i can’t get a beta endpoint together really quickly for this then.

i’ll update by weds

On Mon, Aug 3, 2020 at 2:02 PM Chris Erickson notifications@github.com wrote:

Ultimately we're populating our rep data base with it, so we need a way to get "all mayors" preferably with a single API call if possible. We parse this as part of an import stage, populate the database, then handle individual queries for a user out of band on the result data. That make sense?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstates/issues/issues/87#issuecomment-668162636, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAB6YQWUCKAE5RXY2K4SF3R633TXANCNFSM4OZ2WARQ .