114th Congress Update Strategy

[JoshData]

Hey,

Let's use this issue to plan how to update for the 114th Congress.

Something like:

• [x] Identify current Members that won, copy their term info forward but update dates. Check if anyone changed district/party (not likely). Be careful if any representatives became senators. - @joshdata
• [x] Move incumbents that lost to the historical file. (nb. Not everyone was up for reelection.) - @joshdata
• [x] Identify if any new members served in Congress previously and bring their records forward w/ a new term. - @joshdata
• [x] Add remaining new members.
• [x] Review junior/senior status
• [x] Add end dates to all current leadership_roles (no leadership_role should extend into the next Congress) - @rshorey or @handlers
• [x] Add leadership_roles for the 114th Congress (#235)
• [ ] Not sure what to do about committees. There won't be committee assignments initially.
• [ ] Photos? Did/will GPO publish a new member directory?
• [X] Fill in bioguide IDs when they become available.
• [X] Delete retirees' social-media records.
• [ ] Add new members' social media accounts. (#234)
• [ ] After the 114th Congress begins, fill in missing data.

I'd like to set up a google doc spreadsheet where the rows are all of the offices that were up for election (all representatives, presumably all delegates, not sure about Puerto Rico, and one of the Senate classes) so we can make sure we take care of each election result.

@dwillis Any overlap with open elections data?

Did I miss anything?

cc @handlers @konklone

[dwillis]

I'm happy to set up a google doc spreadsheet with the House and Senate races, indicating those who are continuing and those who are not (and their replacements). Leadership roles in Senate don't carry over; there will be new elections (most have happened already).

[plantfansam]

This seems like a good to-do list and workflow to me!

[dwillis]

Here's a google doc spreadsheet with current and new members (where known): https://docs.google.com/spreadsheets/d/1H8z7Ah4jSlXiuIol3oXoWBR8s6h0OtA62dNlU-kiIlU/edit?usp=sharing

[JoshData]

Wow great.

How were there three Class III senators with elections?

[rshorey]

Senate seniority is actually more complicated than originally thought since in our schema seniority is linked to term, but it usually changes mid-term when a colleague from another senate class loses an election. Moving to issue #225.

[JoshData]

I put a first pass at moving existing legislator records around in a new 114th_congress branch and also creating stub entries for new members based on the spreadsheet @dwillis posted (which is amazing, btw):

https://github.com/unitedstates/congress-legislators/commits/114th_congress

This moves incumbents who lost into the historical file, historical members who are returning into the current file, copies terms forward for re-elected incumbents, adds new stub terms for any existing member that is elected to a different office, and creates stub entries for new members.

The script that did the work is also committed. There are some TODOs to be resolved. We should sanity check the output a bit before committing on top of it in case we need to re-run the script.

[dwillis]

@JoshData: Realized I didn't answer your questions about Class III Senate elections. These were all to fill full term for NJ, MA & OK due to vacancies and, in the case of Coburn, early retirement.

[rshorey]

I spot-checked the current and historical yaml files. The format looks good to me, but I did notice an issue with someone who we retired but shouldn't be that seems to have stemmed from a typo in the google doc. I'm doing some checks on that, and will correct on the doc and then submit a PR to fix the csv on the 2014 branch shortly.

[rshorey]

Here's a list of senator/rep-elects we should check by hand at the end of this process (for example, because they switched from the House to the Senate or they've had discontinuous terms).

Brian Schatz (S, Special election) James Lankford (S, Special election; previously in house) Tim Scott (S, Secial election; previously in house) Tom Cotton (S, previously in house) Corey Gardner (S, previously in house) Bill Cassidy (S, previously in house; election date is weird and might impact things like seniority) Gary Peters (S, previously in house) Steve Daines (S, previously in house) Shelley Moore Capito (S, previously in house) Robert Dold (H, served in house previously but not 113th) Frank Guinta (H, served in house previously but not in 113th) Debbie Dingell (H, replacing her husband of the same last name) Donald Norcross (H, Special election) Alma Adams (H, Special election) Dave Brat (H, Special election)

[dwillis]

Schatz was not previously in the House; he was appointed to the Senate and won a special election. Tim Scott was previously in the House, but was appointed to the Senate in 113th as well.

[rshorey]

Looks like I got the lines for Schatz and Lankford backwards. Fixed. Thanks for the quick heads up!

In the case of all of the special election senators, their terms all end at a different time than the others elected in 2014, so we'll want to double check that. For the house special elections noted here, their terms began on election day, I believe, so we'll want to check those, too.

[JoshData]

I just added the AZ-2 results, which was the last unresolved election, so the branch should now have everyone. We can start reviewing, filling in other ids if anyone wants to tackle them, etc.

[JoshData]

All of the core stuff is ready to go in #230. I want to get this merged asap, so last call.... Thanks for everyone's help getting this far.

[plantfansam]

Oh wow — thanks! :+1: from me on the merge of #230 .

[rshorey]

Sounds good to me. Do we have the promised xml file with bioguide IDs yet?

[JoshData]

No but a little birdie told me to guess the IDs by looking at the next sequential IDs --- I actually committed a script that reports the next set of bioguide IDs. (It doesn't work perfectly.) So I did that and entered all of the bioguide IDs for the new members. (I also copied in the new opensecrets IDs.)

We should be on the lookout for when LIS IDs are posted at http://www.senate.gov/general/contact_information/senators_cfm.xml.

[rshorey]

I'm using that same CSV to deal with contact congress so having them already in the csv is actually even better than xml! I will swipe it and commence those edits on Monday. Thanks for doing this work.

[JoshData]

I put the bioguide/opensecrets IDs right into the YAML.

[plantfansam]

Senate committee assignments:

http://www.republican.senate.gov/public/index.cfm/blog?ID=cccbb684-0ead-473b-8f51-2f5dee96acb2 http://democrats.senate.gov/committees/#%2EVJoOsB0MBg

I'll get going on them ASAP.

[plantfansam]

Some committee updates:

It looks like the Senate has XML for committees but not subcommittees right now.That results in this ugly branch in which we lose subcommittee data. The first commit is just stripping out retired members from committee-membership-current.yaml; the second is the output of a scrape of the Senate committee XML. If anyone has opinions on whether or not to merge, I am all ears.

On the House side, we're waiting for clerk.house.gov to update their committee information pages. I spoke to them about this committee assignments table and they said they were waiting on the House Speaker's office for the data. I presume that's also true of the more committee-specific pages as well.

[JoshData]

I'm OK with merging partial data.

I've wiped all of the committee membership info from GovTrack in the meanwhile so as not to report anything that's no longer true. I'll wait until we have fresh data from the House before pulling any of it in.

[plantfansam]

See #250, which has most committee memberships for the 114th Congress.