search

unitedstates / congress-legislators

Members of the United States Congress, 1789-Present, in YAML/JSON/CSV, as well as committees, presidents, and vice presidents.
Creative Commons Zero v1.0 Universal
2.08k stars 507 forks source link

Committee assignments? #36

Closed JoshData closed 11 years ago

JoshData commented 11 years ago

Anyone know if the House is working on posting them and if the Senate XML up to date?

dwillis commented 11 years ago

I've got them in NYT API, but house is working on them. Senate XML is up to date on cmte-by-cmte basis, at least.

On Saturday, February 9, 2013, Joshua Tauberer wrote:

Anyone know if the House is working on posting them and if the Senate XML up to date?

— Reply to this email directly or view it on GitHubhttps://github.com/unitedstates/congress-legislators/issues/36..

JoshData commented 11 years ago

Okay I guess from laziness I'll wait for the House.

konklone commented 11 years ago

The Senate's XML seems to match the HTML, and include top-level committee assignments for new members. New members don't seem to have been assigned to subcommittees yet, though. Derek, do you know if the Senate assigned subcommittees in their resolutions? Anyway, I'm fine with doing repeated sweeps over time as stuff gets added.

But as written, the committee_memberships script erases House memberships entirely, because it loads in the whole memberships file, looks for all current things, and saves it. It'd need refactoring to support doing one chamber at a time. So I've also been letting it sit. People keep writing in about them missing, though...

On Sat, Feb 9, 2013 at 7:37 PM, Joshua Tauberer notifications@github.comwrote:

Okay I guess from laziness I'll wait for the House.

— Reply to this email directly or view it on GitHubhttps://github.com/unitedstates/congress-legislators/issues/36#issuecomment-13342010..

Developer | sunlightfoundation.com

dwillis commented 11 years ago

The subcommittee assignments are not in the resolutions, no, and while they are in the committee XML, they do not appear to be complete. The House resolutions are already out of date as we've had a couple of late additions and committee resignations.

konklone commented 11 years ago

I updated Senate committee assignments, and this actually did include subcommittee assignments. So I think that's done? House is still not updated...

JoshData commented 11 years ago

Really appreciate that.

dwillis commented 11 years ago

I've added some late additions/changes to House cmte assignments to NYT API, but yeah, still not updated on Clerk's site.

konklone commented 11 years ago

@dwillis, any objections or warnings about me using the NYT API to temporarily fill in our committee assignments for the House?

dwillis commented 11 years ago

I think we're up to date, so no. We do have ppl who served but left a cmte in the response, so anything where the end date is less than today is an active member.

konklone commented 11 years ago

I think I crashed it. :( http://api.nytimes.com/svc/politics/v3/us/legislative/congress/113/house/committees.json?api-key=#### with a proper API key goes boom, either directly, or using the API Request Tool.

dwillis commented 11 years ago

Nope, wasn't you. Fixed now!

konklone commented 11 years ago

Thank you!

On Mon, Mar 11, 2013 at 5:06 PM, Derek Willis notifications@github.comwrote:

Nope, wasn't you. Fixed now!

— Reply to this email directly or view it on GitHubhttps://github.com/unitedstates/congress-legislators/issues/36#issuecomment-14742721 .

Developer | sunlightfoundation.com

konklone commented 11 years ago

And this is just top-level committee members, not subcommittees, right?

JoshData commented 11 years ago

@konklone: Any progress? Can I help?

konklone commented 11 years ago

I stalled when I saw it was just top-level committees. I can just accept that for now, and finish this up tomorrow. Thanks for the poke.

konklone commented 11 years ago

@dwillis - I think you have a bug in your membership listings. The Permanent Select Committee on Intelligence, with an official url of intelligence.house.gov, has Dianne Feinstein as the chair:

http://prototype.nytimes.com/gst/apitool/index.html?api_id=4&request_id=4&congress-number=113&chamber=house&committee-id=&resp_format=json&perform_request=Make+Request&use_pp=on

dwillis commented 11 years ago

Gaah! Will fix.

Fixed now. Sorry about that.

konklone commented 11 years ago

Another, smaller one for you - Jo Ann Emerson has resigned, but her membership is still present in HSAP: http://prototype.nytimes.com/gst/apitool/index.html?api_id=4&request_id=4&congress-number=113&chamber=house&committee-id=HSAP&resp_format=json&perform_request=Make+Request&use_pp=on

konklone commented 11 years ago

I believe I have this done in https://github.com/unitedstates/congress-legislators/commit/d67656ccab55115015b17f58029e62e21a24ec8e, and it looks okay. I'm ignoring any members whose ID doesn't appear in our legislators-current file, and preserving all Senate data (committees whose IDs begin with J or S). Gonna try loading it into Sunlight's stuff now - do feel free to try it out and spot check.

JoshData commented 11 years ago

Cool. No data for HLIG (House Permanent Select Committee on Intelligence) though.

konklone commented 11 years ago

I think that's cause of the bug Derek's working on. When he's fixed it, the script can be re-run and it should (I think) include it.

On Wed, Mar 20, 2013 at 3:25 PM, Joshua Tauberer notifications@github.comwrote:

Cool. No data for HLIG (House Permanent Select Committee on Intelligence) though.

— Reply to this email directly or view it on GitHubhttps://github.com/unitedstates/congress-legislators/issues/36#issuecomment-15197771 .

Developer | sunlightfoundation.com

JoshData commented 11 years ago

Oh I see it's HSIG in the NYT API but HLIG on THOMAS. I can do the fix but I don't want to step on your toes, Eric.

dwillis commented 11 years ago

Ok, I've fixed this issue - the former_members array now shows Emerson (and two Democrats as well), and for other cmtes.

dwillis commented 11 years ago

On the HSIG/HLIG issue, I think HSIG is how the House clerk refers to it, but I could be persuaded to use HLIG if that's what ppl want.

JoshData commented 11 years ago

Spot check- Andy Harris is listed in four committees, but according to his web page he left three to join Approps. http://harris.house.gov/about-me/committees-and-caucuses

Other spot checks look good!

JoshData commented 11 years ago

Let's just special-case HSIG/HLIG here.

konklone commented 11 years ago

Fixed with a workaround for now, where it uses HLIG to query the NYT API, but saves it as HSIG.

That's a good spot check, though I think we'll have to accept the risk of outdated information until the Clerk updates its information. I don't want to make manual adjustments to the NYT-downloaded data, if we're going to replace it.

JoshData commented 11 years ago

Sure. On GovTrack I used to only update membership a few times a year, so I'm not worried about it. Definitely don't want to manually modify it here. Thanks guys! (It's coming up on GovTrack momentarily.)

dwillis commented 11 years ago

Harris is correctly listed as a former member for Science, Transportation and Natural Resources in the API.

konklone commented 11 years ago

So he is - but I'm only using current_members, not former_members. Looking into it...

konklone commented 11 years ago

Fixed it, I had cached some relatively recent version of the NYT responses which didn't have them listed as former members yet. I wasn't passing the force-download flag to the download call, so it was always using the cache and never re-downloading the updated data. The data now excludes what the NYT lists as former members.

JoshData commented 11 years ago

Harris still appears in HSII and HSPW (i.e. one correctly removed, two remain).

konklone commented 11 years ago

@dwillis is clearly on this, they've now been removed from the NYT API, along with a bunch of other fixes. Committed an update.

JoshData commented 11 years ago

Thanks guys! (I think we can close the ticket.)

konklone commented 11 years ago

I don't know - this doesn't feel done until the House updates and we get subcommittees. Anyone have any idea when that'll happen?

dwillis commented 11 years ago

I don't, but we can probably ask them on Friday.

schmod commented 11 years ago

Joint committees are currently only listing Senate members.

Do we have a listing for House members of joint committees anywhere?

konklone commented 11 years ago

I don't know if I've ever seen House membership of joint committees in my life. :/ Anyone else have any insight on this?

On Thu, Mar 21, 2013 at 11:50 AM, schmod notifications@github.com wrote:

Joint committees are currently only listing Senate members.

Do we have a listing for House members of joint committees anywhere?

— Reply to this email directly or view it on GitHubhttps://github.com/unitedstates/congress-legislators/issues/36#issuecomment-15246737 .

Developer | sunlightfoundation.com

JoshData commented 11 years ago

Hasn't been posted on the Clerk site since the 111th Congress.

Eric Mill notifications@github.com wrote:

I don't know if I've ever seen House membership of joint committees in my life. :/ Anyone else have any insight on this?

On Thu, Mar 21, 2013 at 11:50 AM, schmod notifications@github.com wrote:

Joint committees are currently only listing Senate members.

Do we have a listing for House members of joint committees anywhere?

— Reply to this email directly or view it on GitHubhttps://github.com/unitedstates/congress-legislators/issues/36#issuecomment-15246737 .

Developer | sunlightfoundation.com

Reply to this email directly or view it on GitHub: https://github.com/unitedstates/congress-legislators/issues/36#issuecomment-15246886

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

konklone commented 11 years ago

Derek says he's got them in the NYT API, from the resolutions. I'll update the script to add them.

konklone commented 11 years ago

Right now, the NYT has full House and Senate membership for the Joint Economic Committee (JSEC), but only Senate members for the Joint Committee on Taxation (JSTX) and no members for the Library (JSPR) or Printing (JSLC) joint committees.

GPHemsley commented 11 years ago

FYI: http://www.govtrack.us/congress/bills/113/sres88/text

dwillis commented 11 years ago

I'll check on the House assignments.

konklone commented 11 years ago

The House Clerk has posted two things, as of April 16 - an HTML list of (top-level) committee assignments, and a PDF of all committee assignments, including subcommittees.

The PDF is text-based and appears pretty scrapable - but geez, I'd rather not. Still, unless anybody has any alternative suggestions, it's probably best to assume these are the best we'll get in the near future.

dwillis commented 11 years ago

In previous years, this page would have links to the committees that would contain assignments, including subcommittees. So that might be coming, and would be easier to handle than the PDF, I'd think.

JoshData commented 11 years ago

I noticed recently there is committee info buried:

http://clerk.house.gov/member_info/mem_contact_info.aspx?statdis=CA20 http://clerk.house.gov/committee_info/index.aspx?comcode=AP00

But I haven't checked if it's up to date.

dwillis commented 11 years ago

AHA! That is indeed up to date.

konklone commented 11 years ago

Yeah, I just did some spot checks and it does look up to date. Thanks for saving Christmas, Josh!

On Mon, Apr 29, 2013 at 3:31 PM, Derek Willis notifications@github.comwrote:

AHA! That is indeed up to date.

— Reply to this email directly or view it on GitHubhttps://github.com/unitedstates/congress-legislators/issues/36#issuecomment-17188718 .

Developer | sunlightfoundation.com

JoshData commented 11 years ago

For all we know this has been available all Congress.....

JoshData commented 11 years ago

I just overwrote the committee membership file with info from the House Clerk pages: https://github.com/unitedstates/congress-legislators/commit/e883c307734ca936084c3651daeb13f471ff5c21

It was hard to verify that nothing is worse off now because the diff was not very friendly. I think we gained a few missing subcommittees' memberships, but we must have lost JSEC House members.

I'm thinking of closing this issue since I don't know that we can or want to push this toward completeness for joint committees. Maybe we'll open another issue for House members of joint committees if someone cares enough to work on it?