Committee scraper

[schmod]

A lot of my work revolves around committee hearings, and I've noticed that GovTrack, Sunlight, NYT, and the other usual suspects only track committee activity at the most basic levels. It seems like this very important part of our legislative process gets routinely ignored.

It'd be pretty awesome to have a scraper that aggregated all committee-related activity into an easily-digestible data source.

Right now, the data is tied up in fdsys, committee sites, and the daily digest. There's absolutely no good way to automatically cross-reference any of it.

I don't have a specific proposal to put forward, but a congress-committees repository might be a valuable thing. A lot of the groundwork has already been done in congress-legislators.

Things we might want to aggregate:

• Committee metadata (already in unitedstates/congress-legislators)
• (Recent) Historical committee assignments (already being worked on)
• Individual Hearings
  • Summary
  • Members present
  • Bills discussed (if any)
  • Witness list
  • Links to hearing report (fdsys)
  • Links to witness testimony, questions, and submissions
  • Links to transcript (if available)
  • Link to webcast
• Committee reports (govtrack already has this -- maybe we could mine some data out of them?)

  Possible data sources:

• fdsys has PDF hearing reports w/ ample XML metadata attached. A good place to start, but hearing reports often aren't published until 6-8 months after the fact.
• The Daily Digest includes hearing schedules and summaries. Not a ton of information, but easy to grab and parse.
• Individual committee sites. Have the most information, but also the most difficult to parse. A few offer RSS feeds with varying levels of usefulness.
• Committee Reports. Like hearing reports, are fairly stale by the time they're published.

Again, this isn't a concrete proposal, but it's something that I've been tossing around in the back of my head.

[konklone]

I'm a huge proponent of committee information. Committee voting records, in particular, never surface in any free data source, because of how difficult they are to collect.

You've described really well the disparate nature of the data - which is why it hasn't been done yet. Is this something you'd be interested in leading the way on?

I'll note that the House's new Committee Repository has a lot of data now, and a lot of potential. It may have some of what you've asked for above, for individual hearings. The Clerk has also been explicit about growing its scope to include here-to-fore unpublished or decentralized information, such as voting records.