search

openstates / openstates-scrapers

source for Open States scrapers
https://openstates.org
GNU General Public License v3.0
845 stars 462 forks source link

TX: Run committee scraper for lower chamber #1365

Closed grgcombs closed 7 years ago

grgcombs commented 7 years ago

Lower chamber has committees as of Feb 9th. . When does the API scraper run? If it's every night, the scraper isn't picking them up for some reason.

http://www.legis.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C450

grgcombs commented 7 years ago

I picked up Docker for Mac and installed the OpenStates scrapers. (The switch to Docker has made this infinitely easier than in years past).

In any event, the TX committee scraper doesn't pick up any membership for the lower chamber committees. I don't know enough about python to decipher where things go off the rails, but it is completing without any errors or exceptions. It scrapes the committee details (phone, location, etc), but isn't building the membership lists -- the membership array for each lower chamber committee is empty in the database. Not sure if there's some other configuration or pre-conditions that I need or if it's a code issue.

redNixon commented 7 years ago

I ran through this, a lot of pages get the members detected, some still do not have members listed on the pages though. What is odd(maybe I just understand this aspect of Billy), I already had the committees saved from when there were no members for any committee listed. Rerunning import over the committee did not update the database, but deleting the TX collection and reimporting allowed the members to get updated. Is this suppose to happen?

Here is some custom logging showing what I am detecting: INFO billy: Agriculture & Livestock: 7 members added INFO billy: Appropriations: 27 members added INFO billy: Appropriations - S/C on Article II: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C010 INFO billy: Appropriations - S/C on Article III: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C012 INFO billy: Appropriations - S/C on Articles I, IV & V: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C013 INFO billy: Appropriations - S/C on Articles VI, VII & VIII: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C014 INFO billy: Appropriations - S/C on Budget Transparency & Reform: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C016 INFO billy: Business & Industry: 7 members added INFO billy: Calendars: 15 members added INFO billy: Corrections: 7 members added INFO billy: County Affairs: 9 members added INFO billy: Criminal Jurisprudence: 7 members added INFO billy: Culture, Recreation & Tourism: 7 members added INFO billy: Defense & Veterans' Affairs: 7 members added INFO billy: Economic & Small Business Development: 9 members added INFO billy: Economic & Small Business Development-S/C on Small Business: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C237 INFO billy: Elections: 7 members added INFO billy: Energy Resources: 13 members added INFO billy: Environmental Regulation: 9 members added INFO billy: General Investigating & Ethics: 7 members added INFO billy: Government Transparency & Operation: 7 members added INFO billy: Higher Education: 9 members added INFO billy: Higher Education-S/C on Post-Secondary & Workforce Readiness: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C292 INFO billy: Homeland Security & Public Safety: 9 members added INFO billy: House Administration: 11 members added INFO billy: Human Services: 9 members added INFO billy: Insurance: 9 members added INFO billy: International Trade & Intergovernmental Affairs: 7 members added INFO billy: Investments & Financial Services: 7 members added INFO billy: Investments & Financial Services-S/C on Bond Indebtedness: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C272 INFO billy: Judiciary & Civil Jurisprudence: 9 members added INFO billy: Juvenile Justice & Family Issues: 7 members added INFO billy: Land & Resource Management: 7 members added INFO billy: Licensing & Administrative Procedures: 9 members added INFO billy: Local & Consent Calendars: 13 members added INFO billy: Natural Resources: 11 members added INFO billy: Natural Resources-S/C on Special Water Districts: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C392 INFO billy: Pensions: 7 members added INFO billy: Public Education: 11 members added INFO billy: Public Education-S/C on Educator Quality: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C402 INFO billy: Public Health: 11 members added INFO billy: Redistricting: 9 members added INFO billy: Rules & Resolutions: 11 members added INFO billy: Special Purpose Districts: 7 members added INFO billy: State & Federal Power & Responsibility, Select: 9 members added INFO billy: State Affairs: 13 members added INFO billy: Texas Ports, Innovation & Infrastructure, Select: 11 members added INFO billy: Transportation: 13 members added INFO billy: Transportation-S/C on Long-term Infrastructure Planning: No members detected - http://www.capitol.state.tx.us/Committees/MembershipCmte.aspx?LegSess=85R&CmteCode=C472 INFO billy: Urban Affairs: 7 members added INFO billy: Ways & Means: 11 members added

jamesturk commented 7 years ago

odd that it didn't get updated from that run, I'm not sure what that'd be...

TX lower comms in general look good now I think:

https://openstates.org/tx/committees/?chamber=lower

still a few w/o data, but I think those are upstream