openstates / openstates-scrapers

source for Open States scrapers
https://openstates.org
GNU General Public License v3.0
845 stars 465 forks source link

Missing IL State Senators Email Addresses #1166

Closed gitsaxe closed 7 years ago

gitsaxe commented 7 years ago

@mileswwatkins [ { "ILL000001":"pamela@pamelaalthoff.net" }, { "IDL000140":"jacob36thdistrict@gmail.com " }, { "ILL001036":"Aquino.senate2il@gmail.com" }, { "ILL000062": "jason@jasonbarickman.com" }, { "ILL000998":"phardge@senatedan.ilga.gov" }, { "ILL000909":"bertinotarrant49@att.net" }, { "ILL000066":"biss@senatedem.illinois.gov" }, { "ILL000002":"senatorbivins@grics.net" }, { "ILL000004":"billbrady@senatorbillbrady.com" }, { "ILL000913":"info@senatormelindabush.com" }, { "ILL000005":"clayborne@senatedem.state.il.us" }, { "ILL000007":"Collins@senatedem.state.il.us" }, { "ILL000081":"senatorconnelly21@gmail.com" }, { "ILL000918":"tom@senatortomcullerton.com" }, { "ILL000085":"bill@billcunningham.com" }, { "ILL000011":"repdelgado@aol.com" }, { "ILL000014":"forby@senatedem.state.il.us" }, { "ILL000017":"haine@senatedem.state.il.us" }, { "ILL000018":"harmon@senatedem.state.il.us" }, { "ILL000927":"senatorharrisdistrictoffice@gmail.com" }, { "ILL000928":"district19@senatorhastings.com" }, { "ILL000019":"holmes42@senatedem.illinois.gov" }, { "ILL000020":"hunter@senatedem.state.il.us" }, { "ILL000021":"thutchinson@senatedem.ilga.gov" }, { "ILL000027":"46illinois@gmail.com" }, { "ILL000030":"senatorlandek@gmail.com" }, { "ILL000032":"statesenatorlightford@comcast.net" }, { "ILL000033":"senator@link30.org" }, { "ILL000034":"sendavel@midwest.net" }, { "ILL000941":"manar@senatedem.illinois.gov" }, { "ILL000036":"ilsenate20@aol.com" }, { "ILL000037":"SenatorSam@frontier.com" }, { "ILL000038":"senatormccarter@gmail.com" }, { "ILL001035":"dan@danmcconchie.com" }, { "ILL000945":"senator@karenmcconnaughay.com" }, { "ILL000533":"senatorpatmcguire@sbcglobal.net" }, { "ILL000947":"ilsenate29@gmail.com" }, { "ILL000041":"senatorjohnmulroe@att.net" }, { "ILL000043":"SenatorMattMurphy@gmail.com" }, { "ILL001006":"laura@senatorlauramurphy.com" }, { "ILL000042":"sen.amunoz@prodigy.net" }, { "ILL000044":"info@noland.org" }, { "ILL000143":"info@chrisnybo.com" }, { "ILL000952":"senatoroberweis@gmail.com" }, { "ILL000046":"christine@senatorradogno.com" }, { "ILL000047":"info@kwameraoul.com" }, { "ILL000048":"SenatorRezin@gmail.com" }, { "ILL000049":"drighter@consolidated.net" }, { "ILL001040":"senatortomrooney@gmail.com" }, { "ILL000154":"chapin@chapinrose.com" }, { "ILL000051":"msandoval@senatedem.ilga.gov" }, { "ILL000054":"eozier@senatedem.ilga.gov" }, { "ILL000960":"stadelman@senatedem.ilga.gov" }, { "ILL000055":"steans@senatedem.illinois.gov" }, { "ILL000056":"senjohn@adams.net" }, { "ILL000057":"info@senatordavesyverson.com" }, { "ILL000058":"donnetrotter@yahoo.com" }, { "ILL000965": "d.o.5senvanpelt@gmail.com" }, { "ILL001007":"chuck@senweaver.com" } ]

mileswwatkins commented 7 years ago

@fgregg, when you work on IL, if these two sets of emails aren't available on the IL legislative website then can you hard-code them in? Not our typical MO, but better than no contact info.

But wait for @gitsaxe to update these two tickets to include legislator ID, rather than matching on legislator name.

gitsaxe commented 7 years ago

I'm in this today. Thanks for putting back on my radar

Sent from my iPhone

On Dec 9, 2016, at 7:57 AM, Miles Watkins notifications@github.com wrote:

@fgregg, when you work on IL, if these two sets of emails aren't available on the IL legislative website then can you hard-code them in? Not our typical MO, but better than no contact info.

But wait for @gitsaxe to update these two tickets to include legislator ID, rather than matching on legislator name.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jamesturk commented 7 years ago

Just curious, what is the source for these?

gitsaxe commented 7 years ago

Hi James, Our interns call each legislator individually to get updated emails. I actually will have a new batch pretty soon as we have been getting some return to sender notices.

On Tue, Feb 7, 2017 at 11:19 PM, James Turk notifications@github.com wrote:

Just curious, what is the source for these?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstates/openstates/issues/1166#issuecomment-278232002, or mute the thread https://github.com/notifications/unsubscribe-auth/ACou7QvOvmae6e6qD9gc55PudlCvQgMwks5raVB_gaJpZM4LG9Sx .

cliftonmcintosh commented 7 years ago

Any idea if this list is accurate and current? I've sent an email to the organization asking if the list is current. http://kidsroe.org/kids/senators.cfm

gitsaxe commented 7 years ago

Taking a look now

mileswwatkins commented 7 years ago

BTW, we'd want to manually add this contact data as a one-off, as opposed to scraping from a third-party website; OS policy is to only use official sources for continually-scraped info, mostly so that we have a very good reason whenever we publish bad data ;)

gitsaxe commented 7 years ago

Our data is 85% OS data but I manually override email addresses when we receive return to sender notifications with email addresses we get from calling legislators.

@cliftonmcintosh I cross referenced the list above with the entries we have gotten from calling legislators. Hope it it helps:

Pamela Althoff Kids Roe - palthoff@mc.net We updated to pamela@pamelaalthoff.net

Bill Brady Kids Roe - billbrady@senbillbrady.com We updated to billbrady@senatorbillbrady.com

James Clayborne Kids Roe - clayborne@senatedem.state.il.us We updated to clayborne@senatedem.illinois.gov

Jacqui Collins Kids Roe - collins@senatedem.state.il.us We updated to senatorcollins@sbcglobal.net

John Cullerton Kids Roe - statesencullerton@msn.com We updated to jcullerton@senatedem.illinois.gov

William Haine Kids Roe - senhaine@cbnstl.com We updated to aine@senatedem.illinois.gov

Mattie Huner Kids Roe -hunter@senatedem.state.il.us We updated to senator03district@gmail.com

Kimberly Lightford Kids roe - lightford@senatedem.state.il.us We updated to statesenatorlightford@comcast.net

Antonio Munoz Kids Roe - munoz@senatedem.state.il.us We updated to statesenatorlightford@comcast.net

Dale Righter Kids Roe - drighter@mcleodusa.net We updated to drighter@consolidated.net

Martin Sandoval Kids Roe - sandoval@senatedem.state.il.us We updated to msandoval@senatedem.ilga.gov

Ira Silverstein Kids Roe - silverstein@senatedem.state.il.us We updated to eozier@senatedem.ilga.gov

John Sullivan Kids Roe - Sullivan@senatedem.state.il.us We updated to senjohn@adams.net

Donne E. Trotter Kids Roe - donneEtrotter@juno.com We updated to donnetrotter@yahoo.com

cliftonmcintosh commented 7 years ago

@gitsaxe

I can't really vouch for the list from Kids Roe. I was just wondering if it might be useful. Looking at it again, I realize it is probably old. In fact, it lists Barack Obama as a state senator! So perhaps it's not a list that will be very useful.

jamesturk commented 7 years ago

Wanted to bump this, @gitsaxe you mentioned you might have some updated data?

samtregar commented 7 years ago

Any chance this is happening soon? These emails look right in a spot-check but they're scattered all over the place on senator's personal pages. I don't see any obvious way to scrape them.

jamesturk commented 7 years ago

I just did a bulk manual update w/ these, for posterity:

emails = [
    {"ILL000060": "eacevedoed@ilga.gov"},
    {"ILL000985": "Assistance@StateRepCarolAmmons.com"},
    {"ILL000997": "steve@staterep65.com"},
    {"ILL000971": "info@staterep40.com"},
    {"ILL000061": "RepDistrict3@gmail.com"},
    {"ILL001000": "batinick@ilhousegop.org"},
    {"ILL000063": "dvbeiser@sbcglobal.net"},
    {"ILL000064": "rep@pbellock.com"},
    {"ILL000988": "bennett@ilhousegop.org"},
    {"ILL001003": "bourne@ilhousegop.org"},
    {"ILL000068": "repjohnbradley@mychoice.net"},
    {"ILL000069": "dan@rep-danbrady.com"},
    {"ILL000987": "peter@reppeterbreen.org"},
    {"ILL000071": "staterepbrown@gmail.com"},
    {"ILL000991": "staterepterribryant@gmail.com"},
    {"ILL000072": "burkedj@ilga.gov"},
    {"ILL000073": "kellyb@ilga.gov"},
    {"ILL001004": "butler@ilhousegop.org"},
    {"ILL000075": "Repcassidy@gmail.com"},
    {"ILL000076": "john@johncavaletto.com"},
    {"ILL000077": "lchapalavia@ilga.gov"},
    {"ILL000862": "staterepcloonen79@att.net"},
    {"ILL000863": "repdebconroy@gmail.com"},
    {"ILL000082": "staterepcostello@gmail.com"},
    {"ILL000083": "fred@fredcrespo.com"},
    {"ILL000086": "repcurrie@sbcglobal.net"},
    {"ILL000087": "john_damico@sbcglobal.net"},
    {"ILL000865": "repcddavidsmeyer@gmail.com"},
    {"ILL000088": "mdavis@ilga.gov"},
    {"ILL000089": "wdavis@ilga.gov"},
    {"ILL000090": "repdeluca@sbcglobal.net"},
    {"ILL000093": "info@repkendunkin.com"},
    {"ILL000094": "repdurkin@hotmail.com"},
    {"ILL000535": "Repevans33@gmail.com"},
    {"ILL000098": "sara@staterepsara.com"},
    {"ILL000100": "repford@lashawnford.com"},
    {"ILL000101": "mike.fortner@sbcglobal.net"},
    {"ILL000109": "greg@gregharris.org"},
    {"ILL000111": "chad@rephays.com"},
    {"ILL000112": "repehernandez@yahoo.com"},
    {"ILL000287": "Jay@JayHoffman.com"},
    {"ILL000875": "repfranhurley@gmail.com"},
    {"ILL000115": "eljackson34@gmail.com"},
    {"ILL001010": "Sjimenez@ilhousegop.org"},
    {"ILL000118": "repjones.jones@gmail.com"},
    {"ILL000119": "dwightkay112@gmail.com"},
    {"ILL000122": "repdavidleitch@gmail.com"},
    {"ILL000123": "statereplilly@yahoo.com"},
    {"ILL000880": "repmartwick@gmail.com"},
    {"ILL000129": "60thdistrict@gmail.com"},
    {"ILL000130": "repemily@gmail.com"},
    {"ILL000131": "mmcauliffe20@yahoo.com"},
    {"ILL000884": "mitchelldistrict26@att.net"},
    {"ILL000137": "moffitt@midwest.net"},
    {"ILL000990": "staterepphillips@gmail.com"},
    {"ILL000153": "robertbobrita@aol.com"},
    {"ILL000541": "Repsims34@gmail.com"},
    {"ILL001012": "Repskoog@gmail.com"},
    {"ILL000164": "repsosnowski@gmail.com"},
    {"ILL000165": "4repsoto@gmail.com"},
    {"ILL000898": "rep.tabares@gmail.com"},
    {"ILL000167": "illinois32district@gmail.com"},
    {"ILL000169": "Mike@miketryon.com"},
    {"ILL000170": "arthurt@ilga.gov"},
    {"ILL000983": "Litesa@StateRepWallace.com"},
    {"ILL000989": "office@repkeithwheeler.org"},
    {"ILL000996": "winger@ilhousegop.org"},
    {"ILL000177": "michaelz@ilga.gov"}] + [
        {"ILL000001": "pamela@pamelaalthoff.net"},
    {"IDL000140": "jacob36thdistrict@gmail.com"},
    {"ILL001036": "Aquino.senate2il@gmail.com"},
    {"ILL000062": "jason@jasonbarickman.com"},
    {"ILL000998": "phardge@senatedan.ilga.gov"},
    {"ILL000909": "bertinotarrant49@att.net"},
    {"ILL000066": "biss@senatedem.illinois.gov"},
    {"ILL000002": "senatorbivins@grics.net"},
    {"ILL000004": "billbrady@senatorbillbrady.com"},
    {"ILL000913": "info@senatormelindabush.com"},
    {"ILL000005": "clayborne@senatedem.state.il.us"},
    {"ILL000007": "Collins@senatedem.state.il.us"},
    {"ILL000081": "senatorconnelly21@gmail.com"},
    {"ILL000918": "tom@senatortomcullerton.com"},
    {"ILL000085": "bill@billcunningham.com"},
    {"ILL000011": "repdelgado@aol.com"},
    {"ILL000014": "forby@senatedem.state.il.us"},
    {"ILL000017": "haine@senatedem.state.il.us"},
    {"ILL000018": "harmon@senatedem.state.il.us"},
    {"ILL000927": "senatorharrisdistrictoffice@gmail.com"},
    {"ILL000928": "district19@senatorhastings.com"},
    {"ILL000019": "holmes42@senatedem.illinois.gov"},
    {"ILL000020": "hunter@senatedem.state.il.us"},
    {"ILL000021": "thutchinson@senatedem.ilga.gov"},
    {"ILL000027": "46illinois@gmail.com"},
    {"ILL000030": "senatorlandek@gmail.com"},
    {"ILL000032": "statesenatorlightford@comcast.net"},
    {"ILL000033": "senator@link30.org"},
    {"ILL000034": "sendavel@midwest.net"},
    {"ILL000941": "manar@senatedem.illinois.gov"},
    {"ILL000036": "ilsenate20@aol.com"},
    {"ILL000037": "SenatorSam@frontier.com"},
    {"ILL000038": "senatormccarter@gmail.com"},
    {"ILL001035": "dan@danmcconchie.com"},
    {"ILL000945": "senator@karenmcconnaughay.com"},
    {"ILL000533": "senatorpatmcguire@sbcglobal.net"},
    {"ILL000947": "ilsenate29@gmail.com"},
    {"ILL000041": "senatorjohnmulroe@att.net"},
    {"ILL000043": "SenatorMattMurphy@gmail.com"},
    {"ILL001006": "laura@senatorlauramurphy.com"},
    {"ILL000042": "sen.amunoz@prodigy.net"},
    {"ILL000044": "info@noland.org"},
    {"ILL000143": "info@chrisnybo.com"},
    {"ILL000952": "senatoroberweis@gmail.com"},
    {"ILL000046": "christine@senatorradogno.com"},
    {"ILL000047": "info@kwameraoul.com"},
    {"ILL000048": "SenatorRezin@gmail.com"},
    {"ILL000049": "drighter@consolidated.net"},
    {"ILL001040": "senatortomrooney@gmail.com"},
    {"ILL000154": "chapin@chapinrose.com"},
    {"ILL000051": "msandoval@senatedem.ilga.gov"},
    {"ILL000054": "eozier@senatedem.ilga.gov"},
    {"ILL000960": "stadelman@senatedem.ilga.gov"},
    {"ILL000055": "steans@senatedem.illinois.gov"},
    {"ILL000056": "senjohn@adams.net"},
    {"ILL000057": "info@senatordavesyverson.com"},
    {"ILL000058": "donnetrotter@yahoo.com"},
    {"ILL000965": "d.o.5senvanpelt@gmail.com"},
    {"ILL001007": "chuck@senweaver.com"}
]

for x in emails:
    id, email = list(x.items())[0]
    leg = db.legislators.find_one({'_id': id})
    leg['email'] = email
    db.legislators.save(leg)