search

openstates / openstates-scrapers

source for Open States scrapers
https://openstates.org
GNU General Public License v3.0
846 stars 464 forks source link

NY: scraper fails on William Magee #1608

Closed lavaturtle closed 7 years ago

lavaturtle commented 7 years ago

State: NY

When I run the New York scraper, it errors out -- looks like it's hitting an error trying to read addresses from http://assembly.state.ny.us/mem/William-Magee

jacinda@osskil:~/code/openstates$ docker-compose run --rm openstates ny --legislators --fast
Pulling openstates (openstates/openstates:latest)...
latest: Pulling from openstates/openstates
693502eb7dfb: Pull complete
ee872be79472: Pull complete
ebef274f55c6: Pull complete
f148c3f94fab: Pull complete
bcf47cb3df80: Pull complete
ca0b5c206b8e: Pull complete
c22702e0d705: Pull complete
ba7bb8368e56: Pull complete
89e82161bca2: Pull complete
223d2575e767: Extracting [====================================>              ]   176 MB/241.4 MB
29995bd15142: Download complete
d36af9d8deb7: Download complete
20d8e31c807a: Download complete
223d2575e767: Pull complete
29995bd15142: Pull complete
d36af9d8deb7: Pull complete
3c062cfbd4e7: Pull complete
8ddb108e0357: Pull complete
Digest: sha256:49c0d54f72dc5b5176f922c1f4154c43318dbd493c04113426e45050541bccee
Status: Downloaded newer image for openstates/openstates:latest
15:00:39 INFO billy: billy-update abbr=ny
    actions=scrape,import,report
    types=legislators
    sessions=2017-2018
    terms=2017-2018
15:00:42 INFO scrapelib: GET - http://www.nysenate.gov/senators-committees
15:00:42 INFO scrapelib: GET - http://www.nysenate.gov/senators/joseph-p-addabbo-jr/contact
15:00:43 INFO billy: Save person Joseph P. Addabbo Jr
15:00:43 INFO scrapelib: GET - http://www.nysenate.gov/senators/fred-akshar/contact
15:00:44 INFO billy: Save person Fred Akshar
15:00:44 INFO scrapelib: GET - http://www.nysenate.gov/senators/marisol-alcantara/contact
15:00:45 INFO billy: Save person Marisol Alcantara
15:00:45 INFO scrapelib: GET - http://www.nysenate.gov/senators/george-amedore-jr/contact
15:00:45 INFO billy: Save person George A. Amedore, Jr.
15:00:45 INFO scrapelib: GET - http://www.nysenate.gov/senators/tony-avella/contact
15:00:47 INFO billy: Save person Tony Avella
15:00:47 INFO scrapelib: GET - http://www.nysenate.gov/senators/jamaal-bailey/contact
15:00:49 INFO billy: Save person Jamaal Bailey
15:00:49 INFO scrapelib: GET - http://www.nysenate.gov/senators/john-j-bonacic/contact
15:00:50 INFO billy: Save person John J. Bonacic
15:00:50 INFO scrapelib: GET - http://www.nysenate.gov/senators/phil-boyle/contact
15:00:50 INFO billy: Save person Phil Boyle
15:00:50 INFO scrapelib: GET - http://www.nysenate.gov/senators/neil-d-breslin/contact
15:00:52 INFO billy: Save person Neil D. Breslin
15:00:52 INFO scrapelib: GET - http://www.nysenate.gov/senators/john-e-brooks/contact
15:00:52 INFO billy: Save person John E. Brooks
15:00:52 INFO scrapelib: GET - http://www.nysenate.gov/senators/david-carlucci/contact
15:00:52 INFO billy: Save person David Carlucci
15:00:52 INFO scrapelib: GET - http://www.nysenate.gov/senators/leroy-comrie/contact
15:00:52 INFO billy: Save person Leroy Comrie
15:00:52 INFO scrapelib: GET - http://www.nysenate.gov/senators/thomas-d-croci/contact
15:00:54 INFO billy: Save person Thomas D. Croci
15:00:54 INFO scrapelib: GET - http://www.nysenate.gov/senators/john-defrancisco/contact
15:00:55 INFO billy: Save person John A. DeFrancisco
15:00:55 INFO scrapelib: GET - http://www.nysenate.gov/senators/ruben-diaz/contact
15:00:56 INFO billy: Save person Ruben Diaz
15:00:56 INFO scrapelib: GET - http://www.nysenate.gov/senators/martin-malav%C3%A9-dilan/contact
15:00:57 INFO billy: Save person Martin Malavé Dilan
15:00:57 INFO scrapelib: GET - http://www.nysenate.gov/senators/simcha-felder/contact
15:00:57 INFO billy: Save person Simcha Felder
15:00:57 INFO scrapelib: GET - http://www.nysenate.gov/senators/john-j-flanagan/contact
15:00:59 INFO billy: Save person John J. Flanagan
15:00:59 INFO scrapelib: GET - http://www.nysenate.gov/senators/rich-funke/contact
15:01:00 INFO billy: Save person Rich Funke
15:01:00 INFO scrapelib: GET - http://www.nysenate.gov/senators/patrick-m-gallivan/contact
15:01:02 INFO billy: Save person Patrick M. Gallivan
15:01:02 INFO scrapelib: GET - http://www.nysenate.gov/senators/michael-gianaris/contact
15:01:04 INFO billy: Save person Michael Gianaris
15:01:04 INFO scrapelib: GET - http://www.nysenate.gov/senators/martin-j-golden/contact
15:01:04 INFO billy: Save person Martin J. Golden
15:01:04 INFO scrapelib: GET - http://www.nysenate.gov/senators/joseph-griffo/contact
15:01:04 INFO billy: Save person Joseph A. Griffo
15:01:04 INFO scrapelib: GET - http://www.nysenate.gov/senators/jesse-hamilton/contact
15:01:06 INFO billy: Save person Jesse Hamilton
15:01:06 INFO scrapelib: GET - http://www.nysenate.gov/senators/kemp-hannon/contact
15:01:06 INFO billy: Save person Kemp Hannon
15:01:06 INFO scrapelib: GET - http://www.nysenate.gov/senators/pamela-helming/contact
15:01:08 INFO billy: Save person Pamela Helming
15:01:08 INFO scrapelib: GET - http://www.nysenate.gov/senators/brad-hoylman/contact
15:01:08 INFO billy: Save person Brad Hoylman
15:01:08 INFO scrapelib: GET - http://www.nysenate.gov/senators/christopher-jacobs/contact
15:01:10 INFO billy: Save person Chris Jacobs
15:01:10 INFO scrapelib: GET - http://www.nysenate.gov/senators/todd-kaminsky/contact
15:01:10 INFO billy: Save person Todd Kaminsky
15:01:10 INFO scrapelib: GET - http://www.nysenate.gov/senators/timothy-m-kennedy/contact
15:01:12 INFO billy: Save person Timothy M. Kennedy
15:01:12 INFO scrapelib: GET - http://www.nysenate.gov/senators/jeffrey-d-klein/contact
15:01:14 INFO billy: Save person Jeffrey D. Klein
15:01:14 INFO scrapelib: GET - http://www.nysenate.gov/senators/liz-krueger/contact
15:01:15 INFO billy: Save person Liz Krueger
15:01:15 INFO scrapelib: GET - http://www.nysenate.gov/senators/andrew-j-lanza/contact
15:01:17 INFO billy: Save person Andrew J Lanza
15:01:17 INFO scrapelib: GET - http://www.nysenate.gov/senators/william-j-larkin-jr/contact
15:01:18 INFO billy: Save person William J. Larkin, Jr.
15:01:18 INFO scrapelib: GET - http://www.nysenate.gov/senators/george-latimer/contact
15:01:18 INFO billy: Save person George Latimer
15:01:18 INFO scrapelib: GET - http://www.nysenate.gov/senators/kenneth-p-lavalle/contact
15:01:19 INFO billy: Save person Kenneth P. LaValle
15:01:19 INFO scrapelib: GET - http://www.nysenate.gov/senators/betty-little/contact
15:01:20 INFO billy: Save person Betty Little
15:01:20 INFO scrapelib: GET - http://www.nysenate.gov/senators/carl-l-marcellino/contact
15:01:22 INFO billy: Save person Carl L Marcellino
15:01:22 INFO scrapelib: GET - http://www.nysenate.gov/senators/kathleen-marchione/contact
15:01:24 INFO billy: Save person Kathleen A. Marchione
15:01:24 INFO scrapelib: GET - http://www.nysenate.gov/senators/velmanette-montgomery/contact
15:01:26 INFO billy: Save person Velmanette Montgomery
15:01:26 INFO scrapelib: GET - http://www.nysenate.gov/senators/terrence-murphy/contact
15:01:26 INFO billy: Save person Terrence Murphy
15:01:26 INFO scrapelib: GET - http://www.nysenate.gov/senators/thomas-f-omara/contact
15:01:27 INFO billy: Save person Thomas F. O'Mara
15:01:27 INFO scrapelib: GET - http://www.nysenate.gov/senators/robert-g-ortt/contact
15:01:29 INFO billy: Save person Robert G. Ortt
15:01:29 INFO scrapelib: GET - http://www.nysenate.gov/senators/kevin-s-parker/contact
15:01:31 INFO billy: Save person Kevin S. Parker
15:01:31 INFO scrapelib: GET - http://www.nysenate.gov/senators/jose-peralta/contact
15:01:33 INFO billy: Save person Jose Peralta
15:01:33 INFO scrapelib: GET - http://www.nysenate.gov/senators/roxanne-j-persaud/contact
15:01:35 INFO billy: Save person Roxanne J. Persaud
15:01:35 INFO scrapelib: GET - http://www.nysenate.gov/senators/elaine-phillips/contact
15:01:37 INFO billy: Save person Elaine Phillips
15:01:37 INFO scrapelib: GET - http://www.nysenate.gov/senators/michael-h-ranzenhofer/contact
15:01:39 INFO billy: Save person Michael H. Ranzenhofer
15:01:39 INFO scrapelib: GET - http://www.nysenate.gov/senators/patty-ritchie/contact
15:01:41 INFO billy: Save person Patty Ritchie
15:01:41 INFO scrapelib: GET - http://www.nysenate.gov/senators/gustavo-rivera/contact
15:01:42 INFO billy: Save person Gustavo Rivera
15:01:42 INFO scrapelib: GET - http://www.nysenate.gov/senators/joseph-e-robach/contact
15:01:44 INFO billy: Save person Joseph E. Robach
15:01:44 INFO scrapelib: GET - http://www.nysenate.gov/senators/james-sanders-jr/contact
15:01:45 INFO billy: Save person James Sanders Jr.
15:01:45 INFO scrapelib: GET - http://www.nysenate.gov/senators/diane-j-savino/contact
15:01:47 INFO billy: Save person Diane J. Savino
15:01:47 INFO scrapelib: GET - http://www.nysenate.gov/senators/sue-serino/contact
15:01:48 INFO billy: Save person Sue Serino
15:01:48 INFO scrapelib: GET - http://www.nysenate.gov/senators/jos%C3%A9-m-serrano/contact
15:01:50 INFO billy: Save person José M. Serrano
15:01:50 INFO scrapelib: GET - http://www.nysenate.gov/senators/james-l-seward/contact
15:01:51 INFO billy: Save person James L. Seward
15:01:51 INFO scrapelib: GET - http://www.nysenate.gov/senators/daniel-l-squadron/contact
15:01:53 INFO billy: Save person Daniel L. Squadron
15:01:53 INFO scrapelib: GET - http://www.nysenate.gov/senators/toby-ann-stavisky/contact
15:01:53 INFO billy: Save person Toby Ann Stavisky
15:01:53 INFO scrapelib: GET - http://www.nysenate.gov/senators/andrea-stewart-cousins/contact
15:01:54 INFO billy: Save person Andrea Stewart-Cousins
15:01:54 INFO scrapelib: GET - http://www.nysenate.gov/senators/james-tedisco/contact
15:01:56 INFO billy: Save person James Tedisco
15:01:56 INFO scrapelib: GET - http://www.nysenate.gov/senators/david-j-valesky/contact
15:01:56 INFO billy: Save person David J. Valesky
15:01:56 INFO scrapelib: GET - http://www.nysenate.gov/senators/catharine-young/contact
15:01:58 INFO billy: Save person Catharine Young
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/?sh=email
15:01:58 INFO scrapelib: GET - http://www.elections.ny.gov:8080/reports/rwservlet?cmdkey=nysboe_incumbnt
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Peter-J-Abbate-Jr
15:01:58 INFO billy: Save person Abbate, Jr., Peter
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Thomas-J-Abinanti
15:01:58 INFO billy: Save person Abinanti, Thomas
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Carmen-E-Arroyo
15:01:58 INFO billy: Save person Arroyo, Carmen
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Jeffrion-L-Aubry
15:01:58 INFO billy: Save person Aubry, Jeffrion
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/William-A-Barclay
15:01:58 INFO billy: Save person Barclay, William
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Didi-Barrett
15:01:58 INFO billy: Save person Barrett, Didi
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Charles-Barron
15:01:58 INFO billy: Save person Barron , Charles
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Michael-Benedetto
15:01:58 INFO billy: Save person Benedetto, Michael
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Rodneyse-Bichotte
15:01:58 INFO billy: Save person Bichotte, Rodneyse
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Michael-Blake
15:01:58 INFO billy: Save person Blake, Michael
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Ken-Blankenbush
15:01:58 INFO billy: Save person Blankenbush, Ken
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Joseph-Borelli
15:01:58 INFO billy: Save person Borelli, Joseph
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Karl-Brabenec
15:01:58 INFO billy: Save person Brabenec, Karl
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Edward-C-Braunstein
15:01:58 INFO billy: Save person Braunstein, Edward
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/James-F-Brennan
15:01:58 INFO billy: Save person Brennan, James
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Anthony-Brindisi
15:01:58 INFO billy: Save person Brindisi, Anthony
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Harry-B-Bronson
15:01:58 INFO billy: Save person Bronson, Harry
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Alec-Brook-Krasny
15:01:58 INFO billy: Save person Brook-Krasny, Alec
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/David-Buchwald
15:01:58 INFO billy: Save person Buchwald, David
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Marc-W-Butler
15:01:58 INFO billy: Save person Butler, Marc
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Kevin-A-Cahill
15:01:58 INFO billy: Save person Cahill, Kevin
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Karim-Camara
15:01:58 INFO billy: Save person Camara, Karim
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/John-D-Ceretto
15:01:58 INFO billy: Save person Ceretto, John
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Barbara-M-Clark
15:01:58 INFO billy: Save person Clark, Barbara
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/William-Colton
15:01:58 INFO billy: Save person Colton, William
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Vivian-E-Cook
15:01:58 INFO billy: Save person Cook, Vivian
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Jane-L-Corwin
15:01:58 INFO billy: Save person Corwin, Jane
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Marcos-A-Crespo
15:01:58 INFO billy: Save person Crespo, Marcos
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Clifford-W-Crouch
15:01:58 INFO billy: Save person Crouch, Clifford
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Brian-Curran
15:01:58 INFO billy: Save person Curran, Brian
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Michael-Cusick
15:01:58 INFO billy: Save person Cusick, Michael
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Steven-Cymbrowitz
15:01:58 INFO billy: Save person Cymbrowitz, Steven
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Maritza-Davila
15:01:58 INFO billy: Save person Davila, Maritza
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Michael-G-DenDekker
15:01:58 INFO billy: Save person DenDekker, Michael
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Erik-M-Dilan
15:01:58 INFO billy: Save person Dilan, Erik
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Jeffrey-Dinowitz
15:01:58 INFO billy: Save person Dinowitz, Jeffrey
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/David-DiPietro
15:01:58 INFO billy: Save person DiPietro, David
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Janet-L-Duprey
15:01:58 INFO billy: Save person Duprey, Janet
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Steve-Englebright
15:01:58 INFO billy: Save person Englebright, Steve
15:01:58 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Patricia-Fahy
15:01:58 INFO billy: Save person Fahy, Patricia
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Herman-D-Farrell-Jr
15:01:59 INFO billy: Save person Farrell, Jr., Herman
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Gary-D-Finch
15:01:59 INFO billy: Save person Finch, Gary
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Michael-J-Fitzpatrick
15:01:59 INFO billy: Save person Fitzpatrick, Michael
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Christopher-S-Friend
15:01:59 INFO billy: Save person Friend, Christopher
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Sandy-Galef
15:01:59 INFO billy: Save person Galef, Sandy
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/David-F-Gantt
15:01:59 INFO billy: Save person Gantt, David
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Andrew-R-Garbarino
15:01:59 INFO billy: Save person Garbarino, Andrew
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Joseph-M-Giglio
15:01:59 INFO billy: Save person Giglio, Joseph
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Mark-Gjonaj
15:01:59 INFO billy: Save person Gjonaj, Mark
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Deborah-J-Glick
15:01:59 INFO billy: Save person Glick, Deborah
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Phillip-Goldfeder
15:01:59 INFO billy: Save person Goldfeder, Phillip
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Andy-Goodell
15:01:59 INFO billy: Save person Goodell, Andy
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Richard-N-Gottfried
15:01:59 INFO billy: Save person Gottfried, Richard
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Al-Graf
15:01:59 INFO billy: Save person Graf, Al
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Aileen-M-Gunther
15:01:59 INFO billy: Save person Gunther, Aileen
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Stephen-Hawley
15:01:59 INFO billy: Save person Hawley, Stephen
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Carl-E-Heastie
15:01:59 INFO billy: Save person Heastie, Carl
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Andrew-Hevesi
15:01:59 INFO billy: Save person Hevesi, Andrew
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Dov-Hikind
15:01:59 INFO billy: Save person Hikind, Dov
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Earlene-Hooper
15:01:59 INFO billy: Save person Hooper, Earlene
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Ellen-Jaffee
15:01:59 INFO billy: Save person Jaffee, Ellen
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Kimberly-Jean-Pierre
15:01:59 INFO billy: Save person Jean-Pierre, Kimberly
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Mark-Johns
15:01:59 INFO billy: Save person Johns, Mark
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Latoya-Joyner
15:01:59 INFO billy: Save person Joyner, Latoya
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Todd-Kaminsky
15:01:59 INFO billy: Save person Kaminsky, Todd
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Steve-Katz
15:01:59 INFO billy: Save person Katz, Steve
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Brian-Kavanagh
15:01:59 INFO billy: Save person Kavanagh, Brian
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Michael-P-Kearns
15:01:59 INFO billy: Save person Kearns, Michael
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Ron-Kim
15:01:59 INFO billy: Save person Kim, Ron
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Brian-M-Kolb
15:01:59 INFO billy: Save person Kolb, Brian M.
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Kieran-Michael-Lalor
15:01:59 INFO billy: Save person Lalor, Kieran Michael
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Charles-D-Lavine
15:01:59 INFO billy: Save person Lavine, Charles
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Peter-Lawrence
15:01:59 INFO billy: Save person Lawrence, Peter
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Joseph-R-Lentol
15:01:59 INFO billy: Save person Lentol, Joseph
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Barbara-Lifton
15:01:59 INFO billy: Save person Lifton, Barbara
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Guillermo-Linares
15:01:59 INFO billy: Save person Linares, Guillermo
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Peter-D-Lopez
15:01:59 INFO billy: Save person Lopez, Peter
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Donna-A-Lupardo
15:01:59 INFO billy: Save person Lupardo, Donna
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Chad-A-Lupinacci
15:01:59 INFO billy: Save person Lupinacci, Chad
15:01:59 INFO scrapelib: GET - http://assembly.state.ny.us/mem/William-Magee
Traceback (most recent call last):
  File "/usr/local/bin/billy-update", line 9, in <module>
    load_entry_point('billy==1.9.0', 'console_scripts', 'billy-update')()
  File "/opt/openstates/billy/billy/bin/update.py", line 368, in main
    run_record += _run_scraper(stype, args, metadata)
  File "/opt/openstates/billy/billy/bin/update.py", line 102, in _run_scraper
    scraper.scrape(chamber, time)
  File "/srv/openstates-web/openstates/ny/legislators.py", line 236, in scrape
    getattr(self, 'scrape_' + chamber + '_chamber')(term)
  File "/srv/openstates-web/openstates/ny/legislators.py", line 419, in scrape_lower_chamber
    self.scrape_lower_offices(legislator_url, legislator, email)
  File "/srv/openstates-web/openstates/ny/legislators.py", line 443, in scrape_lower_offices
    if address[-1].startswith("Fax: "):
IndexError: list index out of range
hiteshgarg14 commented 7 years ago

I ran it locally and it worked fine:- 

21:27:17 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Chad-A-Lupinacci
21:27:24 INFO billy: Save person Lupinacci, Chad
21:27:24 INFO scrapelib: GET - http://assembly.state.ny.us/mem/William-Magee
21:27:26 INFO billy: Save person Magee, William
21:27:26 INFO scrapelib: GET - http://assembly.state.ny.us/mem/William-B-Magnarelli
21:27:27 INFO billy: Save person Magnarelli, William
21:27:27 INFO scrapelib: GET - http://assembly.state.ny.us/mem/Nicole-Malliotakis

Finally:-

21:29:26 INFO billy: Finished importing 211 legislator files.
lavaturtle commented 7 years ago

This is strange! It looks like data.xpath('following-sibling::div[1]/text()') (on line 432 of legislators.py) is returning an empty list for that one address, when it seems like it should be returning ['Oneonta Phone', '\n607-432-1484'] (and presumably is returning that for everyone else).

lavaturtle commented 7 years ago

I suspect this has to be something weird about my environment, although I'm not sure what. I'm running Docker version 17.03.1-ce, build c6d412e and docker-compose version 1.12.0-rc2, build 08dc2a4 on Ubuntu 16.10 Yakkety.

jamesturk commented 7 years ago

If you run without --fast does this still happen? This may be a cached bad copy of a page locally. You can also wipe out the cache/_cache dirs

lavaturtle commented 7 years ago

Without --fast it works! Thanks!