search

openstates / issues

Having trouble? Looking to contribute? Issues live here!
15 stars 2 forks source link

CT: Vote data incorrect #233

Open mzagaja opened 3 years ago

mzagaja commented 3 years ago

This is the right template to select if you're reporting an issue with bill or vote data from Open States.

Issue Description: Vote information is putting data in incorrect fields. voterName is presenting as Y instead of the name as an actual voter.

URL where this issue may be seen (API or OpenStates.org):

## Bill Information
curl -X "POST" "https://openstates.org/graphql" \
     -H 'X-API-KEY: ' \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{
    "query": "query MyQuery{bills(jurisdiction:\\"Connecticut\\",last:5,updatedSince:\\"2020-01-01\\"){edges{node{identifier updatedAt createdAt legislativeSession{identifier}id votes{edges{node{id t startDate result votes{option voterName note}}}}}cursor}pageInfo{hasNextPage hasPreviousPage endCursor startCursor}totalCount}}",
    "variables": {}
}'
JSON Response ```json { "data": { "bills": { "edges": [ { "node": { "identifier": "HB 7229", "updatedAt": "2020-04-28 21:53:18.396230+00:00", "createdAt": "2017-07-08 05:36:26.458341+00:00", "legislativeSession": { "identifier": "2017" }, "id": "ocd-bill/c0dc3975-dd8b-4db6-9d52-298872d57b78", "votes": { "edges": [ { "node": { "id": "ocd-vote/949e8347-9118-4c3c-a959-fcca3a907918", "startDate": "2017-04-12", "result": "pass", "votes": [ { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" } ] } }, { "node": { "id": "ocd-vote/6481e2fe-494d-4dae-881e-813daf0d20f6", "startDate": "2017-06-07", "result": "pass", "votes": [ { "option": "yes", "voterName": "LEONARD FASANO", "note": "" }, { "option": "yes", "voterName": "GEORGE LOGAN", "note": "" }, { "option": "yes", "voterName": "ANTHONY GUGLIELMO", "note": "" }, { "option": "yes", "voterName": "HEATHER SOMERS", "note": "" }, { "option": "yes", "voterName": "JOHN W. FONFARA", "note": "" }, { "option": "yes", "voterName": "L. SCOTT FRANTZ", "note": "" }, { "option": "yes", "voterName": "CATHERINE A. OSTEN", "note": "" }, { "option": "yes", "voterName": "DOUGLAS MCCRORY", "note": "" }, { "option": "yes", "voterName": "PAUL M. FORMICA", "note": "" }, { "option": "yes", "voterName": "TIM LARSON", "note": "" }, { "option": "yes", "voterName": "KEVIN KELLY", "note": "" }, { "option": "yes", "voterName": "STEVE CASSANO", "note": "" }, { "option": "yes", "voterName": "MARILYN MOORE", "note": "" }, { "option": "yes", "voterName": "BETH BYE", "note": "" }, { "option": "yes", "voterName": "EDWIN A. GOMES", "note": "" }, { "option": "yes", "voterName": "TERRY B. GERRATANA", "note": "" }, { "option": "yes", "voterName": "MICHAEL A. MCLACHLAN", "note": "" }, { "option": "yes", "voterName": "JOHN A. KISSEL", "note": "" }, { "option": "yes", "voterName": "BOB DUFF", "note": "" }, { "option": "yes", "voterName": "KEVIN D. WITKOS", "note": "" }, { "option": "yes", "voterName": "TONI BOUCHER", "note": "" }, { "option": "yes", "voterName": "PAUL DOYLE", "note": "" }, { "option": "yes", "voterName": "CARLO LEONE", "note": "" }, { "option": "yes", "voterName": "GARY WINFIELD", "note": "" }, { "option": "yes", "voterName": "TONY HWANG", "note": "" }, { "option": "yes", "voterName": "MARTIN M. LOONEY", "note": "" }, { "option": "yes", "voterName": "MAE M. FLEXER", "note": "" }, { "option": "yes", "voterName": "TED KENNEDY", "note": "" }, { "option": "yes", "voterName": "CRAIG MINER", "note": "" }, { "option": "yes", "voterName": "LEN SUZIO", "note": "" }, { "option": "yes", "voterName": "HENRI MARTIN", "note": "" }, { "option": "yes", "voterName": "GAYLE SLOSSBERG", "note": "" }, { "option": "yes", "voterName": "ERIC BERTHEL", "note": "" }, { "option": "yes", "voterName": "JOAN V. HARTLEY", "note": "" }, { "option": "yes", "voterName": "ART LINARES", "note": "" }, { "option": "yes", "voterName": "JOE MARKLEY", "note": "" } ] } } ] } }, "cursor": "YXJyYXljb25uZWN0aW9uOjEwNTc5" }, { "node": { "identifier": "HB 7212", "updatedAt": "2020-04-28 21:53:18.377837+00:00", "createdAt": "2017-07-08 05:36:33.781543+00:00", "legislativeSession": { "identifier": "2017" }, "id": "ocd-bill/8c92ca39-1512-46ac-867c-300db4edb769", "votes": { "edges": [ { "node": { "id": "ocd-vote/3b2514e8-4f99-4e76-987f-5402a5d40071", "startDate": "2017-05-16", "result": "pass", "votes": [ { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" } ] } }, { "node": { "id": "ocd-vote/abcfb0df-78f2-499b-92ac-e3f679786df4", "startDate": "2017-06-07", "result": "pass", "votes": [ { "option": "yes", "voterName": "MARILYN MOORE", "note": "" }, { "option": "yes", "voterName": "JOHN W. FONFARA", "note": "" }, { "option": "yes", "voterName": "CATHERINE A. OSTEN", "note": "" }, { "option": "yes", "voterName": "DOUGLAS MCCRORY", "note": "" }, { "option": "yes", "voterName": "PAUL M. FORMICA", "note": "" }, { "option": "yes", "voterName": "TIM LARSON", "note": "" }, { "option": "yes", "voterName": "KEVIN KELLY", "note": "" }, { "option": "yes", "voterName": "STEVE CASSANO", "note": "" }, { "option": "yes", "voterName": "BETH BYE", "note": "" }, { "option": "yes", "voterName": "EDWIN A. GOMES", "note": "" }, { "option": "yes", "voterName": "TERRY B. GERRATANA", "note": "" }, { "option": "yes", "voterName": "MICHAEL A. MCLACHLAN", "note": "" }, { "option": "yes", "voterName": "JOHN A. KISSEL", "note": "" }, { "option": "yes", "voterName": "BOB DUFF", "note": "" }, { "option": "yes", "voterName": "KEVIN D. WITKOS", "note": "" }, { "option": "yes", "voterName": "TONI BOUCHER", "note": "" }, { "option": "yes", "voterName": "PAUL DOYLE", "note": "" }, { "option": "yes", "voterName": "CARLO LEONE", "note": "" }, { "option": "yes", "voterName": "GARY WINFIELD", "note": "" }, { "option": "yes", "voterName": "TONY HWANG", "note": "" }, { "option": "yes", "voterName": "MARTIN M. LOONEY", "note": "" }, { "option": "yes", "voterName": "MAE M. FLEXER", "note": "" }, { "option": "yes", "voterName": "TED KENNEDY", "note": "" }, { "option": "yes", "voterName": "CRAIG MINER", "note": "" }, { "option": "yes", "voterName": "LEN SUZIO", "note": "" }, { "option": "yes", "voterName": "HENRI MARTIN", "note": "" }, { "option": "yes", "voterName": "GAYLE SLOSSBERG", "note": "" }, { "option": "yes", "voterName": "ERIC BERTHEL", "note": "" }, { "option": "yes", "voterName": "JOAN V. HARTLEY", "note": "" }, { "option": "yes", "voterName": "ART LINARES", "note": "" }, { "option": "yes", "voterName": "JOE MARKLEY", "note": "" }, { "option": "yes", "voterName": "LEONARD FASANO", "note": "" }, { "option": "yes", "voterName": "GEORGE LOGAN", "note": "" }, { "option": "yes", "voterName": "ANTHONY GUGLIELMO", "note": "" }, { "option": "yes", "voterName": "HEATHER SOMERS", "note": "" }, { "option": "yes", "voterName": "L. SCOTT FRANTZ", "note": "" } ] } } ] } }, "cursor": "YXJyYXljb25uZWN0aW9uOjEwNTgw" }, { "node": { "identifier": "HB 6347", "updatedAt": "2020-04-28 21:53:18.355811+00:00", "createdAt": "2017-07-08 05:36:03.297403+00:00", "legislativeSession": { "identifier": "2017" }, "id": "ocd-bill/3409f997-9181-42ef-8c0b-be1b3aea5dd7", "votes": { "edges": [ { "node": { "id": "ocd-vote/c947a478-ead9-4ec4-a624-a8aeefb42057", "startDate": "2017-05-25", "result": "pass", "votes": [ { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" } ] } }, { "node": { "id": "ocd-vote/f635e1dd-91c5-4b4f-a633-aacbf168ee5f", "startDate": "2017-06-07", "result": "pass", "votes": [ { "option": "yes", "voterName": "JOHN W. FONFARA", "note": "" }, { "option": "yes", "voterName": "CATHERINE A. OSTEN", "note": "" }, { "option": "yes", "voterName": "DOUGLAS MCCRORY", "note": "" }, { "option": "yes", "voterName": "PAUL M. FORMICA", "note": "" }, { "option": "yes", "voterName": "TIM LARSON", "note": "" }, { "option": "yes", "voterName": "KEVIN KELLY", "note": "" }, { "option": "yes", "voterName": "STEVE CASSANO", "note": "" }, { "option": "yes", "voterName": "MARILYN MOORE", "note": "" }, { "option": "yes", "voterName": "BETH BYE", "note": "" }, { "option": "yes", "voterName": "EDWIN A. GOMES", "note": "" }, { "option": "yes", "voterName": "TERRY B. GERRATANA", "note": "" }, { "option": "yes", "voterName": "MICHAEL A. MCLACHLAN", "note": "" }, { "option": "yes", "voterName": "JOHN A. KISSEL", "note": "" }, { "option": "yes", "voterName": "BOB DUFF", "note": "" }, { "option": "yes", "voterName": "KEVIN D. WITKOS", "note": "" }, { "option": "yes", "voterName": "TONI BOUCHER", "note": "" }, { "option": "yes", "voterName": "PAUL DOYLE", "note": "" }, { "option": "yes", "voterName": "CARLO LEONE", "note": "" }, { "option": "yes", "voterName": "GARY WINFIELD", "note": "" }, { "option": "yes", "voterName": "TONY HWANG", "note": "" }, { "option": "yes", "voterName": "MARTIN M. LOONEY", "note": "" }, { "option": "yes", "voterName": "MAE M. FLEXER", "note": "" }, { "option": "yes", "voterName": "TED KENNEDY", "note": "" }, { "option": "yes", "voterName": "CRAIG MINER", "note": "" }, { "option": "yes", "voterName": "LEN SUZIO", "note": "" }, { "option": "yes", "voterName": "HENRI MARTIN", "note": "" }, { "option": "yes", "voterName": "GAYLE SLOSSBERG", "note": "" }, { "option": "yes", "voterName": "ERIC BERTHEL", "note": "" }, { "option": "yes", "voterName": "JOAN V. HARTLEY", "note": "" }, { "option": "yes", "voterName": "ART LINARES", "note": "" }, { "option": "yes", "voterName": "JOE MARKLEY", "note": "" }, { "option": "yes", "voterName": "LEONARD FASANO", "note": "" }, { "option": "yes", "voterName": "GEORGE LOGAN", "note": "" }, { "option": "yes", "voterName": "ANTHONY GUGLIELMO", "note": "" }, { "option": "yes", "voterName": "HEATHER SOMERS", "note": "" }, { "option": "yes", "voterName": "L. SCOTT FRANTZ", "note": "" } ] } } ] } }, "cursor": "YXJyYXljb25uZWN0aW9uOjEwNTgx" }, { "node": { "identifier": "SB 1058", "updatedAt": "2020-04-28 21:53:18.334227+00:00", "createdAt": "2017-07-08 05:36:36.944007+00:00", "legislativeSession": { "identifier": "2017" }, "id": "ocd-bill/08fadb0e-2c49-4f25-a73b-faa37871ce3f", "votes": { "edges": [ { "node": { "id": "ocd-vote/33c39f4d-21aa-445d-a2f7-639960c59e91", "startDate": "2017-06-07", "result": "pass", "votes": [ { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "no", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "N", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" }, { "option": "other", "voterName": "Y", "note": "" } ] } }, { "node": { "id": "ocd-vote/87986367-d091-4f88-89c6-2085b4632fa1", "startDate": "2017-06-07", "result": "pass", "votes": [ { "option": "yes", "voterName": "JOHN W. FONFARA", "note": "" }, { "option": "yes", "voterName": "CATHERINE A. OSTEN", "note": "" }, { "option": "yes", "voterName": "DOUGLAS MCCRORY", "note": "" }, { "option": "no", "voterName": "PAUL M. FORMICA", "note": "" }, { "option": "yes", "voterName": "TIM LARSON", "note": "" }, { "option": "yes", "voterName": "KEVIN KELLY", "note": "" }, { "option": "yes", "voterName": "STEVE CASSANO", "note": "" }, { "option": "yes", "voterName": "MARILYN MOORE", "note": "" }, { "option": "yes", "voterName": "BETH BYE", "note": "" }, { "option": "yes", "voterName": "EDWIN A. GOMES", "note": "" }, { "option": "yes", "voterName": "TERRY B. GERRATANA", "note": "" }, { "option": "yes", "voterName": "MICHAEL A. MCLACHLAN", "note": "" }, { "option": "no", "voterName": "JOHN A. KISSEL", "note": "" }, { "option": "yes", "voterName": "BOB DUFF", "note": "" }, { "option": "no", "voterName": "KEVIN D. WITKOS", "note": "" }, { "option": "yes", "voterName": "TONI BOUCHER", "note": "" }, { "option": "yes", "voterName": "PAUL DOYLE", "note": "" }, { "option": "yes", "voterName": "CARLO LEONE", "note": "" }, { "option": "yes", "voterName": "GARY WINFIELD", "note": "" }, { "option": "yes", "voterName": "TONY HWANG", "note": "" }, { "option": "yes", "voterName": "MARTIN M. LOONEY", "note": "" }, { "option": "yes", "voterName": "MAE M. FLEXER", "note": "" }, { "option": "yes", "voterName": "TED KENNEDY", "note": "" }, { "option": "yes", "voterName": "CRAIG MINER", "note": "" }, { "option": "yes", "voterName": "LEN SUZIO", "note": "" }, { "option": "yes", "voterName": "HENRI MARTIN", "note": "" }, { "option": "yes", "voterName": "GAYLE SLOSSBERG", "note": "" }, { "option": "yes", "voterName": "ERIC BERTHEL", "note": "" }, { "option": "yes", "voterName": "JOAN V. HARTLEY", "note": "" }, { "option": "yes", "voterName": "ART LINARES", "note": "" }, { "option": "no", "voterName": "JOE MARKLEY", "note": "" }, { "option": "yes", "voterName": "LEONARD FASANO", "note": "" }, { "option": "yes", "voterName": "GEORGE LOGAN", "note": "" }, { "option": "yes", "voterName": "ANTHONY GUGLIELMO", "note": "" }, { "option": "yes", "voterName": "HEATHER SOMERS", "note": "" }, { "option": "yes", "voterName": "L. SCOTT FRANTZ", "note": "" } ] } } ] } }, "cursor": "YXJyYXljb25uZWN0aW9uOjEwNTgy" }, { "node": { "identifier": "HB 7601", "updatedAt": "2020-04-28 21:53:18.311063+00:00", "createdAt": "2018-01-04 05:35:58.528624+00:00", "legislativeSession": { "identifier": "2017" }, "id": "ocd-bill/bab73944-3455-4140-b22f-b1df189a5489", "votes": { "edges": [] } }, "cursor": "YXJyYXljb25uZWN0aW9uOjEwNTgz" } ], "pageInfo": { "hasNextPage": false, "hasPreviousPage": true, "endCursor": "YXJyYXljb25uZWN0aW9uOjEwNTgz", "startCursor": "YXJyYXljb25uZWN0aW9uOjEwNTc5" }, "totalCount": 10584 } } } ```
jamesturk commented 3 years ago

thanks for this report, looking into it!

jamesturk commented 3 years ago

so far no votes in the current session, might circle back to this after a few days so we can get a fix working there then consider a re-scrape of prior sessions

mzagaja commented 3 years ago

Hello, I just wanted to check-in and also extend this bug report since there have been some vote sin the current session. I spot checked this bill.

## Individual bill Information
curl -X "POST" "https://openstates.org/graphql" \
     -H 'X-API-KEY: ' \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{
    "query": "query MyQuery{bill(jurisdiction:\\"Connecticut\\",id:\\"ocd-bill/37b59fbd-bca6-44e4-b0da-b1e45f205ce4\\"){votes{edges{node{id}}}}}",
    "variables": {}
}'

returns

{
  "data": {
    "bill": {
      "votes": {
        "edges": []
      }
    }
  }
}

I am wondering if it has to do with how it finds links on the page. It seems like the legislature isn't offering HTML links anymore. Looking at: https://github.com/openstates/openstates-scrapers/blob/main/scrapers/ct/bills.py#L143-L150

It appears (though I'm not an expert in Python nor the library being using) that it is looking at all the links with VOTE in them, and then within this list if it finds one that ends in .htm it is grabbing the one before it, (line 146) and then runs the scraper against the PDF. Since no .htm link for VOTE is present in the page, it is likely failing.

jamesturk commented 3 years ago

Thanks for this, it does look like we'll need to update CT's vote scraping

mzagaja commented 3 years ago

@jamesturk Is this in the development timeline? If it is I don't want to take a run at it since I have a bit of a learning curve, but if it's a long ways off I might set aside some time to give it a try.

jamesturk commented 3 years ago

Nobody has claimed it yet, it’d be great if you had the resources to at least investigate a fix, glad to work with you on getting it across the line. On Apr 3, 2021, 12:27 PM -0400, Matthew Zagaja @.***>, wrote:

@jamesturk Is this in the development timeline? If it is I don't want to take a run at it since I have a bit of a learning curve, but if it's a long ways off I might set aside some time to give it a try. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

mzagaja commented 3 years ago

I developed a potential fix which is exciting, however when trying to test it using the method in the documentation, I am not seeing JSON files saved:

mzagaja@MacBook-Pro ~/D/o/_/ct (main) [1]> docker-compose run --rm scrape ct bills --fastmode --scrape

19:26:14 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5470&which_year=2021
/root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
19:26:15 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5471&which_year=2021
/root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
19:26:15 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5472&which_year=2021
/root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
19:26:15 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5473&which_year=2021
/root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
19:26:15 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5474&which_year=2021
/root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
19:26:16 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5475&which_year=2021
/root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,

versus suggested output:

08:46:38 INFO scrapelib: GET - http://www.ncga.state.nc.us/gascripts/BillLookUp/BillLookUp.pl?Session=2017&BillID=S1
08:46:39 INFO openstates: save bill SR 1 in 2017 as bill_03c7edb4-72d2-11e7-8df8-0242ac130003.json

There is no difference if I revert to HEAD of main. Do I need to let this run through? I see plenty of files of downloaded bills in my _cache directory.

jamesturk commented 3 years ago

Hmm not sure I’ve seen this before.. Is nothing created in the _data directory?   Are there any error messages either?

James On Apr 11, 2021, 3:34 PM -0400, Matthew Zagaja @.***>, wrote:

I developed a potential fix which is exciting, however when trying to test it using the method in the documentation, I am not seeing JSON files saved: mzagaja@MacBook-Pro ~/D/o/_/ct (main) [1]> docker-compose run --rm scrape ct bills --fastmode --scrape

19:26:14 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5470&which_year=2021 /root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 19:26:15 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5471&which_year=2021 /root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 19:26:15 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5472&which_year=2021 /root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 19:26:15 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5473&which_year=2021 /root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 19:26:15 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5474&which_year=2021 /root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 19:26:16 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5475&which_year=2021 /root/.cache/pypoetry/virtualenvs/openstates-scrapers-vRcYrsYN-py3.7/lib/python3.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.cga.ct.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, versus suggested output: 08:46:38 INFO scrapelib: GET - http://www.ncga.state.nc.us/gascripts/BillLookUp/BillLookUp.pl?Session=2017&BillID=S1 08:46:39 INFO openstates: save bill SR 1 in 2017 as bill_03c7edb4-72d2-11e7-8df8-0242ac130003.json There is no difference if I revert to HEAD of main. Do I need to let this run through? I see plenty of files of downloaded bills in my _cache directory. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

mzagaja commented 3 years ago

No error messages nor anything in the _data directory. Would you like me to make a draft PR to see if you can run my modification on your setup to validate?

jamesturk commented 3 years ago

That’d be great… I’m bewildered as to why nothing is emitted.

James On Apr 13, 2021, 5:21 PM -0400, Matthew Zagaja @.***>, wrote:

No error messages nor anything in the _data directory. Would you like me to make a draft PR to see if you can run my modification on your setup to validate? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

mzagaja commented 3 years ago

I submitted a draft pull request. I also attempted to debug it a little and can share that if I add some basic logging statements, the actual "scrape" is getting run. With no errors thrown I can only conclude there is some sort step that involves writing the JSON that is being missed/skipped.

Running another state like Massachusetts writes JSON from the bill fine.

19:47:22 INFO scrapelib: GET - https://www.cga.ct.gov/asp/cgabillstatus/cgabillstatus.asp?selBillType=Bill&bill_num=HB5375&which_year=2021
Finished scraping webpage
Scraping the bill page from web
jessemortenson commented 8 months ago

To collect some context on this old issue:

braykuka commented 7 months ago

@jessemortenson, in my code review, it tries to connect to this FTP server ftp://ftp.cga.ct.gov. Is it possible to connect to this FTP server without any username & password? I couldn't connect to this server on my end.