search

openelections / openelections-core

Core repo for election results data acquisition, transformation and output.
MIT License
179 stars 98 forks source link

Misleading exception thrown when API server does not return HTTP 200. #277

Closed crass closed 4 years ago

crass commented 5 years ago

I've followed the instructions to get an OpenElections environment setup. However, I ran into this exception:

(openelections-core.git) user@host:/tmp/openelections-core.git$ openelex load.run -s tx
Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/openelections-core.git-YL8ByLcB/bin/openelex", line 11, in <module>
    load_entry_point('OpenElections-Core', 'console_scripts', 'openelex')()
  File "/home/user/.local/share/virtualenvs/openelections-core.git-YL8ByLcB/lib/python3.6/site-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/.local/share/virtualenvs/openelections-core.git-YL8ByLcB/lib/python3.6/site-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "/home/user/.local/share/virtualenvs/openelections-core.git-YL8ByLcB/lib/python3.6/site-packages/click/core.py", line 936, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user/.local/share/virtualenvs/openelections-core.git-YL8ByLcB/lib/python3.6/site-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/.local/share/virtualenvs/openelections-core.git-YL8ByLcB/lib/python3.6/site-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/tmp/openelections-core.git/openelex/tasks/load.py", line 31, in run
    mappings = datasrc.mappings(datefilter)
  File "/tmp/openelections-core.git/openelex/us/tx/datasource.py", line 27, in mappings
    for yr, elecs in list(self.elections(year).items()):
  File "/tmp/openelections-core.git/openelex/base/datasource.py", line 67, in elections
    yr = int(elec['start_date'][:4])
TypeError: string indices must be integers

I've determined after some debugging that this is happening because elec_api.find('tx') is returning Request raised error: 500 (state: tx, datefilter: ). This can be seen by running curl -v 'http://openelections.net/api/v1/election/?state__postal__iexact=tx'.

In order to make this issue less hidden, I suggest that find in openelex/api/elections.py should raise an exception when the response.status_code is not 200.

dwillis commented 5 years ago

@crass Thanks - the metadata API currently is down, so the core repository won't work at the moment. We're working on a fix for this.

crass commented 5 years ago

Is the data returned by the metadata API the same as the data found in the metadata repo? It seems like there could be code to optionally get the data locally. Iirc, I saw an issue somewhere that you guys wanted to move away from an API server to hosting the metadata statically (presumably the API server logic would be moved client side, in which case much of the same code could be used for local metadata).

And while the API being down is the root cause, I consider this bug to be more about the misleading exception.

dwillis commented 5 years ago

@crass Yes, the data is the same, and having the option (or just replacing the API with the local data) is our goal.

But fair point about the exception.

jslap commented 4 years ago

Any news on the metadata API server? Hard to do anything without it.

dwillis commented 4 years ago

Hey folks, we've pushed a change that uses the JSON from the newly static metadata API. Kick the tires and let us know how it works for you.

cc @jslap