search

openelections / openelections-core

Core repo for election results data acquisition, transformation and output.
MIT License
176 stars 96 forks source link

Can't publish raw results files > 100MB to Github #244

Closed dwillis closed 9 years ago

dwillis commented 9 years ago

This applies so far to Pennsylvania data in presidential years. Maybe we split up the files somehow?

ghing commented 9 years ago

Can you provide a stack trace/error message for this?

dwillis commented 9 years ago 
Publishing /Users/200025/code/core/openelex/us/bakery/20080422__pa__primary__precinct__raw.csv
Traceback (most recent call last):
  File "/Users/200025/.virtualenvs/openelex/bin/openelex", line 9, in <module>
    load_entry_point('OpenElections-Core==0.1.0', 'console_scripts', 'openelex')()
  File "/Users/200025/.virtualenvs/openelex/lib/python2.7/site-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "/Users/200025/.virtualenvs/openelex/lib/python2.7/site-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "/Users/200025/.virtualenvs/openelex/lib/python2.7/site-packages/click/core.py", line 936, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/200025/.virtualenvs/openelex/lib/python2.7/site-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/200025/.virtualenvs/openelex/lib/python2.7/site-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/Users/200025/code/core/openelex/tasks/publish.py", line 40, in publish
    publisher.publish(state, datefilter=datefilter, raw=raw)
  File "/Users/200025/code/core/openelex/base/publish.py", line 170, in publish
    self.publish_file(repo, filename)
  File "/Users/200025/code/core/openelex/base/publish.py", line 217, in publish_file
    result = repo.create_file(path, msg, content)
  File "/Users/200025/.virtualenvs/openelex/lib/python2.7/site-packages/github3/decorators.py", line 38, in auth_wrapper
    return func(self, *args, **kwargs)
  File "/Users/200025/.virtualenvs/openelex/lib/python2.7/site-packages/github3/repos/repo.py", line 566, in create_file
    json = self._json(self._put(url, data=dumps(data)), 201)
  File "/Users/200025/.virtualenvs/openelex/lib/python2.7/site-packages/github3/models.py", line 100, in _json
    if self._boolean(response, status_code, 404) and response.content:
  File "/Users/200025/.virtualenvs/openelex/lib/python2.7/site-packages/github3/models.py", line 121, in _boolean
    raise GitHubError(response)
github3.models.GitHubError: 405 <html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>
dwillis commented 9 years ago

I should add that I've added some custom fields to the PA output, and those could be trimmed back to see if that helps.

ghing commented 9 years ago

@dwillis Thanks. Also, according to the GitHub docs on file limits, it seems like a 50MB file should be acceptable.

ghing commented 9 years ago

Though perhaps not through the API.

dwillis commented 9 years ago

Yeah, the 2004 general file is 85 MB and that fails. So it's somewhere north of 50.

ghing commented 9 years ago

Sorry for my poorly articulated comment. The docs say up to 100MB is ok. So I would test out adding the file to repo and pushing manually to see if that works.

dwillis commented 9 years ago

Ok, yeah, that seems to work (you get warnings for files over 50MB, but that's fine).

ghing commented 9 years ago

Ok. So I wonder if there's a limit to file size when using the API.

dwillis commented 9 years ago

Seems like it.