search

nprapps / elections18-general

2018 midterm election back-end: Associated Press data ETL, database, admin panel, and JSON output; iteration upon 2016 GE work
MIT License
3 stars 1 forks source link

Add Fabric task to regularly update the get-caught-up XLSX file #20

Closed mileswwatkins closed 6 years ago

mileswwatkins commented 6 years ago

This will power the pipeline that @freejoe76 is working on now; cc https://github.com/nprapps/elections18-graphics/issues/48

Make sure this task is included (as a background process?) in the main Fabric daemon task.

Also, make sure that the production server that runs this daemon will have OAuth/Google credentials that'll be able to run fab text.update to properly update this file.

mileswwatkins commented 6 years ago

There's some server complaining that I've seen as I try to get this to work, including these tracebacks:

$ docker-compose up daemon

...

daemon_1        | INFO:fabfile.daemons:2018-09-11 16:08:53,766: updating get-caught-up text (runs in parallel)
daemon_1        | INFO:fabfile.daemons:2018-09-11 16:08:54,270: loading results
daemon_1        |
daemon_1        | Warning: local() encountered an error (return code 64) while executing 'elex results --results-level ru --officeids H,S,G,I 2016-11-08 > .data/results---results-level_ru_--officeids_HSGI.csv'
daemon_1        |
daemon_1        | INFO:fabfile.data:2018-09-11 16:09:01,403: results loaded
daemon_1        | INFO:fabfile.daemons:2018-09-11 16:09:01,403: results loaded: 7.637657165527344 seconds
daemon_1        | INFO:fabfile.daemons:2018-09-11 16:09:18,144: results rendered and published: 24.378236532211304 seconds
daemon_1        | INFO:fabfile.daemons:2018-09-11 16:09:18,245: updating get-caught-up text (runs in parallel)
daemon_1        | Traceback (most recent call last):
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/main.py", line 763, in main
daemon_1        |     *args, **kwargs
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/tasks.py", line 427, in execute
daemon_1        |     results['<local-only>'] = task.run(*args, **new_kwargs)
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/tasks.py", line 174, in run
daemon_1        |     return self.wrapped(*args, **kwargs)
daemon_1        |   File "/opt/elections18/fabfile/daemons.py", line 40, in main
daemon_1        |     execute('text.update_in_parallel')
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/tasks.py", line 427, in execute
daemon_1        |     results['<local-only>'] = task.run(*args, **new_kwargs)
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/tasks.py", line 174, in run
daemon_1        |     return self.wrapped(*args, **kwargs)
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/decorators.py", line 184, in inner
daemon_1        |     return func(*args, **kwargs)
daemon_1        |   File "/opt/elections18/fabfile/text.py", line 27, in update_in_parallel
daemon_1        |     update_calendar()
daemon_1        |   File "/opt/elections18/fabfile/text.py", line 46, in update_calendar
daemon_1        |     app_config.CALENDAR_PATH
daemon_1        |   File "/opt/elections18/oauth.py", line 119, in get_document
daemon_1        |     raise KeyError("Error! Google returned a %s error" % response.status)
daemon_1        | KeyError: 'Error! Google returned a 500 error'
elections18-general_daemon_1 exited with code 1

and

$ docker-compose up daemon
elections18-general_database_1 is up-to-date
Starting elections18-general_daemon_1 ... done
Attaching to elections18-general_daemon_1
daemon_1        | INFO:fabfile.daemons:2018-09-11 15:56:20,504: updating get-caught-up text (runs in parallel)
daemon_1        | Traceback (most recent call last):
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/main.py", line 763, in main
daemon_1        |     *args, **kwargs
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/tasks.py", line 427, in execute
daemon_1        |     results['<local-only>'] = task.run(*args, **new_kwargs)
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/tasks.py", line 174, in run
daemon_1        |     return self.wrapped(*args, **kwargs)
daemon_1        |   File "/opt/elections18/fabfile/daemons.py", line 40, in main
daemon_1        |     execute('text.update_in_parallel')
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/tasks.py", line 427, in execute
daemon_1        |     results['<local-only>'] = task.run(*args, **new_kwargs)
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/tasks.py", line 174, in run
daemon_1        |     return self.wrapped(*args, **kwargs)
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/fabric/decorators.py", line 184, in inner
daemon_1        |     return func(*args, **kwargs)
daemon_1        |   File "/opt/elections18/fabfile/text.py", line 27, in update_in_parallel
daemon_1        |     update_calendar()
daemon_1        |   File "/opt/elections18/fabfile/text.py", line 46, in update_calendar
daemon_1        |     app_config.CALENDAR_PATH
daemon_1        |   File "/opt/elections18/oauth.py", line 111, in get_document
daemon_1        |     credentials = get_credentials()
daemon_1        |   File "/opt/elections18/oauth.py", line 92, in get_credentials
daemon_1        |     credentials.refresh()
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/authomatic/core.py", line 772, in refresh
daemon_1        |     return self.provider_class(self, None, self.provider_name).refresh_credentials(self)
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/authomatic/providers/oauth2.py", line 251, in refresh_credentials
daemon_1        |     method='POST')
daemon_1        |   File "/usr/local/lib/python3.6/site-packages/authomatic/providers/oauth2.py", line 162, in create_request_elements
daemon_1        |     'are required to create OAuth 2.0 refresh token request elements!')
daemon_1        | authomatic.exceptions.OAuth2Error: Credentials with valid refresh_token, consumer_key, consumer_secret are required to create OAuth 2.0 refresh token request elements!
elections18-general_daemon_1 exited with code 1

But I can look into these more later, handling them as separate bugs. Much more important come AP tests in October.