search

okfn / measure

Measure is scripts and conventions to build KPI dashboards for projects.
MIT License
17 stars 5 forks source link

Add support for HTTP rate limiting #52

Closed vitorbaptista closed 6 years ago

vitorbaptista commented 6 years ago

Discourse is limiting our requests:

measure.add_discourse_category_resource: Traceback (most recent call last):
measure.add_discourse_category_resource:   File "/usr/local/lib/python3.6/site-packages/datapackage_pipelines_measure/processors/add_discourse_category_resource.py", line 136, in 
measure.add_discourse_category_resource:     category, child_treatment))
measure.add_discourse_category_resource:   File "/usr/local/lib/python3.6/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 55, in spew
measure.add_discourse_category_resource:     for res in resources_iterator:
measure.add_discourse_category_resource:   File "/usr/local/lib/python3.6/site-packages/datapackage_pipelines_measure/processors/add_discourse_category_resource.py", line 132, in process_resources
measure.add_discourse_category_resource:     yield discourse_collector(domain, category, child_treatment, latest_row)
measure.add_discourse_category_resource:   File "/usr/local/lib/python3.6/site-packages/datapackage_pipelines_measure/processors/add_discourse_category_resource.py", line 75, in discourse_collector
measure.add_discourse_category_resource:     child['slug'])
measure.add_discourse_category_resource:   File "/usr/local/lib/python3.6/site-packages/datapackage_pipelines_measure/processors/add_discourse_category_resource.py", line 36, in _make_stats_dict_for_category_id
measure.add_discourse_category_resource:     category_id=category_id)
measure.add_discourse_category_resource:   File "/usr/local/lib/python3.6/site-packages/datapackage_pipelines_measure/processors/discourse_utils.py", line 56, in request_report_from_discourse
measure.add_discourse_category_resource:     category_id=category_id)['report']['data']
measure.add_discourse_category_resource:   File "/usr/local/lib/python3.6/site-packages/datapackage_pipelines_measure/processors/discourse_utils.py", line 32, in request_data_from_discourse
measure.add_discourse_category_resource:     response.content))
measure.add_discourse_category_resource: ValueError: Error raised for domain:discuss.okfn.org, Status code:429. Error message: b'{"errors":["You have performed this action too many times. Please wait 26 seconds before trying again."],"error_type":"rate_limit","extras":{"wait_seconds":26}}'

We need to handle this error, sleeping for a bit, instead of failing.