search

focusconsulting / housing-insights

Bringing open data to affordable housing decision makers in Washington DC. A D3/Javascript based website to visualize data related to affordable housing in Washington DC. Data processing with Python.
http://housinginsights.org
MIT License
58 stars 110 forks source link

add "503" error handling #479

Open emkap01 opened 7 years ago

emkap01 commented 7 years ago

We will need to handle the following error, which can sometimes occur during API calls to the MAR:

Traceback (most recent call last):
  File "load_data.py", line 81, in <module>
    main(arguments)
  File "/repo/python/housinginsights/ingestion/LoadData.py", line 1192, in main
    loader.rebuild()
  File "/repo/python/housinginsights/ingestion/LoadData.py", line 364, in rebuild
    self._process_data_file(manifest_row=manifest_row)
  File "/repo/python/housinginsights/ingestion/LoadData.py", line 960, in _process_data_file
    temp_filepath=temp_filepath)
  File "/repo/python/housinginsights/ingestion/LoadData.py", line 1043, in _load_single_file
    clean_data_row = cleaner.clean(data_row, idx)
  File "/repo/python/housinginsights/ingestion/Cleaners.py", line 471, in clean
    row, column_name='MARADDRESSREPOSITORYID')
  File "/repo/python/housinginsights/ingestion/Cleaners.py", line 401, in add_census_tract_from_mar
    result = mar_api.reverse_address_id(aid=row[column_name])
  File "/repo/python/housinginsights/sources/mar.py", line 219, in reverse_address_id
    raise Exception(err.format(result.status_code))
Exception: An error occurred during request: status 503
jkwening commented 7 years ago

linking #512 to this issue because I think they're related. With all the calls we're making to the mar api, at times, especially during the day or on a bad network we seem to be either:

I'm going to strategically put timer.sleep(0.01) in code, log state at time exception is through, and retry maybe 2-3 more times, waiting a little longer each iteration to see if this resolves this issue.