search

SuffolkLITLab / docassemble-InterviewStats

A docassemble extension.
MIT License
1 stars 0 forks source link

Problem with large number of zip code results #4

Closed nonprofittechy closed 3 years ago

nonprofittechy commented 3 years ago

This works great on the test server, but fails on the production server with the CDC eviction moratorium interview.

Probably because on the production server, there are hundreds of zip codes and the query on line 73 is too big. It might be necessary to group and space out the queries, or create a local cache of the database.

Here is the error:


Error

Retry

    ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Log

Traceback (most recent call last):
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/urllib3/util/retry.py", line 531, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/webapp/server.py", line 7206, in index
    interview.assemble(user_dict, interview_status, old_user_dict, force_question=special_question)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/base/parse.py", line 7320, in assemble
    raise the_error
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/base/parse.py", line 7111, in assemble
    question_result = self.askfor(missingVariable, user_dict, old_user_dict, interview_status, seeking=interview_status.seeking, follow_mc=follow_mc, seeking_question=seeking_question)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/base/parse.py", line 7870, in askfor
    question_result = self.askfor(newMissingVariable, user_dict, old_user_dict, interview_status, variable_stack=variable_stack, questions_tried=questions_tried, seeking=seeking, follow_mc=follow_mc, recursion_depth=recursion_depth, seeking_question=seeking_question)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/base/parse.py", line 7870, in askfor
    question_result = self.askfor(newMissingVariable, user_dict, old_user_dict, interview_status, variable_stack=variable_stack, questions_tried=questions_tried, seeking=seeking, follow_mc=follow_mc, recursion_depth=recursion_depth, seeking_question=seeking_question)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/base/parse.py", line 7870, in askfor
    question_result = self.askfor(newMissingVariable, user_dict, old_user_dict, interview_status, variable_stack=variable_stack, questions_tried=questions_tried, seeking=seeking, follow_mc=follow_mc, recursion_depth=recursion_depth, seeking_question=seeking_question)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/base/parse.py", line 7759, in askfor
    exec_with_trap(question, user_dict)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/base/parse.py", line 8324, in exec_with_trap
    exec(the_question.compute, the_dict)
  File "<code block>", line 1, in <module>
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/InterviewStats/snapshot_geography.py", line 186, in make_usage_map
    geo_loc_counts = grab_geography(agg_df, geo_col, time_col)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/docassemble/InterviewStats/snapshot_geography.py", line 73, in grab_geography
    new_zip_shapes = api_conn.mapservice.query(layer=8, where=full_where)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/cenpy/tiger.py", line 272, in query
    return self.layers[layer_result].query(**kwargs)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/cenpy/tiger.py", line 181, in query
    resp = r.get(self._last_query + "&f=json")
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/share/docassemble/local3.8/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Feel free to test on the production server to hunt this bug down.

BryceStevenWilley commented 3 years ago

The issue above was mostly because the number of zip codes that we were requesting through the URL was much too large. In general, with this many submissions all over the US, ad-hoc requests for shapes is probably going to be incredibly slow.

This should be fixed with https://github.com/SuffolkLITLab/docassemble-InterviewStats/commit/374dde5169e923413ef0f19b5d79fab76538fd91, and some of the follow commits.