search

mozilla / leanplum-data-export

A docker image that runs a script to export Leanplum data into BigQuery
4 stars 4 forks source link

Frequent intermittent errors #22

Open BenWu opened 3 years ago

BenWu commented 3 years ago

There are still errors happening frequently, recently it's almost everyday. This is the log for one error but there may be different ones. Airflow logs get purged too quickly so they're hard to catch.

[2020-10-24 04:21:07,743] {logging_mixin.py:112} INFO - [2020-10-24 04:21:07,743] {pod_launcher.py:125} INFO - b'INFO:root:Creating external table tmp.firefox_android_release_external_leanplum_eventparameters_v2_20201023\n'
[2020-10-24 04:21:08,012] {logging_mixin.py:112} INFO - [2020-10-24 04:21:08,012] {pod_launcher.py:125} INFO - b'INFO:root:Creating external table tmp.firefox_android_release_external_leanplum_events_v2_20201023\n'
[2020-10-24 04:22:08,176] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,176] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2020-10-24 04:22:08,176] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,176] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 426, in _make_request\n'
[2020-10-24 04:22:08,200] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,200] {pod_launcher.py:125} INFO - b'    six.raise_from(e, None)\n'
[2020-10-24 04:22:08,201] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,201] {pod_launcher.py:125} INFO - b'  File "<string>", line 3, in raise_from\n'
[2020-10-24 04:22:08,202] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,202] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 421, in _make_request\n'
[2020-10-24 04:22:08,202] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,202] {pod_launcher.py:125} INFO - b'    httplib_response = conn.getresponse()\n'
[2020-10-24 04:22:08,202] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,202] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/http/client.py", line 1347, in getresponse\n'
[2020-10-24 04:22:08,238] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,238] {pod_launcher.py:125} INFO - b'    response.begin()\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/http/client.py", line 307, in begin\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b'    version, status, reason = self._read_status()\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/http/client.py", line 268, in _read_status\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b'    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")\n'
[2020-10-24 04:22:08,240] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,240] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/socket.py", line 704, in readinto\n'
[2020-10-24 04:22:08,274] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,274] {pod_launcher.py:125} INFO - b'    return self._sock.recv_into(b)\n'
[2020-10-24 04:22:08,274] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,274] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/ssl.py", line 1241, in recv_into\n'
[2020-10-24 04:22:08,316] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,315] {pod_launcher.py:125} INFO - b'    return self.read(nbytes, buffer)\n'
[2020-10-24 04:22:08,317] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,317] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/ssl.py", line 1099, in read\n'
[2020-10-24 04:22:08,321] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,321] {pod_launcher.py:125} INFO - b'    return self._sslobj.read(len, buffer)\n'
[2020-10-24 04:22:08,321] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,321] {pod_launcher.py:125} INFO - b'socket.timeout: The read operation timed out\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,321] {pod_launcher.py:125} INFO - b'\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,322] {pod_launcher.py:125} INFO - b'During handling of the above exception, another exception occurred:\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,322] {pod_launcher.py:125} INFO - b'\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,322] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2020-10-24 04:22:08,322] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,322] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send\n'
[2020-10-24 04:22:08,337] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,336] {pod_launcher.py:125} INFO - b'    resp = conn.urlopen(\n'
[2020-10-24 04:22:08,337] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,337] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 726, in urlopen\n'
[2020-10-24 04:22:08,341] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,341] {pod_launcher.py:125} INFO - b'    retries = retries.increment(\n'
[2020-10-24 04:22:08,342] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,342] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 403, in increment\n'
[2020-10-24 04:22:08,365] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,365] {pod_launcher.py:125} INFO - b'    raise six.reraise(type(error), error, _stacktrace)\n'
[2020-10-24 04:22:08,365] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,365] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 735, in reraise\n'
[2020-10-24 04:22:08,374] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,374] {pod_launcher.py:125} INFO - b'    raise value\n'
[2020-10-24 04:22:08,374] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,374] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 670, in urlopen\n'
[2020-10-24 04:22:08,375] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,374] {pod_launcher.py:125} INFO - b'    httplib_response = self._make_request(\n'
[2020-10-24 04:22:08,375] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,375] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 428, in _make_request\n'
[2020-10-24 04:22:08,375] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,375] {pod_launcher.py:125} INFO - b'    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)\n'
[2020-10-24 04:22:08,375] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,375] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 335, in _raise_timeout\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'    raise ReadTimeoutError(\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b"urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=60)\n"
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'During handling of the above exception, another exception occurred:\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'\n'
[2020-10-24 04:22:08,376] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2020-10-24 04:22:08,377] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,376] {pod_launcher.py:125} INFO - b'  File "/usr/local/bin/leanplum-data-export", line 33, in <module>\n'
[2020-10-24 04:22:08,420] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,420] {pod_launcher.py:125} INFO - b"    sys.exit(load_entry_point('leanplum-data-export', 'console_scripts', 'leanplum-data-export')())\n"
[2020-10-24 04:22:08,420] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,420] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__\n'
[2020-10-24 04:22:08,432] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,432] {pod_launcher.py:125} INFO - b'    return self.main(*args, **kwargs)\n'
[2020-10-24 04:22:08,432] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,432] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main\n'
[2020-10-24 04:22:08,437] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,437] {pod_launcher.py:125} INFO - b'    rv = self.invoke(ctx)\n'
[2020-10-24 04:22:08,437] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,437] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b'    return _process_result(sub_ctx.command.invoke(sub_ctx))\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b'    return ctx.invoke(self.callback, **ctx.params)\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke\n'
[2020-10-24 04:22:08,438] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b'    return callback(*args, **kwargs)\n'
[2020-10-24 04:22:08,439] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,438] {pod_launcher.py:125} INFO - b'  File "/app/leanplum_data_export/leanplum_data_export/__main__.py", line 31, in export_leanplum\n'
[2020-10-24 04:22:08,463] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,463] {pod_launcher.py:125} INFO - b'    exporter.export(date, s3_bucket, bucket, prefix, bq_dataset, table_prefix, version, clean)\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b'  File "/app/leanplum_data_export/leanplum_data_export/export.py", line 62, in export\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b'    self.create_external_tables(gcs_bucket, prefix, date, self.DATA_TYPES,\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b'  File "/app/leanplum_data_export/leanplum_data_export/export.py", line 208, in create_external_tables\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b'    self.bq_client.delete_table(table, not_found_ok=True)\n'
[2020-10-24 04:22:08,464] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,464] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 1055, in delete_table\n'
[2020-10-24 04:22:08,510] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,510] {pod_launcher.py:125} INFO - b'    self._call_api(retry, method="DELETE", path=table.path)\n'
[2020-10-24 04:22:08,512] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,512] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 476, in _call_api\n'
[2020-10-24 04:22:08,512] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,512] {pod_launcher.py:125} INFO - b'    return call()\n'
[2020-10-24 04:22:08,512] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,512] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func\n'
[2020-10-24 04:22:08,538] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,538] {pod_launcher.py:125} INFO - b'    return retry_target(\n'
[2020-10-24 04:22:08,539] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,539] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 184, in retry_target\n'
[2020-10-24 04:22:08,540] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,540] {pod_launcher.py:125} INFO - b'    return target()\n'
[2020-10-24 04:22:08,540] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,540] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 424, in api_request\n'
[2020-10-24 04:22:08,554] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,554] {pod_launcher.py:125} INFO - b'    response = self._make_request(\n'
[2020-10-24 04:22:08,554] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,554] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 288, in _make_request\n'
[2020-10-24 04:22:08,556] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,555] {pod_launcher.py:125} INFO - b'    return self._do_request(\n'
[2020-10-24 04:22:08,556] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,556] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http.py", line 326, in _do_request\n'
[2020-10-24 04:22:08,556] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,556] {pod_launcher.py:125} INFO - b'    return self.http.request(\n'
[2020-10-24 04:22:08,556] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,556] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/google/auth/transport/requests.py", line 464, in request\n'
[2020-10-24 04:22:08,559] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b'    response = super(AuthorizedSession, self).request(\n'
[2020-10-24 04:22:08,559] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 533, in request\n'
[2020-10-24 04:22:08,559] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b'    resp = self.send(prep, **send_kwargs)\n'
[2020-10-24 04:22:08,559] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 646, in send\n'
[2020-10-24 04:22:08,560] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,559] {pod_launcher.py:125} INFO - b'    r = adapter.send(request, **kwargs)\n'
[2020-10-24 04:22:08,560] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,560] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 529, in send\n'
[2020-10-24 04:22:08,560] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,560] {pod_launcher.py:125} INFO - b'    raise ReadTimeout(e, request=request)\n'
[2020-10-24 04:22:08,561] {logging_mixin.py:112} INFO - [2020-10-24 04:22:08,561] {pod_launcher.py:125} INFO - b"requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=60)\n"
[2020-10-24 04:22:09,647] {logging_mixin.py:112} INFO - [2020-10-24 04:22:09,646] {pod_launcher.py:142} INFO - Event: firefox-android-release-export-af18059c had an event of type Failed
[2020-10-24 04:22:09,647] {logging_mixin.py:112} INFO - [2020-10-24 04:22:09,647] {pod_launcher.py:234} INFO - Event with job id firefox-android-release-export-af18059c Failed
[2020-10-24 04:22:09,687] {logging_mixin.py:112} INFO - [2020-10-24 04:22:09,687] {pod_launcher.py:142} INFO - Event: firefox-android-release-export-af18059c had an event of type Failed
[2020-10-24 04:22:09,687] {logging_mixin.py:112} INFO - [2020-10-24 04:22:09,687] {pod_launcher.py:234} INFO - Event with job id firefox-android-release-export-af18059c Failed
BenWu commented 3 years ago

It would be good to just retry the requests after a timeout instead of failing immediately.

Another one

[2020-11-20 20:08:58,939] {pod_launcher.py:156} INFO - b'Traceback (most recent call last):\n'
[2020-11-20 20:08:58,941] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 436, in _error_catcher\n'
[2020-11-20 20:08:58,954] {pod_launcher.py:156} INFO - b'    yield\n'
[2020-11-20 20:08:58,954] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 518, in read\n'
[2020-11-20 20:08:58,959] {pod_launcher.py:156} INFO - b'    data = self._fp.read(amt) if not fp_closed else b""\n'
[2020-11-20 20:08:58,959] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/http/client.py", line 458, in read\n'
[2020-11-20 20:08:59,006] {pod_launcher.py:156} INFO - b'    n = self.readinto(b)\n'
[2020-11-20 20:08:59,007] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/http/client.py", line 502, in readinto\n'
[2020-11-20 20:08:59,008] {pod_launcher.py:156} INFO - b'    n = self.fp.readinto(b)\n'
[2020-11-20 20:08:59,010] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/socket.py", line 704, in readinto\n'
[2020-11-20 20:08:59,044] {pod_launcher.py:156} INFO - b'    return self._sock.recv_into(b)\n'
[2020-11-20 20:08:59,045] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/ssl.py", line 1241, in recv_into\n'
[2020-11-20 20:08:59,074] {pod_launcher.py:156} INFO - b'    return self.read(nbytes, buffer)\n'
[2020-11-20 20:08:59,074] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/ssl.py", line 1099, in read\n'
[2020-11-20 20:08:59,078] {pod_launcher.py:156} INFO - b'    return self._sslobj.read(len, buffer)\n'
[2020-11-20 20:08:59,078] {pod_launcher.py:156} INFO - b'ConnectionResetError: [Errno 104] Connection reset by peer\n'
[2020-11-20 20:08:59,078] {pod_launcher.py:156} INFO - b'\n'
[2020-11-20 20:08:59,078] {pod_launcher.py:156} INFO - b'During handling of the above exception, another exception occurred:\n'
[2020-11-20 20:08:59,079] {pod_launcher.py:156} INFO - b'\n'
[2020-11-20 20:08:59,079] {pod_launcher.py:156} INFO - b'Traceback (most recent call last):\n'
[2020-11-20 20:08:59,079] {pod_launcher.py:156} INFO - b'  File "/usr/local/bin/leanplum-data-export", line 33, in <module>\n'
[2020-11-20 20:08:59,079] {pod_launcher.py:156} INFO - b"    sys.exit(load_entry_point('leanplum-data-export', 'console_scripts', 'leanplum-data-export')())\n"
[2020-11-20 20:08:59,079] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__\n'
[2020-11-20 20:08:59,111] {pod_launcher.py:156} INFO - b'    return self.main(*args, **kwargs)\n'
[2020-11-20 20:08:59,112] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main\n'
[2020-11-20 20:08:59,117] {pod_launcher.py:156} INFO - b'    rv = self.invoke(ctx)\n'
[2020-11-20 20:08:59,117] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke\n'
[2020-11-20 20:08:59,117] {pod_launcher.py:156} INFO - b'    return _process_result(sub_ctx.command.invoke(sub_ctx))\n'
[2020-11-20 20:08:59,117] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke\n'
[2020-11-20 20:08:59,118] {pod_launcher.py:156} INFO - b'    return ctx.invoke(self.callback, **ctx.params)\n'
[2020-11-20 20:08:59,118] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke\n'
[2020-11-20 20:08:59,119] {pod_launcher.py:156} INFO - b'    return callback(*args, **kwargs)\n'
[2020-11-20 20:08:59,119] {pod_launcher.py:156} INFO - b'  File "/app/leanplum_data_export/leanplum_data_export/__main__.py", line 31, in export_leanplum\n'
[2020-11-20 20:08:59,119] {pod_launcher.py:156} INFO - b'    exporter.export(date, s3_bucket, bucket, prefix, bq_dataset, table_prefix, version, clean)\n'
[2020-11-20 20:08:59,119] {pod_launcher.py:156} INFO - b'  File "/app/leanplum_data_export/leanplum_data_export/export.py", line 53, in export\n'
[2020-11-20 20:08:59,120] {pod_launcher.py:156} INFO - b'    csv_file_paths = self.transform_data_file(key, schemas, data_dir, s3_bucket)\n'
[2020-11-20 20:08:59,120] {pod_launcher.py:156} INFO - b'  File "/app/leanplum_data_export/leanplum_data_export/export.py", line 164, in transform_data_file\n'
[2020-11-20 20:08:59,120] {pod_launcher.py:156} INFO - b'    self.s3_client.download_file(bucket, data_file_key, os.path.join(data_dir, "data.ndjson"))\n'
[2020-11-20 20:08:59,120] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 170, in download_file\n'
[2020-11-20 20:08:59,136] {pod_launcher.py:156} INFO - b'    return transfer.download_file(\n'
[2020-11-20 20:08:59,136] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/boto3/s3/transfer.py", line 307, in download_file\n'
[2020-11-20 20:08:59,140] {pod_launcher.py:156} INFO - b'    future.result()\n'
[2020-11-20 20:08:59,140] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 106, in result\n'
[2020-11-20 20:08:59,178] {pod_launcher.py:156} INFO - b'    return self._coordinator.result()\n'
[2020-11-20 20:08:59,179] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 265, in result\n'
[2020-11-20 20:08:59,179] {pod_launcher.py:156} INFO - b'    raise self._exception\n'
[2020-11-20 20:08:59,180] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 126, in __call__\n'
[2020-11-20 20:08:59,195] {pod_launcher.py:156} INFO - b'    return self._execute_main(kwargs)\n'
[2020-11-20 20:08:59,196] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 150, in _execute_main\n'
[2020-11-20 20:08:59,196] {pod_launcher.py:156} INFO - b'    return_value = self._main(**kwargs)\n'
[2020-11-20 20:08:59,196] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/s3transfer/download.py", line 521, in _main\n'
[2020-11-20 20:08:59,214] {pod_launcher.py:156} INFO - b'    for chunk in chunks:\n'
[2020-11-20 20:08:59,214] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/s3transfer/download.py", line 649, in __next__\n'
[2020-11-20 20:08:59,218] {pod_launcher.py:156} INFO - b'    chunk = self._body.read(self._chunksize)\n'
[2020-11-20 20:08:59,218] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/s3transfer/utils.py", line 551, in read\n'
[2020-11-20 20:08:59,219] {pod_launcher.py:156} INFO - b'    value = self._stream.read(*args, **kwargs)\n'
[2020-11-20 20:08:59,219] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/botocore/response.py", line 77, in read\n'
[2020-11-20 20:08:59,238] {pod_launcher.py:156} INFO - b'    chunk = self._raw_stream.read(amt)\n'
[2020-11-20 20:08:59,238] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 540, in read\n'
[2020-11-20 20:08:59,238] {pod_launcher.py:156} INFO - b'    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)\n'
[2020-11-20 20:08:59,238] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/contextlib.py", line 135, in __exit__\n'
[2020-11-20 20:08:59,257] {pod_launcher.py:156} INFO - b'    self.gen.throw(type, value, traceback)\n'
[2020-11-20 20:08:59,258] {pod_launcher.py:156} INFO - b'  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 454, in _error_catcher\n'
[2020-11-20 20:08:59,258] {pod_launcher.py:156} INFO - b'    raise ProtocolError("Connection broken: %r" % e, e)\n'
[2020-11-20 20:08:59,259] {pod_launcher.py:156} INFO - b'urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, \'Connection reset by peer\')", ConnectionResetError(104, \'Connection reset by peer\'))\n'