Track down and fix build issues

[kozbo]

Why

We are experiencing too many build failures. As a developer I would like to have confidence that new code submissions would not be delayed by test failures in unrelated areas.

DoD

• [x] catalog test faailures

[Bento007]

https://travis-ci.com/HumanCellAtlas/data-store/jobs/131199111

======================================================================
FAIL: test_multiple_schema_version_subscription_indexing_and_notification (__main__.TestGCPIndexer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_indexer.py", line 771, in test_multiple_schema_version_subscription_indexing_and_notification
    bundle_key = self.load_test_data_bundle_for_path("fixtures/indexing/bundles/v3/smartseq2/paired_ends")
  File "tests/test_indexer.py", line 1047, in load_test_data_bundle_for_path
    return self.load_test_data_bundle(bundle)
  File "tests/test_indexer.py", line 1065, in load_test_data_bundle
    self.upload_files_and_create_bundle(bundle, self.replica)
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/storage_mixin.py", line 49, in upload_files_and_create_bundle
    version = self.upload_file(file, replica)
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/storage_mixin.py", line 57, in upload_file
    file_uuid=bundle_file.uuid
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/upload_mixin.py", line 50, in upload_file_wait
    source_url=source_url,
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/assert_mixin.py", line 87, in assertResponse
    self.assertIn(response.status_code, expected_code)
AssertionError: 500 not found in (201, 202)

[Bento007]

https://travis-ci.com/HumanCellAtlas/data-store/builds/77498998

======================================================================
ERROR: test_subscription_query_with_multiple_data_types_indexing_and_notification (__main__.TestAWSIndexer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_indexer.py", line 568, in test_subscription_query_with_multiple_data_types_indexing_and_notification
    self.process_new_indexable_object(sample_event)
  File "tests/test_indexer.py", line 135, in process_new_indexable_object
    self.indexer.process_new_indexable_object(event)
  File "/home/travis/build/HumanCellAtlas/data-store/dss/index/indexer.py", line 32, in process_new_indexable_object
    self.index_object(key)
  File "/home/travis/build/HumanCellAtlas/data-store/dss/index/indexer.py", line 44, in index_object
    self._index_bundle(identifier)
  File "/home/travis/build/HumanCellAtlas/data-store/dss/index/indexer.py", line 63, in _index_bundle
    self.backend.index_bundle(bundle)
  File "/home/travis/build/HumanCellAtlas/data-store/dss/index/backend.py", line 95, in index_bundle
    self._delegate(self.index_bundle, args, kwargs)
  File "/home/travis/build/HumanCellAtlas/data-store/dss/index/backend.py", line 121, in _delegate
    raise RuntimeError(f"One or more backends failed: {problems}")
RuntimeError: One or more backends failed: ["Backend ElasticsearchIndexBackend(dryrun=False, notify=True) raised an exception: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='localhost', port=44051): Read timed out. (read timeout=10))"]
----------------------------------------------------------------------

Possible Reason

[2018-06-27T18:13:30,332][WARN ][o.e.b.BootstrapChecks ] [HVedpj-] max file descriptors [30000] for elasticsearch process is too low, increase to at least [65536]` [2018-06-27T18:13:30,334][WARN ][o.e.b.BootstrapChecks ] [HVedpj-] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Making these values exceptable on travis is not possible. It can be done in circle-ci using an elastic search docker container. Though I'm still unsure whether this is the cause.

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/system-config.html

[Bento007]

FAIL: test_reindexing_with_changed_shape (__main__.TestGCPIndexer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_indexer.py", line 286, in test_reindexing_with_changed_shape
    bundle_key = self.load_test_data_bundle_for_path("fixtures/indexing/bundles/v3/smartseq2/paired_ends")
  File "tests/test_indexer.py", line 1047, in load_test_data_bundle_for_path
    return self.load_test_data_bundle(bundle)
  File "tests/test_indexer.py", line 1065, in load_test_data_bundle
    self.upload_files_and_create_bundle(bundle, self.replica)
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/storage_mixin.py", line 49, in upload_files_and_create_bundle
    version = self.upload_file(file, replica)
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/storage_mixin.py", line 57, in upload_file
    file_uuid=bundle_file.uuid
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/upload_mixin.py", line 50, in upload_file_wait
    source_url=source_url,
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/assert_mixin.py", line 98, in assertResponse
    self.assertIn(response.status_code, expected_code)
AssertionError: 500 not found in (201, 202)
----------------------------------------------------------------------

[Bento007]

from circle CLI

FAIL: test_file_size (__main__.TestFileApi)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_file.py", line 364, in test_file_size
    self._test_file_size(Replica.gcp, "gs", self.gs_test_bucket, GSUploader(tempdir, self.gs_test_bucket))
  File "tests/test_file.py", line 394, in _test_file_size
    override_retry_interval=1,
  File "/home/circle/project/tests/infra/assert_mixin.py", line 100, in assertResponse
    self.assertEqual(response.status_code, expected_code)
AssertionError: 301 != 302

----------------------------------------------------------------------

[Bento007]

https://circleci.com/gh/HumanCellAtlas/data-store/2570

ERROR:connexion.decorators.validation:https://127.0.0.1:42307/v1/bundles/155fe3ab-d9d8-49b1-98fc-08384c4d92a3?replica=gcp&version=2018-06-28T201545.058599Z validation error: 'service_unavailable' is not one of ['unhandled_exception', 'illegal_arguments', 'bundle_already_exists']

Failed validating 'enum' in schema['allOf'][1]['properties']['code']:
    {'description': 'Machine-readable error code.  The types of return '
                    'values should not be changed lightly.',
     'enum': ['unhandled_exception',
              'illegal_arguments',
              'bundle_already_exists'],
     'type': 'string'}

On instance['code']:
    'service_unavailable'
INFO:dss:[dispatch] "PUT /v1/bundles/155fe3ab-d9d8-49b1-98fc-08384c4d92a3" 500 {'replica': 'gcp', 'version': '2018-06-28T201545.058599Z'}
Response:
{'detail': "'service_unavailable' is not one of ['unhandled_exception', "
           "'illegal_arguments', 'bundle_already_exists']\n"
           '\n'
           "Failed validating 'enum' in "
           "schema['allOf'][1]['properties']['code']:\n"
           "    {'description': 'Machine-readable error code.  The types of "
           "return '\n"
           "                    'values should not be changed lightly.',\n"
           "     'enum': ['unhandled_exception',\n"
           "              'illegal_arguments',\n"
           "              'bundle_already_exists'],\n"
           "     'type': 'string'}\n"
           '\n'
           "On instance['code']:\n"
           "    'service_unavailable'",
 'status': 500,
 'title': 'Response body does not conform to specification',
 'type': 'about:blank'}
FAIL

 ======================================================================
FAIL: test_bundle_put (__main__.TestBundleApi)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_bundle.py", line 218, in test_bundle_put
    self._test_bundle_put(Replica.gcp, self.gs_test_fixtures_bucket)
  File "tests/test_bundle.py", line 240, in _test_bundle_put
    bundle_version,
  File "tests/test_bundle.py", line 462, in put_bundle
    creator_uid=12345,
  File "/home/circle/project/tests/infra/assert_mixin.py", line 100, in assertResponse
    self.assertEqual(response.status_code, expected_code)
AssertionError: 500 != 201

----------------------------------------------------------------------

[Bento007]

https://circleci.com/gh/HumanCellAtlas/data-store/2575

INFO:dss:[dispatch] "PUT /v1/files/2a2bc471-65ba-48c7-aea6-ed8ae246f0c3" 500 {'version': '2018-06-28T202207.055013Z'}
Response:
{'code': 'unhandled_exception',
 'stacktrace': 'Traceback (most recent call last):\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", '
               'line 387, in _make_request\n'
               '    six.raise_from(e, None)\n'
               '  File "<string>", line 2, in raise_from\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", '
               'line 383, in _make_request\n'
               '    httplib_response = conn.getresponse()\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/http/client.py", '
               'line 1331, in getresponse\n'
               '    response.begin()\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/http/client.py", '
               'line 297, in begin\n'
               '    version, status, reason = self._read_status()\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/http/client.py", '
               'line 258, in _read_status\n'
               '    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/socket.py", '
               'line 586, in readinto\n'
               '    return self._sock.recv_into(b)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/ssl.py", line '
               '1009, in recv_into\n'
               '    return self.read(nbytes, buffer)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/ssl.py", line '
               '871, in read\n'
               '    return self._sslobj.read(len, buffer)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/ssl.py", line '
               '631, in read\n'
               '    v = self._sslobj.read(len, buffer)\n'
               'socket.timeout: The read operation timed out\n'
               '\n'
               'During handling of the above exception, another exception '
               'occurred:\n'
               '\n'
               'Traceback (most recent call last):\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/requests/adapters.py", '
               'line 440, in send\n'
               '    timeout=timeout\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", '
               'line 639, in urlopen\n'
               '    _stacktrace=sys.exc_info()[2])\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/util/retry.py", '
               'line 357, in increment\n'
               '    raise six.reraise(type(error), error, _stacktrace)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/packages/six.py", '
               'line 686, in reraise\n'
               '    raise value\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", '
               'line 601, in urlopen\n'
               '    chunked=chunked)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", '
               'line 389, in _make_request\n'
               '    self._raise_timeout(err=e, url=url, '
               'timeout_value=read_timeout)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", '
               'line 309, in _raise_timeout\n'
               '    raise ReadTimeoutError(self, url, "Read timed out. (read '
               'timeout=%s)" % timeout_value)\n'
               'urllib3.exceptions.ReadTimeoutError: '
               "HTTPSConnectionPool(host='www.googleapis.com', port=443): Read "
               'timed out. (read timeout=5)\n'
               '\n'
               'During handling of the above exception, another exception '
               'occurred:\n'
               '\n'
               'Traceback (most recent call last):\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/cloud_blobstore/gs.py", '
               'line 17, in wrapped\n'
               '    return meth(*args, **kwargs)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/cloud_blobstore/gs.py", '
               'line 145, in upload_file_handle\n'
               '    blob_obj.upload_from_file(src_file_handle, '
               'content_type=content_type)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/google/cloud/storage/blob.py", '
               'line 975, in upload_from_file\n'
               '    size, num_retries, predefined_acl)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/google/cloud/storage/blob.py", '
               'line 892, in _do_upload\n'
               '    num_retries, predefined_acl)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/google/cloud/storage/blob.py", '
               'line 839, in _do_resumable_upload\n'
               '    response = upload.transmit_next_chunk(transport)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/google/resumable_media/requests/upload.py", '
               'line 395, in transmit_next_chunk\n'
               '    retry_strategy=self._retry_strategy)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/google/resumable_media/requests/_helpers.py", '
               'line 101, in http_request\n'
               '    func, RequestsMixin._get_status_code, retry_strategy)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/google/resumable_media/_helpers.py", '
               'line 146, in wait_and_retry\n'
               '    response = func()\n'
               '  File "/home/circle/project/dss/config.py", line 152, in '
               'request\n'
               '    return super().request(*args, **kwargs)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/google/auth/transport/requests.py", '
               'line 186, in request\n'
               '    method, url, data=data, headers=request_headers, '
               '**kwargs)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/requests/sessions.py", '
               'line 508, in request\n'
               '    resp = self.send(prep, **send_kwargs)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/requests/sessions.py", '
               'line 618, in send\n'
               '    r = adapter.send(request, **kwargs)\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/requests/adapters.py", '
               'line 521, in send\n'
               '    raise ReadTimeout(e, request=request)\n'
               'requests.exceptions.ReadTimeout: '
               "HTTPSConnectionPool(host='www.googleapis.com', port=443): Read "
               'timed out. (read timeout=5)\n'
               '\n'
               'During handling of the above exception, another exception '
               'occurred:\n'
               '\n'
               'Traceback (most recent call last):\n'
               '  File "/home/circle/project/dss/error.py", line 26, in '
               'wrapper\n'
               '    return func(*args, **kwargs)\n'
               '  File "/home/circle/project/dss/api/files.py", line 263, in '
               'put\n'
               '    write_file_metadata(handle, dst_bucket, uuid, version, '
               'file_metadata_json)\n'
               '  File "/home/circle/project/dss/storage/files.py", line 26, '
               'in write_file_metadata\n'
               '    io.BytesIO(document.encode("utf-8")))\n'
               '  File '
               '"/home/circle/pyenv/versions/3.6.3/lib/python3.6/site-packages/cloud_blobstore/gs.py", '
               'line 19, in wrapped\n'
               '    raise BlobStoreTimeoutError(ex)\n'
               'cloud_blobstore.BlobStoreTimeoutError: '
               "HTTPSConnectionPool(host='www.googleapis.com', port=443): Read "
               'timed out. (read timeout=5)\n',
 'status': 500,
 'title': "HTTPSConnectionPool(host='www.googleapis.com', port=443): Read "
          'timed out. (read timeout=5)'}
FAIL

======================================================================
FAIL: test_bundle_delete (__main__.TestBundleApi)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_bundle.py", line 314, in test_bundle_delete
    self._test_bundle_delete(Replica.gcp, self.gs_test_fixtures_bucket, True)
  File "tests/test_bundle.py", line 328, in _test_bundle_delete
    bundle_uuid=bundle_uuid,
  File "/home/circle/project/tests/infra/upload_mixin.py", line 50, in upload_file_wait
    source_url=source_url,
  File "/home/circle/project/tests/infra/assert_mixin.py", line 98, in assertResponse
    self.assertIn(response.status_code, expected_code)
AssertionError: 500 not found in (201, 202)

----------------------------------------------------------------------

[Bento007]

This area looks like source of the problem for upload failing. https://github.com/HumanCellAtlas/data-store/blob/9655df05f815d44ade3f8b38742b2d2f7d1d65dc/dss/config.py#L147-L164 We need to make sure we do retries if upload fails with a 500 error from google.

[Bento007]

This could be part of the solution https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2694#issuecomment-366975993

[Bento007]

Another possible solution to retrying google requests

• https://www.peterbe.com/plog/best-practice-with-retries-with-requests
• https://stackoverflow.com/questions/23267409/how-to-implement-retry-mechanism-into-python-requests-library?rq=1

[Bento007]

Reproduced the error https://travis-ci.com/HumanCellAtlas/data-store/jobs/132166037

[Bento007]

https://travis-ci.com/HumanCellAtlas/data-store/builds/77842980

======================================================================
ERROR: test_collections (__main__.TestCollections) [GET latest version of collection]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/opt/python/3.6.3/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/opt/python/3.6.3/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/opt/python/3.6.3/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/python/3.6.3/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 389, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=58027): Read timed out. (read timeout=30.0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "tests/test_collections.py", line 74, in test_collections
    params=dict(replica="aws"))
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/server.py", line 60, in _make_call
    **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/adapters.py", line 521, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=58027): Read timed out. (read timeout=30.0)
======================================================================
ERROR: test_collections (__main__.TestCollections)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/opt/python/3.6.3/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/opt/python/3.6.3/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/opt/python/3.6.3/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/python/3.6.3/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 389, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=58027): Read timed out. (read timeout=30.0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "tests/test_collections.py", line 79, in test_collections
    params=dict(replica="aws", version="9000"))
  File "/home/travis/build/HumanCellAtlas/data-store/tests/infra/server.py", line 60, in _make_call
    **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/requests/adapters.py", line 521, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=58027): Read timed out. (read timeout=30.0)
----------------------------------------------------------------------
Ran 1 test in 62.751s
FAILED (errors=2)
INFO:dss:[dispatch] "GET /v1/collections/10bca9b9-3c3d-471c-8edc-9ebedf1d1102" 200 {'replica': 'aws'}
----------------------------------------

[Bento007]

https://travis-ci.com/HumanCellAtlas/data-store/builds/77884346

======================================================================
ERROR: test_smoketest (__main__.Smoketest) [aws]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_smoketest.py", line 224, in test_smoketest
    self.smoketest(**param)
  File "tests/test_smoketest.py", line 215, in smoketest
    obj = s3.get_object(Bucket=self.notification_bucket, Key=notification_key)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
----------------------------------------------------------------------

[Bento007]

https://travis-ci.com/HumanCellAtlas/data-store/builds/77820537

======================================================================
FAIL: test_smoketest (__main__.Smoketest) [gcp]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_smoketest.py", line 224, in test_smoketest
    self.smoketest(**param)
  File "tests/test_smoketest.py", line 189, in smoketest
    self.assertEqual(len(res['results']), 1)
AssertionError: 5 != 1
----------------------------------------------------------------------

in total 5 duplicate bundles were stored and indexed. Each bundle was stored roughly 25 seconds apart. There was a total of 5 files indexed.