TestGSCopy.test_simple_copy fails sporadically

hannes-ucsc commented 6 years ago

https://travis-ci.org/HumanCellAtlas/data-store/builds/371635683#L1489:

coverage run -p --source=dss tests/test_gscopyclient.py -v
test_simple_copy (__main__.TestGSCopy) ... ERROR
======================================================================
ERROR: test_simple_copy (__main__.TestGSCopy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_gscopyclient.py", line 83, in test_simple_copy
    test_output()
  File "/home/travis/build/HumanCellAtlas/data-store/tests/__init__.py", line 60, in call
    return func(*args, **kwargs)
  File "tests/test_gscopyclient.py", line 80, in test_output
    dst_checksum = self.gs_blobstore.get_cloud_checksum(self.test_bucket, dest_key)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/cloud_blobstore/gs.py", line 176, in get_cloud_checksum
    raise BlobNotFoundError()
cloud_blobstore.BlobNotFoundError
----------------------------------------------------------------------
Ran 1 test in 51.615s
FAILED (errors=1)
Makefile:44: recipe for target 'tests/test_gscopyclient.py' failed
make[1]: *** [tests/test_gscopyclient.py] Error 1
make[1]: Leaving directory '/home/travis/build/HumanCellAtlas/data-store'
Makefile:54: recipe for target 'integration_test' failed
make: *** [integration_test] Error 2

There already is a 30s busy loop around the check so I don't think this is eventual consistency.

hannes-ucsc commented 6 years ago

We should print/log the missing key when the exceptions occurs so we search the logs. Ideally, the missing key should be part of the exception message but that fix would have to occur in cloud-blobstore.

hannes-ucsc commented 6 years ago

https://travis-ci.org/HumanCellAtlas/data-store/builds/371799785#L1483

HumanCellAtlas / data-store

TestGSCopy.test_simple_copy fails sporadically #1190