search

DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
http://toil.ucsc-cgl.org/.
Apache License 2.0
891 stars 241 forks source link

`src/toil/test/lib/test_ec2.py::AMITest::test_fetch_flatcar` is flaky #4726

Open adamnovak opened 8 months ago

adamnovak commented 8 months ago

AWS wasn't taking our credentials for a while recently, and src/toil/test/lib/test_ec2.py::AMITest::test_fetch_flatcar was failing with a complaint that AWS wouldn't accept the credentials to poll for image existence.

https://ucsc-ci.com/databiosphere/toil/-/jobs/57841#L2670

If we can't poll the image at all we should assume it actually exists.

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1472

adamnovak commented 5 months ago

I think this test is still flaky, or flaky again:

[gw2] linux -- Python 3.11.5 /builds/databiosphere/toil/venv/bin/python
Traceback (most recent call last):
  File "/usr/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/usr/lib/python3.11/unittest/case.py", line 623, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/builds/databiosphere/toil/src/toil/test/lib/test_ec2.py", line 83, in test_fetch_flatcar
    ami = aws_marketplace_flatcar_ami_search(self.ec2_client)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/databiosphere/toil/src/toil/lib/retry.py", line 292, in call
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/builds/databiosphere/toil/src/toil/lib/aws/ami.py", line 184, in aws_marketplace_flatcar_ami_search
    response: dict = ec2_client.describe_images(Owners=['aws-marketplace'],  # type: ignore
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/databiosphere/toil/venv/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/databiosphere/toil/venv/lib/python3.11/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the DescribeImages operation: AWS was not able to validate the provided access credentials

https://ucsc-ci.com/databiosphere/toil/-/jobs/63445/raw

We should handle this error as well.