openml / automlbenchmark

OpenML AutoML Benchmarking Framework
https://openml.github.io/automlbenchmark
MIT License
399 stars 132 forks source link

Document connection issue due to out of sync clock #451

Open PGijsbers opened 2 years ago

PGijsbers commented 2 years ago

I experienced connection issues after the server clock got out of sync:

An error occurred (403) when calling the HeadBucket operation: Forbidden
Traceback (most recent call last):
  File "runbenchmark.py", line 181, in <module>
    bench.setup(amlb.SetupMode[args.setup])
  File ".../automlbenchmark/amlb/runners/aws.py", line 164, in setup
    self.bucket = self._create_s3_bucket()
  File ".../automlbenchmark/amlb/runners/aws.py", line 809, in _create_s3_bucket
    raise e
  File ".../automlbenchmark/amlb/runners/aws.py", line 788, in _create_s3_bucket
    self.s3.meta.client.head_bucket(Bucket=bucket_name)
  File ".../automlbenchmark/venv/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File ".../automlbenchmark/venv/lib/python3.8/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden

It comes with this error. In my case the issue were not the credentials, but the clock being out of sync, which was identified through testing with the aws cli:

>aws s3api list-buckets

An error occurred (RequestTimeTooSkewed) when calling the ListBuckets operation: The difference between the request time and the current time is too large.

We need to check if we extract this RequestTimeTooSkewed from the thrown exception. If so, we need to ensure it is shown to the user. If not, we need to mention this possible cause (and/or maybe we can set up a separate test that does throw that error if it's a clock issue, so we can automate the diagnostics).