search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
55 stars 93 forks source link

Authenticating in AWS through Okta from CLI could not connect to the endpoint #8673

Closed swasik closed 57 minutes ago

swasik commented 1 week ago

Issue description

I try to configure SCT access to AWS through Okta but I keep getting "Could not connect to the endpoint URL: https://sts.c.amazonaws.com/" message. I am using Fedora 40. For Docker version and steps I used see below. I followed README from this repo and https://www.notion.so/AWS-864b26157112426f8e74bab61001425d for configuring AWS.

Steps to Reproduce

  1. I pulled scylla-cluster-tests repo.
  2. ./install-prereqs.sh (had some problems with installing all Python requirements but I resolved them manually)
  3. ./install-hydra.sh
  4. Create ~/.okta_aws_login_config acccording to https://www.notion.so/AWS-864b26157112426f8e74bab61001425d (for exact content see below)
  5. ./docker/env/hydra.sh list-images -c aws

Expected behavior: Link to Okta authentication window.

Actual behavior: Exception (see log below).

Impact

I cannot authenticate in AWS.

How frequently does it reproduce?

Always.

Installation details

SCT Version: master (head at 235913f8eec161c142fc7c8ebb7d1606da5e587f) Scylla version (or git commit hash): N/A

Logs

Docker version:

dnf list installed|grep docker

containerd.io.x86_64                                 1.7.22-3.1.fc40                     @docker-ce-stable     
docker-buildx-plugin.x86_64                          0.16.2-1.fc40                       @docker-ce-stable     
docker-ce.x86_64                                     3:27.2.1-1.fc40                     @docker-ce-stable     
docker-ce-cli.x86_64                                 1:27.2.1-1.fc40                     @docker-ce-stable     
docker-ce-rootless-extras.x86_64                     27.2.1-1.fc40                       @docker-ce-stable     
docker-compose-plugin.x86_64                         2.29.2-1.fc40                       @docker-ce-stable

Hydra logs:

./docker/env/hydra.sh list-images -c aws

There is scylladb/hydra:v1.75-event-collect-argus-0.12.7 in local cache, using it.
Going to run './sct.py  list-images -c aws'...
/usr/local/lib/python3.10/site-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "cipher": algorithms.TripleDES,
/usr/local/lib/python3.10/site-packages/paramiko/transport.py:259: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "class": algorithms.TripleDES,
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/botocore/httpsession.py", line 465, in send
    urllib_response = conn.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 802, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 527, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect
    self.sock = conn = self._new_conn()
  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f6044593310>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/swasik/git/scylla-cluster-tests/./sct.py", line 1866, in <module>
    cli.main(prog_name="hydra")
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1654, in invoke
    super().invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/swasik/git/scylla-cluster-tests/./sct.py", line 196, in cli
    try_auth_with_okta()
  File "/home/swasik/git/scylla-cluster-tests/sdcm/utils/aws_okta.py", line 61, in try_auth_with_okta
    if not can_get_to_aws_account():
  File "/home/swasik/git/scylla-cluster-tests/sdcm/utils/aws_okta.py", line 39, in can_get_to_aws_account
    response = sts.get_caller_identity()
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 534, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 959, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 982, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/lib/python3.10/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/usr/local/lib/python3.10/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/usr/local/lib/python3.10/site-packages/botocore/httpsession.py", line 494, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://sts.c.amazonaws.com/"
fruch commented 1 week ago

the url https://sts.c.amazonaws.com/ is very weird

@swasik can you share the output of env | grep SCT and cat ~/.aws/config

seems like for some reason the region is c

fruch commented 1 hour ago

@swasik did you got this figured out ?

swasik commented 57 minutes ago

Sorry, I had notifications misconfigured (routed to my previous work address) so I missed the updates here. You are right - the region was set incorrectly. I have no idea why but after fixing it is failing with different error message - I will try to resolve this.