Alternator uses "salted_hash" as the password

What happened?

Listing the roles/Users created in cqlsh:

cassandra@cqlsh> SELECT * from system_auth.roles;

 role            | can_login | is_superuser | member_of | salted_hash
-----------------+-----------+--------------+-----------+------------------------------------------------------------------------------------------------------------
         db_user |     False |        False |      null | $6$E2QUdN1.HUEq8hUf$DajMK7y/JX1JZFQkwrw7MBwm9ALVk9jIF6QbAJYRJgD8FDnccu3MiLIcxLYB7RfFy4J3P7ViScVt.5Q4SMfRI0
       cassandra |      True |         True |      null | $6$S0kE.KaWEwRtzNH9$.A2xIt9YRH0zZIGmLN3crt2TJlSQjd2htqTtRl3z/nxSn9h3VpOQm/6XR97ui5/anV6q7K4eMWAhGaZ9t7cUP/
 alternator_user |      True |        False |      null | $6$eaYznUb2qJXs3are$xUxSlUq4IAvfM6sECrZ3fTa3fpcL0z1gatOWh0aal.3/rGQYcyL/ZX57heDqDat5M/lPf078WMoz9uwAP8NtH1

Able to connect using csql db_user and password:

kubectl exec -n scylla -it scylla-us-east-1-us-east-1a-0 -c scylla -- cqlsh --username db_user --password password

Warning: Using a password on the command line interface can be insecure.
Recommendation: use the credentials file to securely provide the password.

Connected to scylla at 0.0.0.0:9042
[cqlsh 6.2.0 | Scylla 5.4.3-0.20240211.cf42ca0c2a65 | CQL spec 3.3.1 | Native protocol v4]

When trying to access the alternator using awscli or boto3 I get this error:

export AWS_ACCESS_KEY_ID=db_user
export AWS_SECRET_ACCESS_KEY=password
export AWS_DEFAULT_REGION=us-east-1

scylla aws  --endpoint-url 'http://127.0.0.1:8888'  dynamodb create-table --table-name MusicCollection --attribute-definitions AttributeName=Artist,AttributeType=S AttributeName=SongTitle,AttributeType=S --key-schema AttributeName=Artist,KeyType=HASH AttributeName=SongTitle,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

An error occurred (UnrecognizedClientException) when calling the CreateTable operation: The security token included in the request is invalid.

Working:

export AWS_ACCESS_KEY_ID=db_user
export AWS_SECRET_ACCESS_KEY=$6$E2QUdN1.HUEq8hUf$DajMK7y/JX1JZFQkwrw7MBwm9ALVk9jIF6QbAJYRJgD8FDnccu3MiLIcxLYB7RfFy4J3P7ViScVt.5Q4SMfRI0
export AWS_DEFAULT_REGION=us-east-1

scylla aws  --endpoint-url 'http://127.0.0.1:8888'  dynamodb create-table --table-name MusicCollection --attribute-definitions AttributeName=Artist,AttributeType=S AttributeName=SongTitle,AttributeType=S --key-schema AttributeName=Artist,KeyType=HASH AttributeName=SongTitle,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
{
    "TableDescription": {
        "AttributeDefinitions": [
            {
                "AttributeName": "Artist",
                "AttributeType": "S"
            },
            {
                "AttributeName": "SongTitle",
                "AttributeType": "S"
            }
        ],
        "TableName": "MusicCollection",
        "KeySchema": [
            {
                "AttributeName": "Artist",
                "KeyType": "HASH"
            },
            {
                "AttributeName": "SongTitle",
                "KeyType": "RANGE"
            }
        ],
        "TableStatus": "ACTIVE",
        "CreationDateTime": "2024-04-03T19:51:20+03:00",
        "ProvisionedThroughput": {
            "ReadCapacityUnits": 5,
            "WriteCapacityUnits": 5
        },
        "TableId": "65d645d0-f1da-11ee-8903-227db54d6c7c"
    }
}

Seems by using the "salted_hash" as the actual password, the connection works. This seems very counter intuitive, is this by design or a flaw?

What did you expect to happen?

Expecting the password for db_user in scylla and alternator to be the same. Instead alternator uses the value of the salted_hash as the actual password. ie:

export AWS_ACCESS_KEY_ID=db_user
export AWS_SECRET_ACCESS_KEY=password
export AWS_DEFAULT_REGION=us-east-1

scylla aws  --endpoint-url 'http://127.0.0.1:8888'  dynamodb create-table --table-name MusicCollection --attribute-definitions AttributeName=Artist,AttributeType=S AttributeName=SongTitle,AttributeType=S --key-schema AttributeName=Artist,KeyType=HASH AttributeName=SongTitle,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
{
    "TableDescription": {
        "AttributeDefinitions": [
            {
                "AttributeName": "Artist",
                "AttributeType": "S"
            },
            {
                "AttributeName": "SongTitle",
                "AttributeType": "S"
            }
        ],
        "TableName": "MusicCollection",
        "KeySchema": [
            {
                "AttributeName": "Artist",
                "KeyType": "HASH"
            },
            {
                "AttributeName": "SongTitle",
                "KeyType": "RANGE"
            }
        ],
        "TableStatus": "ACTIVE",
        "CreationDateTime": "2024-04-03T19:51:20+03:00",
        "ProvisionedThroughput": {
            "ReadCapacityUnits": 5,
            "WriteCapacityUnits": 5
        },
        "TableId": "65d645d0-f1da-11ee-8903-227db54d6c7c"
    }
}

How can we reproduce it (as minimally and precisely as possible)?

Deploy scylla-operator.
Deploy ScyllaDB

Create a role or a user:

CREATE ROLE db_user WITH PASSWORD = 'password' ;

Attempt to connect.

Scylla Operator version

Scylla-Operator 1.11.3 & 1.12.0

Kubernetes platform name and version

```console $ kubectl version Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.11 ```

Please attach the must-gather archive.

...

Anything else we need to know?

No response

scylladb / scylla-operator