When TLS encryption is enabled, hostname validation cannot be enforced for multiple/all cluster nodes, which have individual certificates

Issue description

When enforcing hostname validation, scylla-bench requires to provide server hostname for validation. If provided, only one server name is accepted/processed.

When cluster nodes configured to have individual certificates, it is not possible to provide all names for validation when scylla-bench is executed against several cluster nodes.

There are probably 2 alternatives to consider:

do not require server name at all. In this case scylla-bench should parse names/IPs used for connection (-nodes parameter value) and compare name/IP with the identifiers that are included in SAN extension of server certificate, when it is provided by the corresponding server during SSL handshake.
Adjust parsing of parameters, so that -tls-server-name parameter can take a string of comma-separated names/IPs. E.g. : -tls-server-name 10.10.10.10,10.10.10.11,10.10.10.12. Or that the parameter can be provided several times, for each node that the commands connects to. E.g.: -tls-server-name 10.10.10.10 -tls-server-name 10.10.10.11 -tls-server-name 10.10.10.12.

Steps to Reproduce

Deploy a multinode cluster, e.g. with 3 DB nodes, with client encryption enabled in scylla.yaml:

client_encryption_options:
certificate: /etc/scylla/ssl_conf/client/client.crt
enabled: true
keyfile: /etc/scylla/ssl_conf/client/client.key
truststore: /etc/scylla/ssl_conf/ca.pem

For each cluster node create and distribute their individual certificates. The certificates should include SAN extension with included IP identifiers, specific for a node: db-node-1

ubuntu@dmitriy-loader-node-ad1c2d9d-1:~$ ssh 34.244.102.85 openssl x509 -text -noout -in /etc/scylla/ssl_conf/db.crt | grep -A1 'Subject Alternative Name'
        X509v3 Subject Alternative Name: 
            DNS:dmitriy-db-node-ad1c2d9d-1, IP Address:10.4.0.162, IP Address:34.244.102.85, DNS:ec2-34-244-102-85.eu-west-1.compute.amazonaws.com, DNS:ip-10-4-0-162.eu-west-1.compute.internal

db-node-2

ubuntu@dmitriy-loader-node-ad1c2d9d-1:~$ ssh 54.246.18.8 openssl x509 -text -noout -in /etc/scylla/ssl_conf/db.crt | grep -A1 'Subject Alternative Name'
        X509v3 Subject Alternative Name: 
            DNS:dmitriy-db-node-ad1c2d9d-2, IP Address:10.4.0.174, IP Address:54.246.18.8, DNS:ec2-54-246-18-8.eu-west-1.compute.amazonaws.com, DNS:ip-10-4-0-174.eu-west-1.compute.internal

db-node-3

ubuntu@dmitriy-loader-node-ad1c2d9d-1:~$ ssh 34.247.196.216 openssl x509 -text -noout -in /etc/scylla/ssl_conf/db.crt | grep -A1 'Subject Alternative Name'
        X509v3 Subject Alternative Name: 
            DNS:dmitriy-db-node-ad1c2d9d-3, IP Address:10.4.3.175, IP Address:34.247.196.216, DNS:ec2-34-247-196-216.eu-west-1.compute.amazonaws.com, DNS:ip-10-4-3-175.eu-west-1.compute.internal

From a client node execute scylla-bench command against all cluster nodes, with enabled TLS, but without host verification enforced. Expected result: the command executes the requested workload Actual result: as expected, the command executes the requested workload

ubuntu@dmitriy-loader-node-ad1c2d9d-1:~$ scylla-bench -workload sequential -mode write -duration 5s -nodes 34.244.102.85,54.246.18.8,34.247.196.216 -tls
Configuration
Mode:           write
Workload:       sequential
Timeout:        5s
Max error number at row: unlimited
Max error number:   unlimited
Retries:        
number:       10
min interval:     80ms
max interval:     1s
handler:      sb
Consistency level:  quorum
Partition count:    10000
Clustering rows:    100
Clustering row size:    Fixed(4)
Rows per request:   1
Page size:      1000
Concurrency:        16
Connections:        4
Maximum rate:       unlimited
Client compression: true
Hdr memory consumption: 5247232 bytes

time   ops/s  rows/s errors max    99.9th 99th   95th   90th   median mean   
1s   22599   22599      0 4.2ms  1.7ms  1.3ms  1.1ms  983µs  688µs  700µs  
2s   22672   22672      0 4.2ms  1.7ms  1.2ms  1ms    950µs  688µs  697µs  
3s   22793   22793      0 2.3ms  1.5ms  1.2ms  1ms    950µs  688µs  693µs  
...

Re-execute the same scylla-bench command, but with host verification enforced and server names provided. Expected result: the command executes the requested workload Actual result:
- the scylla-bench accepts only one name, if the names are provided separately. For other names it throws error: failed to connect to error;
- the command doesn't accept nodes if they are provided as a comma-separated string.

The names are provided separately:

ubuntu@dmitriy-loader-node-ad1c2d9d-1:~$ scylla-bench -workload sequential -mode write -duration 5s -nodes 34.244.102.85,54.246.18.8,34.247.196.216 -tls -tls-host-verification -tls-ca-cert-file /etc/scylla/ssl_conf/ca.pem  -tls-server-name 34.244.102.85 -tls-server-name 54.246.18.8 -tls-server-name 34.247.196.216
2024/05/29 11:44:34 gocql: unable to dial control conn 34.244.102.85:9042: x509: certificate is valid for 10.4.0.162, 34.244.102.85, not 34.247.196.216
2024/05/29 11:44:37 error: failed to connect to "[HostInfo hostname=\"10.4.0.162\" connectAddress=\"10.4.0.162\" peer=\"10.4.0.162\" rpc_address=\"10.4.0.162\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.4.0.162\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"87fd1104-4a1f-484c-bca1-f4288bf34c66\" version=\"v3.0.8\" state=UP num_tokens=256]" due to error: x509: certificate is valid for 10.4.0.162, 34.244.102.85, not 34.247.196.216
2024/05/29 11:44:37 error: failed to connect to "[HostInfo hostname=\"10.4.0.174\" connectAddress=\"10.4.0.174\" peer=\"10.4.0.174\" rpc_address=\"10.4.0.174\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.4.0.174\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"94b83540-0af7-4e8e-aab0-cdf8459577df\" version=\"v3.0.8\" state=UP num_tokens=256]" due to error: x509: certificate is valid for 10.4.0.174, 54.246.18.8, not 34.247.196.216

Configuration
Mode:           write
Workload:       sequential
Timeout:        5s
Max error number at row: unlimited
Max error number:   unlimited
Retries:        
  number:       10
  min interval:     80ms
  max interval:     1s
  handler:      sb
Consistency level:  quorum
...

The names are provided as a comma-separated list:

ubuntu@dmitriy-loader-node-ad1c2d9d-1:~$ scylla-bench -workload sequential -mode write -duration 5s -nodes 34.244.102.85,54.246.18.8,34.247.196.216 -tls -tls-host-verification -tls-ca-cert-file /etc/scylla/ssl_conf/ca.pem  -tls-server-name 34.244.102.85,54.246.18.8,34.247.196.216
2024/05/29 11:43:33 gocql: unable to create session: unable to discover protocol version: remote error: tls: illegal parameter

Installation details

scylla-bench: scylladb/hydra-loaders:scylla-bench-v0.1.20 (docker) Scylla version: 5.4.4-0.20240228.58a1be93b212 with build-id f385c89e3d29e70e0a8beab1dfefcace5e909775 Platforn: AWS/eu-west-1 DB nodes num: 3 DB node instance type: i4i.large Loaders num: 1 Loader node instance type: c5.large

Fixes: https://github.com/scylladb/qa-tasks/issues/1728

scylladb / scylla-bench