wazuh / wazuh-kubernetes

Wazuh - Wazuh Kubernetes
https://wazuh.com/
GNU General Public License v2.0
265 stars 163 forks source link

Change service name for cURL errors #581

Closed vcerenu closed 8 months ago

vcerenu commented 9 months ago

The connection between Wazuh manager and Wazuh indexer was recently added for the deployment of the new Vulnerability Detection, which uses the cURL libraries to make the connection.

When performing a connection test with cURL we obtained the following error:

root@wazuh-manager-master-0:/etc/filebeat# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" - -cacert "/etc/ssl/root-ca.pem" "https://wazuh-indexer-0.wazuh-indexer:9200"
curl: (60) SSL: certificate subject name '*.wazuh-indexer' does not match target host name 'wazuh-indexer-0.wazuh-indexer'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@wazuh-manager-master-0:/etc/filebeat#

This error is generated when the character "-" is found within the CN using a wildcard to be able to generate the connection from any Wazuh indexer node that is created within the K8S cluster.

We have carried out tests with names without "-" and the results have been satisfactory:

root@wazuh-manager-master-0:/# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" --cacert " /etc/ssl/root-ca.pem" "https://0.wazuh.indexer:9200"
{
   "name" : "wazuh-indexer-0",
   "cluster_name" : "wazuh",
   "cluster_uuid" : "3w73me1MQTmRsucMVh_tfA",
   "version" : {
     "number" : "7.10.2",
     "build_type" : "rpm",
     "build_hash" : "eee49cb340edc6c4d489bcd9324dda571fc8dc03",
     "build_date" : "2023-09-20T23:54:29.889267151Z",
     "build_snapshot" : false,
     "lucene_version" : "9.7.0",
     "minimum_wire_compatibility_version" : "7.10.0",
     "minimum_index_compatibility_version" : "7.0.0"
   },
   "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
root@wazuh-manager-master-0:/#

We will proceed to modify the name of the Wazuh indexer service so that we do not have this error.

It should be noted that the problem only occurs when the CN of the certificate contains a wildcard and among the remaining characters there is a "-"

c-bordon commented 9 months ago

Update report

I was carrying out several tests, and unfortunately, the only valid character for the DNS definition is - or lower case. For this reason, I chose to try wazuhindexer. but despite having made the changes the error persists:

The Service "wazuh.indexer" is invalid: metadata.name: Invalid value: "wazuh.indexer": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name',  or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-kubernetes$ wazuh/certs/indexer_cluster/generate_certs.sh
Root CA
Admin cert
create: admin-key-temp.pem
create: admin-key.pem
create: admin.csr
Ignoring -days without -x509; not generating a certificate
create: admin.pem
Certificate request self-signature ok
subject=C = US, L = California, O = Company, CN = admin
* Node cert
create: node-key-temp.pem
create: node-key.pem
create: node.csr
Ignoring -days without -x509; not generating a certificate
create: node.pem
Certificate request self-signature ok
subject=C = US, L = California, O = Company, CN = *.wazuhindexer
* dashboard cert
create: dashboard-key-temp.pem
create: dashboard-key.pem
create: dashboard.csr
Ignoring -days without -x509; not generating a certificate
create: dashboard.pem
Certificate request self-signature ok
subject=C = US, L = California, O = Company, CN = dashboard
* Filebeat cert
create: filebeat-key-temp.pem
create: filebeat-key.pem
create: filebeat.csr
Ignoring -days without -x509; not generating a certificate
create: filebeat.pem
Certificate request self-signature ok
subject=C = US, L = California, O = Company, CN = filebeat
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-kubernetes$ kubectl get svc -n wazuh
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
dashboard       LoadBalancer   10.98.203.141   <pending>     443:30521/TCP                    5m32s
indexer         LoadBalancer   10.111.57.161   <pending>     9200:31635/TCP                   5m32s
wazuh           LoadBalancer   10.110.36.37    <pending>     1515:31159/TCP,55000:30298/TCP   5m32s
wazuh-cluster   ClusterIP      None            <none>        1516/TCP                         5m32s
wazuh-workers   LoadBalancer   10.103.73.82    <pending>     1514:30860/TCP                   5m32s
wazuhindexer    ClusterIP      None            <none>        9300/TCP                         5m32s
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-kubernetes$ kubectl get pods -n wazuh
NAME                               READY   STATUS    RESTARTS   AGE
wazuh-dashboard-689b897f8b-f2qqv   1/1     Running   0          64s
wazuh-indexer-0                    1/1     Running   0          64s
wazuh-manager-master-0             1/1     Running   0          64s
wazuh-manager-worker-0             1/1     Running   0          64s

Screenshot_20240205_163617

root@wazuh-manager-master-0:/# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" --cacert "/etc/ssl/root-ca.pem" "https://wazuhindexer:9200"
curl: (60) SSL: certificate subject name '*.wazuhindexer' does not match target host name 'wazuhindexer'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@wazuh-manager-master-0:/# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" --cacert "/etc/ssl/root-ca.pem" "https://wazuh-indexer-0.wazuhindexer:9200"
curl: (6) Could not resolve host: wazuh-indexer-0.wazuhindexer
root@wazuh-manager-master-0:/# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" --cacert "/etc/ssl/root-ca.pem" "https://wazuh-indexer-0:9200"
curl: (6) Could not resolve host: wazuh-indexer-0
root@wazuh-manager-master-0:/# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" --cacert "/etc/ssl/root-ca.pem" "https://wazuh-indexer-0.wazuhindexer:9200"
curl: (6) Could not resolve host: wazuh-indexer-0.wazuhindexer
root@wazuh-manager-master-0:/#
vcerenu commented 8 months ago

Several tests were carried out with the certificates used by Wazuh indexer:

The DNS field was added to the certificate configuration with each of the cluster nodes to be raised (wazuh-indexer-0, wazuh-indexer-1 and wazuh-indexer-2). The name of the service was modified, having had problems with the "-" in the URL names, so that it takes another value. An investigation was made regarding the possible use of several CNs within one certificate, which was an unsatisfactory search.

For all cases, the CN of the certificates to be created was modified. In all cases the error was the same:

$ kubectl exec --stdin --tty pod/wazuh-manager-master-0 -n wazuh -- /bin/bash
root@wazuh-manager-master-0:/# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" --cacert " /etc/ssl/root-ca.pem" "https://wazuh-indexer-0.wazuh-indexer:9200"
curl: (60) SSL: certificate subject name '*.wazuh-indexer' does not match target host name 'wazuh-indexer-0.wazuh-indexer'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@wazuh-manager-master-0:/#
$ kubectl exec --stdin --tty pod/wazuh-manager-master-0 -n wazuh -- /bin/bash
root@wazuh-manager-master-0:/# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" --cacert " /etc/ssl/root-ca.pem" "https://wazuh-indexer-0.wazuhindexer:9200"
curl: (60) SSL: certificate subject name '*.wazuhindexer' does not match target host name 'wazuh-indexer-0.wazuhindexer'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@wazuh-manager-master-0:/#

According to the investigation of the error, for the wildcard certificate to be taken correctly, the domain must be more limited, at least it must have a domain name with an extension, something that is not possible to do with a Kubernetes service since they do not allow the use of . within their names.

There were 2 cases that gave satisfactory results in the test:

1) Use the fully qualified domain name of the first statefulset replica:

In this case, the full domain name of the first node of the statefulset was used because it is where all the services that want to access Wazuh indexer point, such as Wazuh dashboard and Wazuh manager, and this way the wildcard is not necessary. A deployment test was carried out with this change and it was satisfactory, since both the Wazuh indexer nodes were able to connect to each other, and the Wazuh dashboard, Wazuh manager and Filebeat were connected correctly.

$ wazuh/certs/indexer_cluster/generate_certs.sh
Root CA
Generating RSA private key, 2048 bit long modulus (2 primes)
.............................+++++
...................................+++++
e is 65537 (0x010001)
Admin cert
create: admin-key-temp.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
........+++++
..................................................+++++
e is 65537 (0x010001)
create: admin-key.pem
create: admin.csr
Ignoring -days; not generating a certificate
create: admin.pem
Signature ok
subject=C = US, L = California, O = Company, CN = admin
Getting CA Private Key
* Node cert
create: node-key-temp.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
...........................+++++
..................................................+++++
e is 65537 (0x010001)
create: node-key.pem
create: node.csr
Ignoring -days; not generating a certificate
create: node.pem
Signature ok
subject=C = US, L = California, O = Company, CN = wazuh-indexer-0.wazuh-indexer
Getting CA Private Key
* dashboard cert
create: dashboard-key-temp.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
..........................................................................+++++
................+++++
e is 65537 (0x010001)
create: dashboard-key.pem
create: dashboard.csr
Ignoring -days; not generating a certificate
create: dashboard.pem
Signature ok
subject=C = US, L = California, O = Company, CN = dashboard
Getting CA Private Key
* Filebeat cert
create: filebeat-key-temp.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
...+++++
...............................................................+++++
e is 65537 (0x010001)
create: filebeat-key.pem
create: filebeat.csr
Ignoring -days; not generating a certificate
create: filebeat.pem
Signature ok
subject=C = US, L = California, O = Company, CN = filebeat
Getting CA Private Key
$ kubectl exec --stdin --tty pod/wazuh-manager-master-0 -n wazuh -- /bin/bash
root@wazuh-manager-master-0:/# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" --cacert "/etc/ssl/root-ca.pem" "https://wazuh-indexer-0.wazuh-indexer:9200/_cat/nodes"
192.168.24.148 42 66 32 0.90 1.04 0.56 dimr cluster_manager,data,ingest,remote_cluster_client - wazuh-indexer-1
192.168.49.62  34 67 22 0.30 0.47 0.30 dimr cluster_manager,data,ingest,remote_cluster_client * wazuh-indexer-0
192.168.55.90  15 66 33 0.49 0.57 0.29 dimr cluster_manager,data,ingest,remote_cluster_client - wazuh-indexer-2
$ kubectl get all -n wazuh
NAME                                   READY   STATUS    RESTARTS   AGE
pod/wazuh-dashboard-7dc9c86b86-gkmp6   1/1     Running   0          47m
pod/wazuh-indexer-0                    1/1     Running   0          47m
pod/wazuh-indexer-1                    1/1     Running   0          4m10s
pod/wazuh-indexer-2                    1/1     Running   0          117s
pod/wazuh-manager-master-0             1/1     Running   0          4m7s
pod/wazuh-manager-worker-0             1/1     Running   0          4m1s
pod/wazuh-manager-worker-1             1/1     Running   0          47m

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP                                                                       PORT(S)                          AGE
service/dashboard       LoadBalancer   10.100.31.18     abdae32ccb3db4ce898db250924bca5d-409611784.us-west-1.elb.amazonaws.com            443:31669/TCP                    47m
service/indexer         LoadBalancer   10.100.155.170   a34d683f8a6a04895aa011109bb8f73a-752980879.us-west-1.elb.amazonaws.com            9200:30967/TCP                   47m
service/wazuh           LoadBalancer   10.100.32.162    ad1e00405599b4e329bf331bc07fc545-726319015.us-west-1.elb.amazonaws.com            1515:32096/TCP,55000:32029/TCP   47m
service/wazuh-cluster   ClusterIP      None             <none>                                                                            1516/TCP                         47m
service/wazuh-indexer   ClusterIP      None             <none>                                                                            9300/TCP                         47m
service/wazuh-workers   LoadBalancer   10.100.105.215   internal-af7f0fd2b3c1840979641c7a8735b361-357176006.us-west-1.elb.amazonaws.com   1514:32368/TCP                   47m

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/wazuh-dashboard   1/1     1            1           47m

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/wazuh-dashboard-7dc9c86b86   1         1         1       47m

NAME                                    READY   AGE
statefulset.apps/wazuh-indexer          3/3     47m
statefulset.apps/wazuh-manager-master   1/1     47m
statefulset.apps/wazuh-manager-worker   2/2     47m
$ kubectl exec --stdin --tty pod/wazuh-manager-master-0 -n wazuh -- /bin/bash
root@wazuh-manager-master-0:/# filebeat test output
elasticsearch: https://wazuh-indexer-0.wazuh-indexer:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.100.155.170
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2

image

2) Use the load balancer name:

In this case, the name Load Balancer was used so that when trying to connect to Wazuh indexer it tries against any of the available nodes. A test deployment was performed with this change and was successful, the Wazuh indexer nodes were able to connect to each other, and the Wazuh dashboard, Wazuh manager, and Filebeat connected successfully.

$ wazuh/certs/indexer_cluster/generate_certs.sh
Root CA
Generating RSA private key, 2048 bit long modulus (2 primes)
......................................................................................................+++++
.......................................................+++++
e is 65537 (0x010001)
Admin cert
create: admin-key-temp.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
...............................................+++++
....+++++
e is 65537 (0x010001)
create: admin-key.pem
create: admin.csr
Ignoring -days; not generating a certificate
create: admin.pem
Signature ok
subject=C = US, L = California, O = Company, CN = admin
Getting CA Private Key
* Node cert
create: node-key-temp.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
...........................+++++
........+++++
e is 65537 (0x010001)
create: node-key.pem
create: node.csr
Ignoring -days; not generating a certificate
create: node.pem
Signature ok
subject=C = US, L = California, O = Company, CN = indexer
Getting CA Private Key
* dashboard cert
create: dashboard-key-temp.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
.................................................................................................................+++++
.............+++++
e is 65537 (0x010001)
create: dashboard-key.pem
create: dashboard.csr
Ignoring -days; not generating a certificate
create: dashboard.pem
Signature ok
subject=C = US, L = California, O = Company, CN = dashboard
Getting CA Private Key
* Filebeat cert
create: filebeat-key-temp.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
...............+++++
................................+++++
e is 65537 (0x010001)
create: filebeat-key.pem
create: filebeat.csr
Ignoring -days; not generating a certificate
create: filebeat.pem
Signature ok
subject=C = US, L = California, O = Company, CN = filebeat
Getting CA Private Key
$ kubectl exec --stdin --tty pod/wazuh-manager-master-0 -n wazuh -- /bin/bash
root@wazuh-manager-master-0:/# curl -u "admin:SecretPassword" --cert "/etc/ssl/filebeat.pem" --key "/etc/ssl/filebeat.key" --cacert "/etc/ssl/root-ca.pem" "https://indexer:9200/_cat/nodes"
192.168.3.237  23 65 98 3.31 1.92 0.84 dimr cluster_manager,data,ingest,remote_cluster_client - wazuh-indexer-2
192.168.7.32   20 68 39 0.66 0.52 0.22 dimr cluster_manager,data,ingest,remote_cluster_client * wazuh-indexer-0
192.168.17.194 40 66 45 0.58 0.38 0.15 dimr cluster_manager,data,ingest,remote_cluster_client - wazuh-indexer-1
root@wazuh-manager-master-0:/#
$ kubectl get all -n wazuh
NAME                                   READY   STATUS    RESTARTS   AGE
pod/wazuh-dashboard-7dc9c86b86-lnvw6   1/1     Running   0          5m50s
pod/wazuh-indexer-0                    1/1     Running   0          5m50s
pod/wazuh-indexer-1                    1/1     Running   0          4m57s
pod/wazuh-indexer-2                    1/1     Running   0          3m55s
pod/wazuh-manager-master-0             1/1     Running   0          5m49s
pod/wazuh-manager-worker-0             1/1     Running   0          5m49s
pod/wazuh-manager-worker-1             1/1     Running   0          5m49s

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP                                                                        PORT(S)                          AGE
service/dashboard       LoadBalancer   10.100.204.150   aba9aa5d6c2ec49e3a3c2c1e1ec949de-488707172.us-west-1.elb.amazonaws.com             443:31443/TCP                    5m54s
service/indexer         LoadBalancer   10.100.63.162    a8284819012444309b9fdd1b634bbacf-1227941024.us-west-1.elb.amazonaws.com            9200:32199/TCP                   5m53s
service/wazuh           LoadBalancer   10.100.37.193    af45e658b3d5947c2b56b9bd26e89fe1-1807544138.us-west-1.elb.amazonaws.com            1515:30706/TCP,55000:32187/TCP   5m53s
service/wazuh-cluster   ClusterIP      None             <none>                                                                             1516/TCP                         5m52s
service/wazuh-indexer   ClusterIP      None             <none>                                                                             9300/TCP                         5m52s
service/wazuh-workers   LoadBalancer   10.100.98.157    internal-aa8efd1dd3bdc4d0a966887fd9f06c6d-1636301497.us-west-1.elb.amazonaws.com   1514:30370/TCP                   5m51s

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/wazuh-dashboard   1/1     1            1           5m51s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/wazuh-dashboard-7dc9c86b86   1         1         1       5m51s

NAME                                    READY   AGE
statefulset.apps/wazuh-indexer          3/3     5m51s
statefulset.apps/wazuh-manager-master   1/1     5m50s
statefulset.apps/wazuh-manager-worker   2/2     5m50s
root@wazuh-manager-master-0:/# filebeat test output
elasticsearch: https://indexer:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.100.63.162
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2
root@wazuh-manager-master-0:/# 

image

The second option is more convenient since all the services that communicate with it are added to the LB of Wazuh indexer, ceasing to depend heavily on the first replica of the statefulset, something that we did not have before.