GNMI Client Auth Failed with Server Certificate Authentication Error

[wumiaont]

Description

Telemetry tests are using gnmi client to connect to Sonic chassis and get data through gnmi. It's found that the connect would not be a success. Further analysis found that the server certificate and CA root cert on chassis has issues.

Findings: 1) server certificate and CA roots are all self signed certs. server cert is not signed by the CA.

admin@ixre-egl-board29:/etc/sonic/telemetry$ openssl x509 -in streamingtelemetryserver.cer -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 13:d6:3b:36:97:59:fd:11:5a:f1:f8:c0:fa:80:ef:8f:51:8c:12:e4 Signature Algorithm: sha256WithRSAEncryption Issuer: CN = ndastreamingservertest Validity Not Before: Jun 28 18:15:04 2024 GMT Not After : Jul 28 18:15:04 2024 GMT Subject: CN = ndastreamingservertest Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:98:47:4e:34:31:d6:c0:a3:0c:77:6f:e7:b7:f4: 46:0d:c2:9b:43:5d:21:a0:d5:b3:59:cd:41:5d:5e: 86:c7:a3:ec:ab:e5:f8:22:50:54:e1:b9:1e:33:66: 2b:a8:f6:9c:6b:a7:56:7e:3a:26:d2:10:57:ca:1d: 22:72:1a:cc:3d:13:04:80:ed:e2:59:08:06:80:68: 19:06:33:c9:e6:3f:c3:15:ce:f1:52:c1:51:f5:0a: ac:4d:83:65:09:b2:a4:71:35:1b:c2:91:8b:a9:c0: fd:ba:8a:95:70:df:f0:2e:8b:2e:88:07:27:ef:81: ed:b4:bb:71:d5:ff:c5:d3:7b:07:21:9f:c4:28:d5: 5e:d1:cf:70:8a:63:6c:c9:b1:89:9e:58:25:6b:1e: a8:b5:2a:1a:39:a0:bc:56:4c:49:e3:f9:32:fa:5d: 9d:93:d4:03:c1:ef:80:8a:5d:ba:a5:26:01:4b:62: 45:e5:9e:2b:6a:32:64:5f:04:99:14:e6:4b:f3:f5: 34:c9:54:fc:7a:a4:b8:48:00:ef:1c:62:aa:b4:0b: 19:b1:b9:04:38:7c:2a:81:c0:7a:c3:db:94:97:61: ca:d2:98:c8:db:00:3d:8b:1a:bd:87:f8:cd:ae:34: f5:53:6c:2d:70:ad:28:6a:f3:60:1b:18:3e:a4:9f: 39:47 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 7F:9B:25:C0:CC:DD:1F:A9:20:40:5A:9B:E7:C2:1C:0C:C0:5C:22:05 X509v3 Authority Key Identifier: 7F:9B:25:C0:CC:DD:1F:A9:20:40:5A:9B:E7:C2:1C:0C:C0:5C:22:05 X509v3 Basic Constraints: critical CA:TRUE Signature Algorithm: sha256WithRSAEncryption Signature Value: 3d:70:eb:c5:91:27:aa:8a:b2:d4:69:2c:2e:04:64:7c:5a:39: 5f:7b:9c:26:3d:e9:48:44:09:34:d7:00:00:87:7d:05:80:35: 18:2c:3e:55:45:f9:47:90:b4:a7:23:59:cc:2b:69:43:f7:a6: cf:4d:24:86:a6:25:83:cb:e4:38:a5:ad:48:e9:7e:9e:f7:6f: bd:9f:33:22:70:1e:e7:f3:cc:ee:5f:4e:cc:a1:9b:b0:d0:bb: 2d:8d:3e:37:61:71:fe:22:a9:e3:96:1e:c3:c3:9f:8f:ee:d1: 16:f1:b3:b9:01:d0:43:74:28:b6:eb:3c:91:29:31:3a:f9:f5: 01:88:ad:46:c9:b3:3a:fc:8f:f9:84:12:79:66:1c:28:6b:f7: 53:f9:23:f9:4d:c2:50:12:d6:75:1b:59:12:62:51:a1:39:ed: 90:7d:25:50:67:d6:a6:86:dc:8f:9e:98:41:c6:a0:9f:d9:95: a9:c7:cf:0d:08:35:24:39:55:18:b3:b2:18:c2:c7:95:f2:4f: 60:39:96:83:53:49:f5:a0:fd:2a:2a:6f:de:ae:2b:99:b5:a9: 69:30:f7:33:9c:57:4e:90:a6:16:9d:8d:6c:35:ba:fa:d2:4f: f4:cc:9e:96:71:31:d3:76:09:1b:40:d9:dc:c6:ea:72:84:52: b3:5e:f3:c4 admin@ixre-egl-board29:/etc/sonic/telemetry$ ls -al total 28 drwxr-xr-x 2 root root 4096 Jun 6 16:31 . drwxr-xr-x 1 root root 4096 Jul 13 01:57 .. -rw-r--r-- 1 root root 1147 Jun 28 18:15 dsmsroot.cer -rw------- 1 root root 1704 Jun 28 18:15 dsmsroot.key -rw-r--r-- 1 root root 1147 Jun 28 18:15 streamingtelemetryserver.cer -rw------- 1 root root 1704 Jun 28 18:15 streamingtelemetryserver.key admin@ixre-egl-board29:/etc/sonic/telemetry$ openssl x509 -in dsmsroot.cer -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 25:da:dc:d2:3a:9c:3a:47:9f:fa:e7:c8:aa:72:fb:6a:09:a1:d6:56 Signature Algorithm: sha256WithRSAEncryption Issuer: CN = ndastreamingclienttest Validity Not Before: Jun 28 18:15:05 2024 GMT Not After : Jul 28 18:15:05 2024 GMT Subject: CN = ndastreamingclienttest Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:ee:90:08:41:67:15:1f:28:03:6b:8d:be:f6:4d: ce:e3:f7:f1:07:35:30:4b:63:6d:e5:34:30:2f:99: cb:96:86:26:5b:01:6d:9e:64:19:56:3a:7d:a1:37: 28:be:0a:5c:90:ee:58:cf:1b:5c:28:c2:8f:1f:26: 9e:38:25:a3:68:e9:d0:5a:88:3b:a8:e4:4a:4c:3f: c9:09:16:5e:8a:61:70:06:a5:f9:34:93:d3:de:df: b0:b1:1d:77:a9:37:36:ed:79:03:6e:cc:3d:34:01: e4:0d:05:d5:23:c7:01:d2:3c:4b:7c:c5:df:6a:00: 71:08:3b:6e:3c:47:aa:e6:6c:27:ff:45:01:9b:07: 5d:34:78:79:2e:eb:27:92:9c:b5:61:b5:5b:27:71: 14:fe:86:c2:ba:7b:1e:4e:e9:e0:0c:68:ca:15:c8: 6b:f8:9e:2a:33:d2:2a:93:32:1c:91:fc:85:63:a0: 85:b4:36:0d:3b:ee:5d:cd:6f:23:de:c2:2e:13:64: 96:cb:fe:ef:d0:bb:c5:81:ce:5f:0a:f3:55:63:5d: 06:79:3b:0e:e6:20:f8:0d:88:be:69:f9:6d:fc:9c: a5:cd:0d:51:ab:14:01:2a:77:c9:91:af:af:46:86: b7:eb:a7:cd:97:04:dd:6b:c1:33:a8:3c:94:9b:17: 96:25 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 82:51:16:1C:C9:CF:8A:11:78:80:90:DD:AD:BA:11:25:6C:01:C9:E9 X509v3 Authority Key Identifier: 82:51:16:1C:C9:CF:8A:11:78:80:90:DD:AD:BA:11:25:6C:01:C9:E9 X509v3 Basic Constraints: critical CA:TRUE Signature Algorithm: sha256WithRSAEncryption Signature Value: e8:a2:6a:44:6f:ed:15:81:df:15:68:45:fc:50:7e:c0:9c:72: b8:00:46:78:22:4d:59:88:f2:93:4b:92:8c:fa:94:06:6a:2d: 1e:26:63:d5:b1:28:a9:30:87:81:9b:57:f4:71:16:27:5e:49: 73:85:54:32:c2:78:47:8a:e6:10:b0:28:f2:3e:b8:c8:c2:0e: 89:d3:3e:c7:3c:ad:b3:6c:a1:29:9d:71:12:b4:9f:ab:c3:91: 36:9b:0f:0b:be:bc:4f:0a:57:a0:32:23:87:cf:78:73:0d:3b: 64:df:cf:1a:a3:2b:57:88:24:d1:33:a9:84:0c:99:ea:a1:87: 4d:b9:56:91:3f:06:4d:2e:33:9a:de:af:f1:1b:c9:70:3b:8b: 59:e6:0d:a8:bc:ef:11:2e:de:fc:40:28:fd:51:85:a1:a1:47: aa:b7:fb:80:35:0e:c6:80:44:b4:10:05:a8:cd:3d:04:30:dd: 4c:79:96:b8:95:22:59:a6:31:57:21:6b:0d:05:ff:01:a3:03: e9:5e:55:c1:eb:c0:d6:26:db:bf:dc:d8:78:17:c6:49:0e:05: 51:33:c8:71:a3:8e:3e:ca:0d:1e:e1:0e:03:df:5c:2e:57:c6: ae:ec:bb:f5:49:f1:d4:2f:1b:88:3d:26:06:d4:69:00:02:d7: ea:45:0f:35

2) gnmi client runns from ptfserver will not succeed. It will work if pass --notls with the command.
root@6b065efa96a8:~# python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.06.239 -p 8080 -m get -x proc/uptime -xt OTHERS -o "ndastreamingservertest" Traceback (most recent call last): File "/root/gnxi/gnmi_cli_py/py_gnmicli.py", line 643, in main() File "/root/gnxi/gnmi_cli_py/py_gnmicli.py", line 568, in main creds = _build_creds(target, port, get_cert, certs, notls) File "/root/gnxi/gnmi_cli_py/py_gnmicli.py", line 413, in _build_creds rcert = ssl.get_server_certificate((target, port)).encode('utf-8') File "/usr/lib/python2.7/ssl.py", line 1005, in get_server_certificate with closing(context.wrap_socket(sock)) as sslsock: File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket _context=self) File "/usr/lib/python2.7/ssl.py", line 599, in init self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:727)

Steps to reproduce the issue:

1. On ptf server, run the following command: python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.06.239 -p 8080 -m get -x proc/uptime -xt OTHERS -o "ndastreamingservertest". It will fail.
2. Run python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.06.239 -p 8080 -m get -x proc/uptime -xt OTHERS -o "ndastreamingservertest" -n will have data back. root@6b065efa96a8:~# python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.06.239 -p 8080 -m get -x proc/uptime -xt OTHERS -o "ndastreamingservertest" -n Performing GetRequest, encoding=JSON_IETF to 10.250.06.239 with the following gNMI Path [elem { name: "proc" } elem { name: "uptime" } ] The GetResponse is below

{ "idle": 47597.99, "total": 3152.27 }

Output of show version:

master and 202405 branch

[vmittal-msft]

@wumiaont please post more data after verification with 202205 code keeping everything else same. @qiluo-msft can further help if needed.

[liuh-80]

ACK, will check and update here.

[wumiaont]

This is what I am running 2205 and get.

By default TELEMETRY|gnmi client_auth is true and this is telemetry service running inside telemetry: usr/sbin/telemetry -logtostderr --server_crt /etc/sonic/telemetry/streamingtelemetryserver.cer --server_key /etc/sonic/telemetry/streamingtelemetryserver.key --ca_crt /etc/sonic/telemetry/dsmsroot.cer --port 50051 -v=2

When I ran gnmi client from ptf server. It fails with these: python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.6.231 -p 50051 -m get -x proc/uptime -xt OTHERS -o "ndastreamingservertest" Performing GetRequest, encoding=JSON_IETF to 10.250.6.231 with the following gNMI Path [elem { name: "proc" } elem { name: "uptime" } ] E0719 16:43:35.826124547 25817 ssl_transport_security.cc:523] Corruption detected. E0719 16:43:35.826138403 25817 ssl_transport_security.cc:499] error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE E0719 16:43:35.826142152 25817 secure_endpoint.cc:205] Decryption error: TSI_DATA_CORRUPTED Client receives an exception 'failed to connect to all addresses' indicating gNMI server is shut down and Exiting ...

The client authentication failed with this.

[wumiaont]

Second test still with 2205. Modify CONFIG_DB to make client_auth to be false. sonic-db-cli CONFIG_DB HSET "TELEMETRY|gnmi" "client_auth" "false" . Restart telemetry. Run gnmi client from PTF server. It works. This allows no client authentication of server cert and it works.

root@6b065efa96a8:/# python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.6.231 -p 50051 -m get -x proc/uptime -xt OTHERS -o "ndastreamingservertest" Performing GetRequest, encoding=JSON_IETF to 10.250.6.231 with the following gNMI Path [elem { name: "proc" } elem { name: "uptime" } ] The GetResponse is below { "idle": 40348.99, "total": 6154.85 }

[wumiaont]

I have similar result for 202405 with above result. The one works is because of not doing client authentication. If servers cert needs client authentication it won't work.

It could be a design as server cert is a self signed cert here.

[wumiaont]

@qiluo-msft

[wumiaont]

Looks to me with the fresh image (no config in CONFIG_DB for GNMI), we could not make gnmi client work. Inside gnmi container this is the telemetry process: /usr/sbin/telemetry -logtostderr --noTLS --port 8080 --allow_no_client_auth -v=2 --threshold 100 --idle_conn_duration 5. It seems will not support tls at this moment. --notls will work.

If we need to make gnmi work, we have to config the certs location under GNMI|certs and set GNMI|gnmi client_auth to be false and restart gnmi service. Then the gnmi client access will work since it bypass client auth. Which makes sense as server cert is a self signed cert.

I can close this issue if this is by design. I assume in a real product line we should generate real server cert signed by a trusted CA.

[zbud-msft]

This is by design