search

openconfig / gnmic

gNMIc is a gNMI CLI client and collector
https://gnmic.openconfig.net
Apache License 2.0
170 stars 55 forks source link

gnmic 0.35.1 - failed to create a gRPC client for target XXX: context deadline exceeded #367

Closed tiago-amado closed 6 months ago

tiago-amado commented 7 months ago

Description

  1. gnmictls setup with 0.34.3 working fine for SROS
  2. After upgrade to 0.35.1 its returning an error:
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
target "172.20.20.4:57400", capabilities request failed: failed to create a gRPC client for target "172.20.20.4:57400" : 172.20.20.4:57400: context deadline exceeded
Error: one or more requests failed
  1. Downgrade to the previous release solves the error.

Outputs

    • Ok 
      
[root@AF6 CLAB_CA]# gnmic version
version : 0.34.3
commit : d461fa9
date : 2023-11-14T02:13:11Z
gitURL : https://github.com/openconfig/gnmic
docs : https://gnmic.openconfig.net/
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4 -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
    • nokia-conf, Nokia, 23.10.R2
    • nokia-state, Nokia, 23.10.R2
    • nokia-li-state, Nokia, 23.10.R2 supported encodings:
    • JSON
    • BYTES
    • PROTO
    • JSON_IETF

[root@AF6 CLAB_CA]#


2. Upgrade NOK
```bash
[root@AF6 ~]#
[root@AF6 ~]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)"
gnmic 0.35.1 is available. Changing from version 0.34.3.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.35.1/gnmic_0.35.1_linux_x86_64.tar.gz
Preparing to install gnmic 0.35.1 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.35.1
commit : 91f4205
   date : 2024-01-25T21:10:47Z
gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net/
[root@AF6 ~]# 
[root@AF6 ~]# 
[root@AF6 ~]# gnmic version
version : 0.35.1
commit : 91f4205
   date : 2024-01-25T21:10:47Z
gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net/
[root@AF6 ~]# 
[root@AF6 ~]# 
[root@AF6 ~]# cd /home/tiago/clab_projects/CLAB_CA/
[root@AF6 CLAB_CA]# ls -al
total 20
drwxr-xr-x  2 root root   66 Jan 31 12:31 .
drwxrwxrwx 15 root root 4096 Jan 31 12:14 ..
-rw-r--r--  1 root root 1675 Jan 31 12:15 ca.key
-rw-r--r--  1 root root 1367 Jan 31 12:15 ca.pem
-rw-r--r--  1 root root 1675 Jan 31 12:31 cert.key
-rw-r--r--  1 root root 1367 Jan 31 12:31 cert.pem
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
target "172.20.20.4:57400", capabilities request failed: failed to create a gRPC client for target "172.20.20.4:57400" : 172.20.20.4:57400: context deadline exceeded
Error: one or more requests failed
[root@AF6 CLAB_CA]#
  1. Downgrade Ok 
    
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)" -- -v 0.34.3
gnmic 0.34.3 is available. Changing from version 0.35.1.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.34.3/gnmic_0.34.3_linux_x86_64.tar.gz
Preparing to install gnmic 0.34.3 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.34.3
commit : d461fa9
date : 2023-11-14T02:13:11Z
gitURL : https://github.com/openconfig/gnmic
docs : https://gnmic.openconfig.net/
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# pwd
/home/tiago/clab_projects/CLAB_CA
[root@AF6 CLAB_CA]# ls -al
total 20
drwxr-xr-x  2 root root   66 Jan 31 12:31 .
drwxrwxrwx 15 root root 4096 Jan 31 12:14 ..
-rw-r--r--  1 root root 1675 Jan 31 12:15 ca.key
-rw-r--r--  1 root root 1367 Jan 31 12:15 ca.pem
-rw-r--r--  1 root root 1675 Jan 31 12:31 cert.key
-rw-r--r--  1 root root 1367 Jan 31 12:31 cert.pem
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
- nokia-conf, Nokia, 23.10.R2
- nokia-state, Nokia, 23.10.R2
- nokia-li-state, Nokia, 23.10.R2
supported encodings:
- JSON
- BYTES
- PROTO
- JSON_IETF

[root@AF6 CLAB_CA]#

karimra commented 7 months ago

Can you add the --debug flag and share the log output in the failed case ?

tiago-amado commented 7 months ago

debug.txt

karimra commented 7 months ago

This is due to a change in go-grpc v1.60 https://github.com/grpc/grpc-go/pull/6776 that enforces this section of RFC7540. go-grpc started filtering out forbidden cipher suites set in the TLS profile. The remaining ones don't overlap with the ones supported by SROS => auth handshake fails.

I will set the cipher suites manually so that go-grpc doesn't filter them out. Thanks for reporting this.

karimra commented 7 months ago

@tiago-amado v0.35.2 should advertise the same cipher suites as before go-grpc v1.60, please give it a try and let me know

tiago-amado commented 7 months ago

Hi @karimra , I've tested and its still failing.

OK with v0.34.3

[root@AF6 CLAB_CA]# gnmic version
version : 0.34.3
 commit : d461fa9
   date : 2023-11-14T02:13:11Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 

[root@AF6 CLAB_CA]# gnmic -a clab-cert01-sr  -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
  - nokia-conf, Nokia, 23.10.R2
  - nokia-state, Nokia, 23.10.R2
  - nokia-li-state, Nokia, 23.10.R2
supported encodings:
  - JSON
  - BYTES
  - PROTO
  - JSON_IETF

[root@AF6 CLAB_CA]#

NOK after upgrade to v0.35.2

[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)" -- -v 0.35.2
gnmic 0.35.2 is available. Changing from version 0.34.3.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.35.2/gnmic_0.35.2_linux_x86_64.tar.gz
Preparing to install gnmic 0.35.2 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.35.2
 commit : bd70a53
   date : 2024-02-06T00:46:16Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 

[root@AF6 CLAB_CA]# gnmic -a clab-cert01-sr  -u admin -p admin --tls-ca ca.pem capabilities
target "clab-cert01-sr", capabilities request failed: failed to create a gRPC client for target "clab-cert01-sr" : clab-cert01-sr:57400: context deadline exceeded
Error: one or more requests failed
[root@AF6 CLAB_CA]#

OK after downgrade to v0.33.0

[root@AF6 ~]# cd /home/tiago/clab_projects/CLAB_CA/
[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)" -- -v 0.33.0
gnmic 0.33.0 is available. Changing from version 0.35.2.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.33.0/gnmic_0.33.0_linux_x86_64.tar.gz
Preparing to install gnmic 0.33.0 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.33.0
 commit : d8f931e
   date : 2023-10-08T17:08:01Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a clab-cert01-sr  -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
  - nokia-conf, Nokia, 23.10.R2
  - nokia-state, Nokia, 23.10.R2
  - nokia-li-state, Nokia, 23.10.R2
supported encodings:
  - JSON
  - BYTES
  - PROTO
  - JSON_IETF

[root@AF6 CLAB_CA]#
tiago-amado commented 7 months ago

gnmic_v0.35.2_debug.txt

karimra commented 7 months ago

Can you share which cipher suites are configured on your SROS?

tiago-amado commented 7 months ago

This one: tls-rsa-with3des-ede-cbc-sha Attached the SROS config: gnmic_v0.35.2_SR-config.txt

tiago-amado commented 6 months ago

Hi @karimra. I've tested with the latest 0.36.2 and its working now! many thanks!

OK with v0.36.2

after upgrade its working

[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)"  
gnmic 0.36.2 is available. Changing from version 0.35.2.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.36.2/gnmic_0.36.2_linux_x86_64.tar.gz
Preparing to install gnmic 0.36.2 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.36.2
 commit : a7844a6d
   date : 2024-03-05T20:10:26Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
  - nokia-conf, Nokia, 23.10.R2
  - nokia-state, Nokia, 23.10.R2
  - nokia-li-state, Nokia, 23.10.R2
supported encodings:
  - JSON
  - BYTES
  - PROTO
  - JSON_IETF

[root@AF6 CLAB_CA]#

NOK with v0.35.2

I've downgrade to v0.35.2 to retest and it is still failing. I've added more ciphers on SROS and did not solve.

[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)"  -- -v 0.35.2
gnmic 0.35.2 is available. Changing from version 0.36.2.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.35.2/gnmic_0.35.2_linux_x86_64.tar.gz
Preparing to install gnmic 0.35.2 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.35.2
 commit : bd70a53
   date : 2024-02-06T00:46:16Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
target "172.20.20.4:57400", capabilities request failed: failed to create a gRPC client for target "172.20.20.4:57400" : 172.20.20.4:57400: context deadline exceeded
Error: one or more requests failed
[root@AF6 CLAB_CA]#
karimra commented 6 months ago

Thanks, I think I messed up some imports in v0.35.2.