search

CiscoDevNet / yangsuite

Cisco YANG Suite provides a set of tools and plugins to learn, test, and adopt YANG programmable interfaces such as NETCONF, RESTCONF, gNMI and more.
Other
244 stars 75 forks source link

gRPC Telemetry: ERROR [django.request:222] Start servicer: Failed to bind to address #89

Closed Chenhu-source closed 1 year ago

Chenhu-source commented 1 year ago

Hi Team，

when I click the "start receiver", the server monitor always reminds us in a failure. Did you meet the issue?

Screenshot 2023-03-09 at 16 04 24

docker-yangsuite-1 | E0308 03:42:44.470673018 66 server_chttp2.cc:48] {"created":"@1678246964.470609640","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":935,"referenced_errors":[{"created":"@1678246964.470601669","description":"Unable to configure socket","fd":9,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":217,"referenced_errors":[{"created":"@1678246964.470578209","description":"Cannot assign requested address","errno":99,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":191,"os_error":"Cannot assign requested address","syscall":"bind"}]}]} docker-yangsuite-1 | [08/Mar/2023 03:42:44] ERROR [django.request:222] Start servicer: Failed to bind to address 10.106.116.197:60001; set GRPC_VERBOSITY=debug environment variable to see detailed error message.: /grpctelemetry/servicer/60001/start docker-yangsuite-1 | [pid: 66|app: 0|req: 10/18] 72.163.224.64 () {60 vars in 1307 bytes} [Wed Mar 8 03:42:44 2023] POST /grpctelemetry/servicer/60001/start => generated 2 bytes in 4 msecs (HTTP/2.0 500) 4 headers in 251 bytes (2 switches on core 0) docker-nginx-1 | 72.163.224.64 - - [08/Mar/2023:03:42:44 +0000] "POST /grpctelemetry/servicer/60001/start HTTP/2.0" 500 2 "https://10.106.116.197:8443/grpctelemetry/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/110.0"

miott commented 1 year ago

"Start servicer: Failed to bind to address 10.106.116.197:60001;".

You were unable to establish a TCP connection to 10.106.116.197:60001. Verify the device telemetry configuration is correct. That configuration must have the IP address of the yangsuite server and port 60001.

Yangsuite is a listener in this case and once the TCP connection is established, when the device has data, it is supposed to stream it to yangsuite.

Chenhu-source commented 1 year ago

"Start servicer: Failed to bind to address 10.106.116.197:60001;".

You were unable to establish a TCP connection to 10.106.116.197:60001. Verify the device telemetry configuration is correct. That configuration must have the IP address of the yangsuite server and port 60001.

Yangsuite is a listener in this case and once the TCP connection is established, when the device has data, it is supposed to stream it to yangsuite.

I checked the communication. port 60001 couldn't be reachable due to port 60001 didn't exist on the Yangsuite server.

2023-03-09 07:12:17.594115 10.124.39.37 -> 10.106.116.197 TCP 21268 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175223819 TSER=0 WS=10 2023-03-09 07:12:18.596811 10.124.39.37 -> 10.106.116.197 TCP 21268 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175224120 TSER=0 WS=10 2023-03-09 07:12:19.596352 10.124.39.37 -> 10.106.116.197 TCP 21269 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175224419 TSER=0 WS=10 2023-03-09 07:12:20.596830 10.124.39.37 -> 10.106.116.197 TCP 21269 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175224720 TSER=0 WS=10 2023-03-09 07:12:21.596869 10.124.39.37 -> 10.106.116.197 TCP 21270 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175225020 TSER=0 WS=10 2023-03-09 07:12:22.600162 10.124.39.37 -> 10.106.116.197 TCP 21270 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175225321 TSER=0 WS=10 2023-03-09 07:12:23.597946 10.124.39.37 -> 10.106.116.197 TCP 21271 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175225620 TSER=0 WS=10 2023-03-09 07:12:24.600139 10.124.39.37 -> 10.106.116.197 TCP 21271 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175225921 TSER=0 WS=10 2023-03-09 07:12:27.661030 10.124.39.37 -> 10.106.116.197 TCP 21272 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175226839 TSER=0 WS=10 2023-03-09 07:12:28.663478 10.124.39.37 -> 10.106.116.197 TCP 21272 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175227140 TSER=0 WS=10 2023-03-09 07:12:29.662029 10.124.39.37 -> 10.106.116.197 TCP 21273 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175227439 TSER=0 WS=10 2023-03-09 07:12:30.663494 10.124.39.37 -> 10.106.116.197 TCP 21273 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175227740 TSER=0 WS=10 2023-03-09 07:12:31.664254 10.124.39.37 -> 10.106.116.197 TCP 21274 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=175228040 TSER=0 WS=10

[test@server ~]$ netstat -ntlp (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN - tcp 0 0 0.0.0.0:8443 0.0.0.0: LISTEN - tcp 0 0 0.0.0.0:443 0.0.0.0: LISTEN - tcp 0 0 0.0.0.0:9339 0.0.0.0: LISTEN - tcp 0 0 0.0.0.0:57344 0.0.0.0: LISTEN - tcp 0 0 0.0.0.0:57345 0.0.0.0: LISTEN - tcp 0 0 0.0.0.0:50051 0.0.0.0: LISTEN - tcp 0 0 0.0.0.0:50052 0.0.0.0: LISTEN - tcp 0 0 0.0.0.0:80 0.0.0.0: LISTEN - tcp6 0 0 :::22 ::: LISTEN - tcp6 0 0 :::8443 ::: LISTEN - tcp6 0 0 :::443 ::: LISTEN - tcp6 0 0 :::9339 ::: LISTEN - tcp6 0 0 :::57344 ::: LISTEN - tcp6 0 0 :::57345 ::: LISTEN - tcp6 0 0 :::50051 ::: LISTEN - tcp6 0 0 :::50052 ::: LISTEN - tcp6 0 0 :::80 ::: LISTEN - [test@server ~]$

[test@server ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e3180480a2a1 nginx:latest "/docker-entrypoint.…" 2 days ago Up 28 hours 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:8443->8443/tcp, :::8443->8443/tcp docker-nginx-1 f7bcf68d549a backup:latest "/code/runcron.sh" 2 days ago Up 28 hours docker-backup-1 f71a0f505e28 yangsuite:latest "/yangsuite/migrate…" 2 days ago Up 28 hours 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:9339->9339/tcp, :::9339->9339/tcp, 0.0.0.0:50051-50052->50051-50052/tcp, :::50051-50052->50051-50052/tcp, 0.0.0.0:57344-57345->57344-57345/tcp, :::57344-57345->57344-57345/tcp docker-yangsuite-1 [test@server ~]$

miott commented 1 year ago

I see. If you look at the docker-compose.yml file you will see some common port mapping for gRPC.

        # gRPC MDT telemetry insecure port (IOS-XE)
        - "57344:57344"
        # gRPC MDT telemetry secure port (IOS-XE)
        - "57345:57345"

If you change the device config telemetry port to one of those ports and set the yangsuite server to listen on 57344 or 57345 the traffic should get through to the container.

To make it more dynamic without editing the docker-compose.yml every time, you could use environment settings. Add a couple ports to docker-compose.yml to this:

        # gRPC dynamic telemetry port
        - ${GRPC_PORT1}:60001
        # gRPC dynamic telemetry secure port
        - ${GRPC_PORT2}:60002

From your terminal stop the containers and do this:

export GRPC_PORT1=60001
export GRPC_PORT2=60002
docker-compose up

Set your yangsuite grpc server to listen on 60001 for clear channel and/or 60002 for TLS secure channel. Set as many ports as you want. Let us know if this works for you.

Chenhu-source commented 1 year ago
  1. In this way, I could set up ports 60001 and 60002 on the server successfully.

[test@server docker]# netstat -ntlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0: LISTEN 3385/docker-proxy
tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN 986/sshd
tcp 0 0 0.0.0.0:8443 0.0.0.0: LISTEN 3362/docker-proxy
tcp 0 0 0.0.0.0:443 0.0.0.0: LISTEN 3203/docker-proxy
tcp 0 0 0.0.0.0:9339 0.0.0.0: LISTEN 3180/docker-proxy
tcp 0 0 0.0.0.0:57344 0.0.0.0: LISTEN 3116/docker-proxy
tcp 0 0 0.0.0.0:57345 0.0.0.0: LISTEN 3094/docker-proxy
tcp 0 0 0.0.0.0:60001 0.0.0.0: LISTEN 3072/docker-proxy
tcp 0 0 0.0.0.0:60002 0.0.0.0: LISTEN 3049/docker-proxy
tcp 0 0 0.0.0.0:50051 0.0.0.0: LISTEN 3159/docker-proxy
tcp 0 0 0.0.0.0:50052 0.0.0.0: LISTEN 3137/docker-proxy
tcp6 0 0 :::80 ::: LISTEN 3392/docker-proxy
tcp6 0 0 :::22 ::: LISTEN 986/sshd
tcp6 0 0 :::8443 ::: LISTEN 3371/docker-proxy
tcp6 0 0 :::443 ::: LISTEN 3211/docker-proxy
tcp6 0 0 :::9339 ::: LISTEN 3187/docker-proxy
tcp6 0 0 :::57344 ::: LISTEN 3123/docker-proxy
tcp6 0 0 :::57345 ::: LISTEN 3101/docker-proxy
tcp6 0 0 :::60001 ::: LISTEN 3079/docker-proxy
tcp6 0 0 :::60002 ::: LISTEN 3058/docker-proxy
tcp6 0 0 :::50051 ::: LISTEN 3166/docker-proxy
tcp6 0 0 :::50052 ::: LISTEN 3145/docker-proxy
[test@server docker]#

[test@server docker]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 37a60d3358f6 alpine "/bin/sh" 18 minutes ago Up 18 minutes angry_napier ab5aa3601503 nginx:latest "/docker-entrypoint.…" 58 minutes ago Up 58 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:8443->8443/tcp, :::8443->8443/tcp docker-nginx-1 fb3c48aeca60 backup:latest "/code/runcron.sh" 58 minutes ago Up 58 minutes docker-backup-1 d019510faf98 yangsuite:latest "/yangsuite/migrate…" 58 minutes ago Up 58 minutes 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:9339->9339/tcp, :::9339->9339/tcp, 0.0.0.0:50051-50052->50051-50052/tcp, :::50051-50052->50051-50052/tcp, 0.0.0.0:57344-57345->57344-57345/tcp, :::57344-57345->57344-57345/tcp, 0.0.0.0:60001-60002->60001-60002/tcp, :::60001-60002->60001-60002/tcp docker-yangsuite-1 [test@server docker]#

  1. But the ERROR is still there, 60001 couldn't be reachable. even I log in to the server and test port 60001, it show us "Connection refused"
Screenshot 2023-03-10 at 14 14 39

docker-yangsuite-1 | E0310 06:13:54.173235735 63 server_chttp2.cc:48] {"created":"@1678428834.173199841","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":935,"referenced_errors":[{"created":"@1678428834.173176872","description":"Unable to configure socket","fd":10,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":217,"referenced_errors":[{"created":"@1678428834.173162662","description":"Cannot assign requested address","errno":99,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":191,"os_error":"Cannot assign requested address","syscall":"bind"}]}]} docker-yangsuite-1 | [10/Mar/2023 06:13:54] ERROR [django.request:222] Start servicer: Failed to bind to address 10.106.116.197:60001; set GRPC_VERBOSITY=debug environment variable to see detailed error message.: /grpctelemetry/servicer/60001/start docker-nginx-1 | 72.163.224.64 - - [10/Mar/2023:06:13:54 +0000] "POST /grpctelemetry/servicer/60001/start HTTP/2.0" 500 2 "https://10.106.116.197:8443/grpctelemetry/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/110.0" docker-yangsuite-1 | [pid: 63|app: 0|req: 7/35] 72.163.224.64 () {60 vars in 1307 bytes} [Fri Mar 10 06:13:54 2023] POST /grpctelemetry/servicer/60001/start => generated 2 bytes in 4 msecs (HTTP/2.0 500) 4 headers in 251 bytes (1 switches on core 0)

[test@server cisco]# ssh -v -p 60001 cisco@10.106.116.197 OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: configuration requests final Match pass debug1: re-parsing configuration debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: Connecting to 10.106.116.197 [10.106.116.197] port 60001. debug1: connect to address 10.106.116.197 port 60001: Connection refused ssh: connect to host 10.106.116.197 port 60001: Connection refused <<<<<<<<<<<<<<<<<<<<<<<< [test@server cisco]# [test@server cisco]#

[test@server docker]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:50:56:bd:28:b4 brd ff:ff:ff:ff:ff:ff altname enp2s0 inet 10.106.116.197/27 brd 10.106.116.223 scope global noprefixroute ens32 valid_lft forever preferred_lft forever inet6 fe80::250:56ff:febd:28b4/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: br-4ec5f82462ea: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:6d:46:f5:4e brd ff:ff:ff:ff:ff:ff inet 172.18.0.1/16 brd 172.18.255.255 scope global br-4ec5f82462ea valid_lft forever preferred_lft forever inet6 fe80::42:6dff:fe46:f54e/64 scope link valid_lft forever preferred_lft forever 4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:1e:57:bc:06 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:1eff:fe57:bc06/64 scope link valid_lft forever preferred_lft forever 12: vethfce51f8@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4ec5f82462ea state UP group default link/ether 7e:60:48:90:25:c8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::7c60:48ff:fe90:25c8/64 scope link valid_lft forever preferred_lft forever 14: vethbd73de3@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4ec5f82462ea state UP group default link/ether 82:dc:66:92:5e:51 brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet6 fe80::80dc:66ff:fe92:5e51/64 scope link valid_lft forever preferred_lft forever 16: veth21a1a68@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4ec5f82462ea state UP group default link/ether 82:84:71:77:a9:06 brd ff:ff:ff:ff:ff:ff link-netnsid 2 inet6 fe80::8084:71ff:fe77:a906/64 scope link valid_lft forever preferred_lft forever 18: veth5528d5d@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default link/ether 32:11:32:52:e6:fe brd ff:ff:ff:ff:ff:ff link-netnsid 3 inet6 fe80::3011:32ff:fe52:e6fe/64 scope link valid_lft forever preferred_lft forever [test@server docker]#

2023-03-10 03:07:35.390212 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42778 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:07:44.037146 10.124.39.37 -> 10.106.116.197 TCP 42779 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196741754 TSER=0 WS=10 2023-03-10 03:07:44.166537 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42779 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:07:44.853887 10.124.39.37 -> 10.106.116.197 TCP 42780 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196741999 TSER=0 WS=10 2023-03-10 03:07:44.983354 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42780 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:07:45.279701 10.124.39.37 -> 10.106.116.197 TCP 42781 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196742127 TSER=0 WS=10 2023-03-10 03:07:45.409158 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42781 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:07:54.040342 10.124.39.37 -> 10.106.116.197 TCP 42782 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196744755 TSER=0 WS=10 2023-03-10 03:07:54.169954 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42782 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:07:54.853825 10.124.39.37 -> 10.106.116.197 TCP 42783 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196744999 TSER=0 WS=10 2023-03-10 03:07:54.983324 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42783 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:07:55.228298 10.124.39.37 -> 10.106.116.197 TCP 42784 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196745111 TSER=0 WS=10 2023-03-10 03:07:55.357823 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42784 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:08:04.031929 10.124.39.37 -> 10.106.116.197 TCP 42785 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196747752 TSER=0 WS=10 2023-03-10 03:08:04.161395 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42785 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:08:04.855486 10.124.39.37 -> 10.106.116.197 TCP 42786 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196747999 TSER=0 WS=10 2023-03-10 03:08:04.984779 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42786 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:08:05.268016 10.124.39.37 -> 10.106.116.197 TCP 42787 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196748123 TSER=0 WS=10 2023-03-10 03:08:05.397304 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42787 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 2023-03-10 03:08:14.026293 10.124.39.37 -> 10.106.116.197 TCP 42788 > 60001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 TSV=196750750 TSER=0 WS=10 2023-03-10 03:08:14.155728 10.106.116.197 -> 10.124.39.37 TCP 60001 > 42788 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

miott commented 1 year ago

Lots of great information. Bottom line:

10.124.39.37 -> 10.106.116.197 TCP 42779 > 60001 [SYN] - Cisco router.
10.106.116.197 -> 10.124.39.37 TCP 60001 > 42782 [RST, ACK] - network card from device yangsuite is installed on.

What is the telemetry configuration, and, what is the telemetry status showing on Cisco device?

Chenhu-source commented 1 year ago

It shows us "Transmit Error"

N93180EX-1-netconf(conf-tm-dest)# show running-config telemetry

!Command: show running-config telemetry !Running configuration last done at: Mon Mar 13 10:46:03 2023 !Time: Mon Mar 13 10:46:08 2023

version 9.3(7) Bios:version 07.69 feature telemetry

telemetry destination-group 1 ip address 10.106.116.197 port 60001 protocol gRPC encoding GPB use-vrf management sensor-group 1 data-source NX-API path "show environment power" depth 0 path "show system resources" depth 0 path "show version" depth 0 subscription 1 dst-grp 1 snsr-grp 1 sample-interval 10000

N93180EX-1-netconf# show telemetry transport

Session Id IP Address Port Encoding Transport Status

0 10.106.116.197 60001 GPB gRPC Transmit Error <<<<<<<<<<

Retry buffer Size: 10485760
Event Retry Messages (Bytes): 0
Timer Retry Messages (Bytes): 0
Total Retries sent: 0
Total Retries Dropped: 0

N93180EX-1-netconf#

N93180EX-1-netconf# ping 10.106.116.197 vrf management PING 10.106.116.197 (10.106.116.197): 56 data bytes 64 bytes from 10.106.116.197: icmp_seq=0 ttl=45 time=132.169 ms 64 bytes from 10.106.116.197: icmp_seq=1 ttl=45 time=128.827 ms 64 bytes from 10.106.116.197: icmp_seq=2 ttl=45 time=128.523 ms 64 bytes from 10.106.116.197: icmp_seq=3 ttl=45 time=130.285 ms 64 bytes from 10.106.116.197: icmp_seq=4 ttl=45 time=128.336 ms

--- 10.106.116.197 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 128.336/129.628/132.169 ms N93180EX-1-netconf#

That made the N9K telemetry thread stuck an error.

[03/13/23 11:09:25.748 UTC 132ef 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:09:30.215 UTC 132f0 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:09:31.025 UTC 132f1 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:09:31.359 UTC 132f2 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:09:40.221 UTC 132f3 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:09:41.059 UTC 132f4 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:09:41.403 UTC 132f5 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:09:50.283 UTC 132f6 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:09:51.038 UTC 132f7 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:09:51.378 UTC 132f8 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:10:00.183 UTC 132f9 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress [03/13/23 11:10:01.028 UTC 132fa 16695] [3450702656][tm_gpb_wrapper.cc:3029] getsockopt error reason:Operation now in progress 5001 entries printed

miott commented 1 year ago

Totally missed this. Kept concentrating on the ports. The docker container has its internal IP address. You can find out what that is using this command:

sudo docker container inspect docker-yangsuite-1

A lot of json will print out and near the end you will see "IPAddress": \<internal IP address>. For me it was "IPAddress": "172.18.0.2",

<snip>
                    "EndpointID": "bb7cec80f24424f373ccf1a74f320dc0d823c5a8da509021c770ac578c083ce6",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

Used that IP address in the gRPC window and everything started working. Keep in mind, the Cisco device still needs to have telemetry configured to point at the IP address the docker container is hosted on (not the internal docker container IP address).