gnmic with tls-ca : context deadline exceeded

[mghildiy]

I am trying to run a gnmic command with tls-ca option:

gnmic -u admin -p admin --tls-ca /path/to/root-ca -a <ip>:<port> capabilities

I have configured gNMI on mimic, with TLS certificate and key(corresponding to root ca) configured.

But this doesn't work, with error:

target ".......", capabilities request failed: failed to create a gRPC client for target ......: context deadline exceeded
Error: one or more requests failed
But with --insecure flag things work. So something is not working when I try to do TLS verification.

What can be missing in my whole setup, either on gNMI side or from gnmic side?

[karimra]

Run the same command with the debug flag -d to get more details on the error.

[mghildiy]

I tried with -d option, and a part of log is(replaced ip, port for security reasons):

2023/11/06 08:45:06.511317 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Channel Connectivity change to IDLE
2023/11/06 08:45:06.511364 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Subchannel Connectivity change to CONNECTING
2023/11/06 08:45:06.511388 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Subchannel picks a new address "<ip>:<port>" to connect
2023/11/06 08:45:06.511472 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000402890, {CONNECTING <nil>}
2023/11/06 08:45:06.511514 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Channel Connectivity change to CONNECTING
2023/11/06 08:45:06.513179 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:65: [gnmic] [core] grpc: addrConn.createTransport failed to connect to {<ip>:<port> <ip>:<port> <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
2023/11/06 08:45:06.513222 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Subchannel Connectivity change to TRANSIENT_FAILURE
2023/11/06 08:45:06.513287 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000402890, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"}
2023/11/06 08:45:06.513307 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Channel Connectivity change to TRANSIENT_FAILURE
2023/11/06 08:45:10.075950 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Subchannel Connectivity change to IDLE
2023/11/06 08:45:10.076063 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000402890, {IDLE connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"}
2023/11/06 08:45:10.076085 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Channel Connectivity change to IDLE
2023/11/06 08:45:10.076131 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Subchannel Connectivity change to CONNECTING
2023/11/06 08:45:10.076156 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Subchannel picks a new address "<ip>:<port>" to connect
2023/11/06 08:45:10.076184 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000402890, {CONNECTING <nil>}
2023/11/06 08:45:10.076221 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Channel Connectivity change to CONNECTING
2023/11/06 08:45:10.077759 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:65: [gnmic] [core] grpc: addrConn.createTransport failed to connect to {<ip>:<port> <ip>:<port> <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
2023/11/06 08:45:10.077830 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Subchannel Connectivity change to TRANSIENT_FAILURE
2023/11/06 08:45:10.077875 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000402890, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"}
2023/11/06 08:45:10.077898 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Channel Connectivity change to TRANSIENT_FAILURE
2023/11/06 08:45:10.860153 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Channel Connectivity change to SHUTDOWN
2023/11/06 08:45:10.860238 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.44.0/grpclog/logger.go:53: [gnmic] [core] Subchannel Connectivity change to SHUTDOWN
2023/11/06 08:45:10.860307 /home/runner/work/gnmic/gnmic/app/logging.go:13: [gnmic] target "<ip>:<port>", capabilities request failed: failed to create a gRPC client for target "<ip>:<port>" : <ip>:<port>: context deadline exceeded
target "<ip>:<port>", capabilities request failed: failed to create a gRPC client for target "<ip>:<port>" : <ip>:<port>: context deadline exceeded

Not very clear abt exact cause but this part seems to say something wrong with tls handshake:

transport: authentication handshake failed: tls: first record does not look like a TLS handshake

Though I am not sure of this issue.

[karimra]

That means the server does not support TLS or is not configured to support TLS.