GET failure due to TLS failure : cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs"

Shikha-Chowdhary commented 1 year ago

Hello,

I would like to use the same server certificate on multiple NEs, hence not adding IP in the cert. But the secure GET request fails with the following error during handshake.

The C++ grpc gives an option to override the target name-> args.SetSslTargetNameOverride(name); //This signifies common name present in X509 certificate
Any such option available in gnmic too?

}. Err: connection error: desc = **"transport: authentication handshake failed: x509: cannot validate certificate for 192.168.0.15 because it doesn't contain any IP SANs"**

2023/07/10 11:48:36.612273 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/grpclog/logger.go:53: [gnmic] [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: authenti**cation handshake failed: x509: cannot validate certificate for 192.168.0.15 because it doesn't contain any IP SANs"**

2023/07/10 11:48:36.612400 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000621920, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: x509: cannot validate certificate for 192.168.0.15 because it doesn't contain any IP SANs"}

-Thanks Shikha

karimra commented 1 year ago

Can you give more context to your issue? How are you using gNMIc in this case: client or server? Can you show the TLS config you pass to gNMIc (config file and/or flag) ?

Shikha-Chowdhary commented 1 year ago

I am using gnmic as a client , without mutual authentication/mTLS

gnmic --config ~/dialin.yaml --tls-ca ~/certificates/SERVER/root1k.crt get --path /shelves/shelf --debug

dialin.yaml contents

targets:
hostname1:
name: "Shikha"
address: 192.168.0.15:10161
username: ADMIN
password: ADMIN

karimra commented 1 year ago

I see, the option of overriding the serverName when checking the hostname in the returned certificate is not exposed by gNMIc. I could add it as config option.

To make sure I understand; You plan to use the same certificate on all devices, that certificate will have a single serverName that you want use to override the real hostname during certificate verification ?

Shikha-Chowdhary commented 1 year ago

Yup The host name is matched against the Common Name in the certificate's Subject. That's why the single server name is added as common-name.

Shikha-Chowdhary commented 1 year ago

Another similar issue but when using GNMIC in tunnel server mode

When used a standalone client, TLS works if host IP is present in server certificate, which is expected.
The same certificate fails for client, when GNMIC is used a tunnel server with the following error.
Here "shikha12345" is my tunnel target name/ID

The above fix will take of this scenario as well ?

"transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match shikha12345

...
  "Addresses": [
    {
      "Addr": "shikha12345",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Type": 0,
      "Metadata": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2023/07/12 05:02:18.429447 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/grpclog/logger.go:53: [gnmic] [core] [Channel #5] Channel switches to new LB policy "pick_first"
2023/07/12 05:02:18.429484 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/grpclog/logger.go:53: [gnmic] [core] [Channel #5 SubChannel #6] Subchannel created
2023/07/12 05:02:18.429505 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/grpclog/logger.go:53: [gnmic] [core] [Channel #5] Channel Connectivity change to CONNECTING
2023/07/12 05:02:18.429553 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/grpclog/logger.go:53: [gnmic] [core] [Channel #5 SubChannel #6] Subchannel Connectivity change to CONNECTING
2023/07/12 05:02:18.429571 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/grpclog/logger.go:53: [gnmic] [core] [Channel #5 SubChannel #6] Subchannel picks a new address "shikha12345" to connect
2023/07/12 05:02:18.429618 /home/runner/work/gnmic/gnmic/app/tunnel.go:201: [gnmic] dialing tunnel connection for tunnel target "shikha12345"
2023/07/12 05:02:18.429684 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc00060e918, {CONNECTING <nil>}
2023/07/12 05:02:18.998960 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/grpclog/logger.go:53: [gnmic] [core] Creating new client transport to "{\n  \"Addr\": \"shikha12345\",\n  \"ServerName\": \"shikha12345\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}": connection error: desc = "transport: authentication handshake failed: x509: certificate is not valid for any names, but wanted to match shikha12345"
...
...

YAML contents for secure client layer, but insecure/plaintext tunnel layer.


username: ADMIN
password: ADMIN
log: true
tls-ca: /home/shikha/CertsIP/root1k.crt

tunnel-server: insecure: true address: ":50051" target-wait-time: 30s

karimra commented 1 year ago

The fix should work for connections made over a tunnel as well, proving you configure the tls-server-name attribute

Shikha-Chowdhary commented 1 year ago

It's working for both GNMIC as client & as tunnel server. Thanks a lot

Shikha-Chowdhary commented 1 year ago

The new option works well when config is given through YAML, but fails when given from command line.

Is this expected ?

1)bash-4.2$ gnmic -u ADMIN -p ADMIN  --tls-ca  ~/certificates/SERVER/root1k.crt  --tls-server-name "Shikha" get -a 192.168.0.15:10161  --path /shelves/shelf --debug 

Error: unknown flag: --tls-server-name
bash-4.2$ gnmic version
version : 0.31.2
 commit : 5f06069
   date : 2023-07-17T17:41:38Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
bash-4.2$

2) WORKING ->

 gnmic --config ~/dialin.yaml    get -a 192.168.0.15:10161  --path /shelves/shelf --debug 
username: ADMIN
password: ADMIN
log: true
tls-ca: /home/shikha/certificates/SERVER/root1k.crt
tls-server-name: "Shikha"
#tls-cert: /home/ateli/CA_Certs/CA_2048/clientCert.pem
#tls-key: /home/ateli/CA_Certs/CA_2048/clientCertKey.pem

karimra commented 1 year ago

Is this expected ?

Nope, forgot to bind the flag...

karimra commented 1 year ago

@Shikha-Chowdhary v0.31.3 should be good

Shikha-Chowdhary commented 1 year ago

New option available through command line now, but fails with some RSA error. ( Working through config file)

Command -> $ gnmic --tls-ca ~/CertsIP/root1k.crt --tls-server-name "shikha" get -a 192.168.0.15:10161 -u ADMIN -p ADMIN --path /shelves/shelf --debug
Error -> .... new client transport to "{Addr: \"192.168.0.15:10161\", ServerName: \"shikha\", }": connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"shikha\")"

Logs


(devops)bash-4.2$ ^C
(devops)bash-4.2$ gnmic --tls-ca  ~/CertsIP/root1k.crt --tls-server-name "shikha"  get -a 192.168.0.15:10161 -u ADMIN -p ADMIN  --path /shelves/shelf --debug
2023/07/20 05:47:36.751355 /home/runner/work/gnmic/gnmic/app/app.go:221: [gnmic] version=0.31.3, commit=4d78179, date=2023-07-19T23:36:39Z, gitURL=https://github.com/openconfig/gnmic, docs=https://gnmic.openconfig.net
...
...
tls-ca: /home/shikha/CertsIP/root1k.crt
tls-cert: ""
tls-key: ""
tls-max-version: ""
tls-min-version: ""
tls-server-name: shikha
tls-version: ""
token: ""
upgrade-use-pkg: false
use-tunnel-server: false
username: ADMIN

2023/07/20 05:47:36.754336 /home/runner/work/gnmic/gnmic/app/app.go:274: [gnmic] address='[192.168.0.15:10161]'([]string) 2023/07/20 05:47:36.754383 /home/runner/work/gnmic/gnmic/app/app.go:274: [gnmic] debug='true'(bool) 2023/07/20 05:47:36.754472 /home/runner/work/gnmic/gnmic/app/app.go:274: [gnmic] get-path='[/shelves/shelf]'([]string) 2023/07/20 05:47:36.754545 /home/runner/work/gnmic/gnmic/app/app.go:274: [gnmic] password='ADMIN'(string) 2023/07/20 05:47:36.754716 /home/runner/work/gnmic/gnmic/app/app.go:274: [gnmic] tls-ca='/home/shikha/CertsIP/root1k.crt'(string) 2023/07/20 05:47:36.754743 /home/runner/work/gnmic/gnmic/app/app.go:274: [gnmic] tls-server-name='shikha'(string) 2023/07/20 05:47:36.754767 /home/runner/work/gnmic/gnmic/app/app.go:274: [gnmic] username='ADMIN'(string) 2023/07/20 05:47:36.754857 /home/runner/work/gnmic/gnmic/config/config.go:376: [config] cmd=get, flagName=help, changed=false, isSetInFile=false 2023/07/20 05:47:36.754894 /home/runner/work/gnmic/gnmic/config/config.go:376: [config] cmd=get, flagName=model, changed=false, isSetInFile=false 2023/07/20 05:47:36.754936 /home/runner/work/gnmic/gnmic/config/config.go:376: [config] cmd=get, flagName=path, changed=true, isSetInFile=true 2023/07/20 05:47:36.754954 /home/runner/work/gnmic/gnmic/config/config.go:376: [config] cmd=get, flagName=prefix, changed=false, isSetInFile=false 2023/07/20 05:47:36.754972 /home/runner/work/gnmic/gnmic/config/config.go:376: [config] cmd=get, flagName=processor, changed=false, isSetInFile=false 2023/07/20 05:47:36.754989 /home/runner/work/gnmic/gnmic/config/config.go:376: [config] cmd=get, flagName=target, changed=false, isSetInFile=false 2023/07/20 05:47:36.755030 /home/runner/work/gnmic/gnmic/config/config.go:376: [config] cmd=get, flagName=type, changed=false, isSetInFile=false 2023/07/20 05:47:36.755050 /home/runner/work/gnmic/gnmic/config/config.go:376: [config] cmd=get, flagName=values-only, changed=false, isSetInFile=false 2023/07/20 05:47:36.755383 /home/runner/work/gnmic/gnmic/config/targets.go:45: [config] targets: map[192.168.0.15:10161:{"name":"192.168.0.15:10161","address":"192.168.0.15:10161","username":"ADMIN","password":"****","timeout":10000000000,"insecure":false,"tls-ca":"/home/shikha/CertsIP/root1k.crt","tls-cert":"","tls-key":"","skip-verify":false,"tls-server-name":"shikha","buffer-size":100,"retry-timer":10000000000,"log-tls-secret":false,"gzip":false,"token":""}] 2023/07/20 05:47:36.755439 /home/runner/work/gnmic/gnmic/config/actions.go:49: [config] actions: map[] 2023/07/20 05:47:36.755467 /home/runner/work/gnmic/gnmic/config/processors.go:45: [config] processors: map[] 2023/07/20 05:47:36.756580 /home/runner/work/gnmic/gnmic/app/get.go:125: [gnmic] sending gNMI GetRequest: prefix='', path='[elem:{name:"shelves"} elem:{name:"shelf"}]', type='ALL', encoding='JSON', models='[]', extension='[]' to 192.168.0.15:10161 2023/07/20 05:47:36.756635 /home/runner/work/gnmic/gnmic/app/app.go:516: [gnmic] creating gRPC client for target "192.168.0.15:10161" 2023/07/20 05:47:36.758212 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Channel created 2023/07/20 05:47:36.758271 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] original dial target is: "192.168.0.15:10161" 2023/07/20 05:47:36.758352 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] dial target "192.168.0.15:10161" parse failed: parse "192.168.0.15:10161": first path segment in URL cannot contain colon 2023/07/20 05:47:36.758375 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] fallback to scheme "passthrough" 2023/07/20 05:47:36.758425 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/192.168.0.15:10161 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} 2023/07/20 05:47:36.758445 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Channel authority set to "shikha" 2023/07/20 05:47:36.758812 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Resolver state updated: { "Addresses": [ { "Addr": "192.168.0.15:10161", "ServerName": "", "Attributes": null, "BalancerAttributes": null, "Type": 0, "Metadata": null } ], "ServiceConfig": null, "Attributes": null } (resolver returned new addresses) 2023/07/20 05:47:36.758906 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Channel switches to new LB policy "pick_first" 2023/07/20 05:47:36.758972 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1 SubChannel #2] Subchannel created 2023/07/20 05:47:36.759040 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Channel Connectivity change to CONNECTING 2023/07/20 05:47:36.759123 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING 2023/07/20 05:47:36.759158 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1 SubChannel #2] Subchannel picks a new address "192.168.0.15:10161" to connect 2023/07/20 05:47:36.759335 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000b14ed0, {CONNECTING } 2023/07/20 05:47:38.501389 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] Creating new client transport to "{Addr: \"192.168.0.15:10161\", ServerName: \"shikha\", }": connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"shikha\")" 2023/07/20 05:47:38.501507 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:65: [gnmic] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "192.168.0.15:10161", ServerName: "shikha", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"shikha\")" 2023/07/20 05:47:38.501573 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"shikha\")" 2023/07/20 05:47:38.501666 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000b14ed0, {TRANSIENT_FAILURE connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"shikha\")"} 2023/07/20 05:47:38.501690 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1] Channel Connectivity change to TRANSIENT_FAILURE 2023/07/20 05:47:39.502580 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"shikha\")" 2023/07/20 05:47:39.502697 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000b14ed0, {IDLE connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"shikha\")"} 2023/07/20 05:47:39.502777 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING 2023/07/20 05:47:39.502813 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] [Channel #1 SubChannel #2] Subchannel picks a new address "192.168.0.15:10161" to connect 2023/07/20 05:47:39.502992 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.1/grpclog/logger.go:53: [gnmic] [core] pickfirstBalancer: UpdateSubConnState: 0xc000b14ed0, {CONNECTING } ^C received signal 'interrupt'. terminating... (devops)bash-4.2$ ^C

karimra commented 1 year ago

It looks like the name is picked up properly. Does the tls-ca certificate used in that example contain the same name?

Shikha-Chowdhary commented 1 year ago

Yup

I am using the same server certificate, CA file and the same network element
All is good when given through YAML file, so I am able to continue working .
But behavior from command line is different for this parameter.

karimra commented 1 year ago

Can you share the debug logs from both successful and failure cases ?

Shikha-Chowdhary commented 1 year ago

Compared the logs on my own first and found that I was giving wrong value of "tls-ca" through command line. That's why getting the RSA error.

"tls-server-name" option working through config file as well as command line.

My bad, apologies for confusion.

karimra commented 1 year ago

All good, glad it works.

openconfig / gnmic

GET failure due to TLS failure : cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs" #172