Closed Krisscut closed 1 year ago
Roytak
commented
1 year ago Hello, thank you for reporting and investigating this. All of the CTN entries with matching fingerprint should be tried, but like you said the username is only tried to be obtained from the first match. It will be fixed soon.
Krisscut
commented
1 year ago Hi !
Thanks for the confirmation ! I guess you were already working on it on your branch netconf3 ? 😀
By the way, to workaround that problem, I tried to configure 2 endpoints for a client, with each client using a dedicated client cert but I got the same issue. (in the log I could see connection failed on the first cert check done, but even though the correct cert was in the second endpoint, connection never went through !) So it might be another problem to look into as well.
In the end, since I had 2 certificates, I added the two cert in the same entry in the trustore. Then, in the endpoint I refered to these in the clientcerts, and added 2 entries in the cert-maps but with different fingerprints.
Roytak
commented
1 year ago Hello,
I guess you were already working on it on your branch netconf3 ? 😀
Not really, even though there have been a lot of changes this was bugged there as well. So I'm happy it'll be fixed.
By the way, to workaround that problem, I tried to configure 2 endpoints for a client, with each client using a dedicated client cert but I got the same issue. (in the log I could see connection failed on the first cert check done, but even though the correct cert was in the second endpoint, connection never went through !)
Alright, I think I understand what you meant now, I will look into it.
In the end, since I had 2 certificates, I added the two cert in the same entry in the trustore. Then, in the endpoint I refered to these in the clientcerts, and added 2 entries in the cert-maps but with different fingerprints.
Good, unfortunately though you'll have to wait at least a week for the fix for your original issue to be merged, since Michal is currently on holiday.
Krisscut
commented
1 year ago Good, unfortunately though you'll have to wait at least a week for the fix for your original issue to be merged, since Michal is currently on holiday.
That's fine, for now this workaround it sufficient enough, and Michal fully deserves to take a break given the past support he gave us !
By the way, to workaround that problem, I tried to configure 2 endpoints for a client, with each client using a dedicated client cert but I got the same issue. (in the log I could see connection failed on the first cert check done, but even though the correct cert was in the second endpoint, connection never went through !)
Alright, I think I understand what you meant now, I will look into it.
Just to clarify this point because I'm not sure I was clear, I tried with something like this:
tls_trustore.xml
<truststore xmlns="urn:ietf:params:xml:ns:yang:ietf-truststore">
<certificates>
<name>odusimcerts1</name>
<certificate>
<name>EP-ODU-NO-SAN</name>
<cert>cert key</cert>
</certificate>
<certificates>
<name>odusimcerts2</name>
<certificate>
<name>EP-ODU-SAN</name>
<cert>cert key</cert>
</certificate>
</certificates>
<certificates>
<name>cacerts</name>
<certificate>
<name>cacert</name>
<cert>cert key</cert>
</certificate>
</certificates>
</truststore>Then in the call-home part I had 2 endpoints:
tls_callhome.xml
<netconf-server xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-server">
<call-home>
<netconf-client>
<name>NC-Client-1</name>
<endpoints>
<endpoint>
<name>tls-endpoint-NO-SAN</name>
<tls>
<tcp-client-parameters>
<remote-address>172.1.1.5</remote-address>
<remote-port>4335</remote-port>
<keepalives>
<idle-time>1</idle-time>
<max-probes>3</max-probes>
<probe-interval>1</probe-interval>
</keepalives>
</tcp-client-parameters>
<tls-server-parameters>
<server-identity>
<keystore-reference>
<asymmetric-key>genkey</asymmetric-key>
<certificate>orusimcert</certificate>
</keystore-reference>
</server-identity>
<client-authentication>
<required/>
<ca-certs>cacerts</ca-certs>
<client-certs>odusimcerts1</client-certs>
<cert-maps>
<cert-to-name>
<id>1</id>
<fingerprint>fingerprint1</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:specified</map-type>
<name>oranuser@o-ran.org</name>
</cert-to-name>
</cert-maps>
</client-authentication>
</tls-server-parameters>
</tls>
</endpoint>
<endpoint>
<name>tls-endpoint-SAN</name>
<tls>
<tcp-client-parameters>
<remote-address>172.1.1.5</remote-address>
<remote-port>4335</remote-port>
<keepalives>
<idle-time>1</idle-time>
<max-probes>3</max-probes>
<probe-interval>1</probe-interval>
</keepalives>
</tcp-client-parameters>
<tls-server-parameters>
<server-identity>
<keystore-reference>
<asymmetric-key>genkey</asymmetric-key>
<certificate>orusimcert</certificate>
</keystore-reference>
</server-identity>
<client-authentication>
<required/>
<ca-certs>cacerts</ca-certs>
<client-certs>odusimcerts2</client-certs>
<cert-maps>
<cert-to-name>
<id>1</id>
<fingerprint>fingerprint2</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:san-rfc822-name</map-type>
</cert-to-name>
</cert-maps>
</client-authentication>
</tls-server-parameters>
</tls>
</endpoint>
</endpoints>
<connection-type>
<persistent/>
</connection-type>
</netconf-client>
</call-home>
</netconf-server>So from my understanding, it should for instance have failed the first endpoint, but then manage to work on the second one, if I used the cert matching the SAN entry. But from my test it seems it failed the first endpoint check and as such never manage to work for the second as well.
But maybe the endpoint system is not intended for that, and more to have an endpoint ssh and the other one tls. I just wanted to share the potential issue since I stumbled upon it !
Roytak
commented
1 year ago Hello, I have recreated what you described.
But from my test it seems it failed the first endpoint check and as such never manage to work for the second as well.
You are right, from my testing the authentication, or rather CTN, fails on the first endpoint (as expected), but never manages to get to the second endpoint where it should work.
I managed to make it work by setting some reconnect-strategy params (just so I don't have to wait for so long) and by calling nc_accept_callhome in a cycle. The logs then look like this:
[INF]: Call Home client "ch_tls" endpoint "endpt" connecting...
[INF]: Trying to connect via IPv4 to 127.0.0.1:10010.
[INF]: Successfully connected to 127.0.0.1:10010 over IPv4.
[INF]: Accepted a connection on 127.0.0.1:10010 from 127.0.0.1:38320.
[INF]: Server certificate verified (domain "127.0.0.1").
[INF]: Cert verify: depth 1.
[INF]: Cert verify: subject: /C=CZ/ST=Some-State/L=Brno/O=CESNET/OU=TMC/CN=serverca.
[INF]: Cert verify: issuer: /C=CZ/ST=Some-State/L=Brno/O=CESNET/OU=TMC/CN=serverca.
[INF]: Cert verify CTN: cert fail, cert-to-name will continue on the next cert in chain.
[INF]: Cert verify: depth 0.
[INF]: Cert verify: subject: /C=CZ/L=Brno/OU=Cesnet/CN=TMC/emailAddress=test@test.com.
[INF]: Cert verify: issuer: /C=CZ/ST=Some-State/L=Brno/O=CESNET/OU=TMC/CN=serverca.
** [INF]: Cert-to-name unsuccessful, dropping the new client. **
[ERR]: Client certificate error (application verification failure).
[ERR]: SSL accept failed (certificate verify failed).
libyang[2]: Searching for "ietf-inet-types" in "/home/roman/libnetconf2/build".
libyang[2]: Newer revision than "ietf-inet-types@2013-07-15" not found, using this as the latest revision.
libyang[2]: Searching for "ietf-yang-types" in "/home/roman/libnetconf2/build".
libyang[2]: Newer revision than "ietf-yang-types@2013-07-15" not found, using this as the latest revision.
** [ERR]: Communication socket unexpectedly closed. **
[INF]: Call Home client "ch_tls" timeout of 5 seconds expired, reconnecting.
[INF]: Call Home client "ch_tls" endpoint "endpt" failed connection attempt limit 1 reached.
[INF]: Call Home client "ch_tls" endpoint "endpt2" connecting...
[INF]: Trying to connect via IPv4 to 127.0.0.1:10011.
[INF]: Successfully connected to 127.0.0.1:10011 over IPv4.
[INF]: Accepted a connection on 127.0.0.1:10011 from 127.0.0.1:57704.
[INF]: Server certificate verified (domain "127.0.0.1").
[INF]: Cert verify: depth 1.
[INF]: Cert verify: subject: /C=CZ/ST=Some-State/L=Brno/O=CESNET/OU=TMC/CN=serverca.
[INF]: Cert verify: issuer: /C=CZ/ST=Some-State/L=Brno/O=CESNET/OU=TMC/CN=serverca.
[INF]: Cert verify CTN: cert fail, cert-to-name will continue on the next cert in chain.
[INF]: Cert verify: depth 0.
[INF]: Cert verify: subject: /C=CZ/L=Brno/OU=Cesnet/CN=TMC/emailAddress=test@test.com.
[INF]: Cert verify: issuer: /C=CZ/ST=Some-State/L=Brno/O=CESNET/OU=TMC/CN=serverca.
[INF]: Cert verify CTN: entry with a matching fingerprint found.
** [INF]: Cert verify CTN: new client username recognized as "FooBar@Example.COM". **
[INF]: Client certificate verified.The client certificate's fingerprint is in the 2nd endpoint and it has a DNS SAN map type. The potential issue might lie here [ERR]: Communication socket unexpectedly closed.. And this is probably what didn't work for you (that's why I called nc_accept_callhome multiple times). Just because the authentication failed, the file descriptor does not have to be closed in my opinion. I will discuss this behaviour (if it is expected or so) with Michal, so thank you.
But maybe the endpoint system is not intended for that, and more to have an endpoint ssh and the other one tls. I just wanted to share the potential issue since I stumbled upon it !
The endpoints are intended for this exact purpose, you may have any number of TLS or SSH endpoints (but not UNIX socket) for a CH client. If a connection fails a reconnection is attempted based on your connection-type and reconnect-strategy settings. But for that you have to handle the client side correctly.
Krisscut
commented
1 year ago Thanks for taking the time to reproduce and investigate it, much appreciated !
It's worth noting on my side even if the ticket is opened on libnetconf2 , I'm using it through the netopeer2-server and netopeer2-cli so I don't think I'm able to use your workaround of calling the nc_callhome repeatedly.
So you mean on client side the connection is dropped as well and we need to retry the listen in loop ?
Should we create a separate issue to track this since I guess it's not exactly the same root cause ?
Roytak
commented
1 year ago Not a problem and my bad, I completely forgot you mentioned that you were using netopeer2. I take back what I wrote about using the client side correctly.
So you mean on client side the connection is dropped as well and we need to retry the listen in loop ?
My knowledge of netopeer2 only goes so far, but based on my analysis of libnetconf2 behavior, the connection is definitely dropped, most likely from the client side. You'll have to wait to hear from Michal if this is expected or not, but I am assuming that it is indeed not the right behavior.
Should we create a separate issue to track this since I guess it's not exactly the same root cause ?
If you're willing to take the time and provide further information (verbosity logs, etc.) then that would be a great idea. I'd take another look into it based on the information.
Roytak
commented
1 year ago Hi again,
firstly I'd like to thank you, we managed to find and address an issue similar to what you described.
Secondly as to the second issue you mentioned - the use case with netopeer2-cli won't be fixed at least for now. If you wish to use the two endpoints like this, you will have to run netopeer2-cli in a cycle (or just try to accept the Call Home session multiple times somehow). That is because based on the server's Call Home Reconnect Strategy parameters the server tries to connect to another endpoint after some number of unsuccessful attempts and by doing so it might eventually connect to the right one.
Hope this answers your questions.
Krisscut
commented
1 year ago Hi,
Thanks a lot for solving these issues. Yes no problem for the usecase with netopeer2-cli, I tried to go this rout eof having 2 endpoints as an experiment but the solution with 2 certs in the truststore is working fine for me in the meantime. Do you know when a new official version will be released to embed those fixes ?
I usually try to use "released" version, but I might try with devel otherwise.
Thanks again, I will mark the issue as solved since I think everything is fixed !
michalvasko
commented
1 year ago There should be a new release this week, if some problems occur the next.
Hi,
On a setup using netopeer2-server, we are trying to perform a TLS connection without a username in the certificate. In the cert-maps part of the xml to configure the call-home / tls-server-parameters, we have multiple entries for a given fingerprint. With current implementation in libnetconf2 it seems only the first entry matching the fingerprint is used, and in case of error the cert validation fail with a message Certificate has no SANs or failed to retrieve them.
This is incorrect if we refer to this yang: https://github.com/CESNET/netopeer2/blob/964abf5cf8df77e809aca8865ad7ca7e8dd9d800/modules/ietf-x509-cert-to-name%402014-12-10.yang#L197-L266
I checked the code, issue seems to be here: https://github.com/CESNET/libnetconf2/blob/ac1786b742cff30a57b7ca51862b68b308cc87ef/src/session_server_tls.c#L870-L897 If no SAN username is found in the nc_tls_ctn_get_username_from_cert function, (https://github.com/CESNET/libnetconf2/blob/ac1786b742cff30a57b7ca51862b68b308cc87ef/src/session_server_tls.c#L193C1-L225), it should go back in the nc_tls_cert_to_name function fetch the next cert-to-name entry which matches the fingerprint and try again with it until either a username is found, or all the entry are exhausted.
Versions used netopeer2-2.1.62 libnetconf2-2.1.34
tls_callhome.xml configuration file used
Logs
Line 104073: [INF]: LN: Listening on 172.22.0.2:6513 for TLS connections. Line 104726: [INF]: LN: Call Home client "NC-Client-1" endpoint "default-tls" connecting... Line 104727: [INF]: LN: Trying to connect via IPv4 to 172.21.0.2:4335. Line 104728: [INF]: LN: Successfully connected to 172.21.0.2:4335 over IPv4. Line 104732: [INF]: LN: Cert verify: depth 1. Line 104733: [INF]: LN: Cert verify: subject: /CN=CB007967TA_SLCA. Line 104734: [INF]: LN: Cert verify: issuer: /CN=CB007967TA_SLCA. Line 104735: [INF]: LN: Cert verify CTN: cert fail, cert-to-name will continue on the next cert in chain. Line 104736: [INF]: LN: Cert verify: depth 0. Line 104737: [INF]: LN: Cert verify: subject: /CN=ODU_EE_27_1. Line 104738: [INF]: LN: Cert verify: issuer: /CN=CB007967TA_SLCA. Line 104739: [INF]: LN: Cert verify CTN: entry with a matching fingerprint found. Line 104740: [WRN]: LN: Certificate has no SANs or failed to retrieve them. Line 104741: [INF]: LN: Cert-to-name unsuccessful, dropping the new client. Line 104742: [ERR]: LN: Client certificate error (application verification failure). Line 104743: [ERR]: LN: SSL accept failed (certificate verify failed).