Agents using LE cert with ecdsa

[D4V3M0NK]

Describe the bug We've been using MC2 for about 6 years now and have a number of agents (mainly Linux but a couple of Windows servers too) accessing our MC2 server which runs under an LE certificate (public internet > NGINX SSL proxy > MC2). We have to have TLS connections end to end (even within our environment) so we do have the myexternal.crt configured within MC2 (see config).

Today we generated a new LE certificate, loaded it in both the proxy and MC2 ... and now all our systems are unable to connect. When we roll back the older certificate (which expires midnight tonight) everything comes back online again.

I've checked to ensure that the DNS entries in the certs are identical and they are (the MC2 server is available using two URLs). However, I have noticed that the new LE cert is encrypted with ecdsa-with-SHA384 whereas the older (soon to expire) certificate is encrypted with sha256WithRSAEncryption.

I don't see any errors when MC2 starts up, so I'm quite convinced that it's able to handle the new certificate - however I'm aware of the fact that the agent may not be as up to date as the server is, and as a result, worry that it's unable to use a cert signed with ECDSA, as opposed to RSA.

Is there any method to this madness of thinking?! Can the agents handle ECDSA certs? I've checked the logs on the remote systems and all they report is the version of the agent.

please also see MeshCentralRouter Version below...

Server Software (please complete the following information):

• OS: Ubuntu 22 Server
• Virtualization:AWS EC2 instance
• Network: Proxied WAN
• Version: 1.1.24
• Node: 20.15.0

Client Device (please complete the following information):

• Device: Virtual machine
• OS: Ubuntu 18
• Network: Remote over WAN
• Browser: N/A (server version)
• MeshCentralRouter Version: When I attempt to run "meshagent --version" I get "unable to open database" on the remote...?

Remote Device (please complete the following information): Nto sure what the different is between a Client Device and a Remote Device?

Your config.json file

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "settings": {
    "cert": "mc2.gov.mydomain.com",
    "WANonly": true,
    "trustedproxy": ["sslproxyA/24","sslProxB/24"],
    "cookieIpCheck": false,
    "plugins": { "enabled": true }
  },
  "domains": {
    "": {
      "title": "MC2",
      "title2": "Remote Support Services",
      "TitlePicture": "title.png",
      "welcomePicture": "bg.jpg",
      "welcomePictureFullScreen": true,
      "welcomeText": "This...penalities.",
      "passwordRequirements": {
              "force2factor": true,
              "min": 12,
              "upper": 1,
              "lower": 1,
              "numeric": 1,
              "nonalpha": 1,
              "oldPasswordBan": 24
      },
      "autocomplete": false,
      "orphanAgentUser" : "admin@domain.com",
      "certUrl": "file:///home/ubuntu/mc2/meshcentral-data/myexternalcert.crt",
      "_ignoreAgentHashCheck": true
    }
  }
}

[si458]

if you are using a reverse proxy (nginx) you shouldnt need to load any certificates into meshcentral

1. set cert to your dns name of choice meshcentral.mydomain.com
2. set trustedproxy to your reverse proxy ip address
3. set tlsoffload to your reverse proxy ip address
4. set certurl to the the website url you are using for your meshcentral https://meshcentral.mydomain.com
5. set nginx to forward to http://meshcentralipaddress:443 DONT SET IT TO USE HTTPS

edit: https://ylianst.github.io/MeshCentral/meshcentral/#nginx-reverse-proxy-setup

[si458]

also note you already asked a similar question in the past and @Ylianst replied with basically the same as me above you need to tell meshcentral to load the certificate from your reverse proxy rather than the file https://github.com/Ylianst/MeshCentral/issues/5296

[D4V3M0NK]

Unfortunately due to the internal networking, we're unable to do that - which is why we're taking the same cert that's on the proxy server, and copying it into myexternal.crt

[D4V3M0NK]

(and reverting the certificate back to the older one, agents run without issue)

[si458]

@D4V3M0NK so you cant simply curl -v https://meshcentral.mydomain.com on your meshcentral machine and get a reply? that seems a bit odd or very strict networking? but yes ive checked the functions in meshcentral and the decode of the certs reads // Decode a RSA certificate and hash the public key, if this is not RSA, skip this. so indeed it doesnt work yet with ECDSA certificates, but sadly all my lets encrypt certificates are RSA256, even the one i just issued today from my meshcentral dev machine, so i need to get a ECDSA certificate to then do some testing, as we do indeed use forge nodejs package so in theory it should work maybe?

[D4V3M0NK]

@si458 - I sincerely appreciate your responses and you're correct, we have a requirement for incredibly secure networking. I really appreciate you taking the time to look into this and I'm glad that the issue has at least been identified.

LE will (by default) now produce EC certs as long as certbot 2.0 or later is being used. We use their docker container to re-generate the certs and, according to their documentation, ECDSA certs will be produced, unless prior certs were RSA. For some reason that didn't work in our case, we historically keep all prior certs for verification purposes, and found that between yesterday and 90 days prior, the key changed from RSA to EC.

We've been able to re-generate RSA certs, but it's still not working as we expected so there's likely something else going on that I am investigating (I suspect a DNS mismatch, but...)

Could I ask - is there any additional logging that could be enabled on the agent's side? Reason being is that we used URL-A to register the agents against a couple of years back. Due to processes that would take too long to get into, we recently (6 months?) use URL-B now - with the proxies routing between the two. With the exception of the MFA on the server side, this "dual DNS" works without any issues - up until these certs. Are you aware of any other areas (specifically on the agent side) where this DNS would run foul of processing the certs? All generated certs have both URL-A and URL-B as SubjectAltNames (along with a range of other hosts that we use to point to the proxy server: we then use AWS Application Load Balancing to route the traffic to Auto Scaling Groups based upon the hostname being called).

[si458]

@D4V3M0NK oh wow you have a very complex setup! way too complex for me to even understand!

but from what im aware when i look at the meshcentral code, all meshcentral does is verify that the ssl hash the remote agent sent it is the same as its ssl hash it got from the certurl, if they dont match then bye bye disconnect.

as for having mutliple dns entries in the SSL, i dont think this should matter so long as whichever url you use for the meshagent to connect to, is the same as the certurl in meshcentral, as meshcentral ONLY checks 1 SSL cert, and if its invalid for whatever reason, it then checks again just be to sure when a remote agent connects again.

i could be totally mistaken to everything ive said, im still learning all the things meshcentral has hidden/feature it has as i fix bugs and discover things by talking to people and there setups!

one other thing to note, from what ive read with other issues, meshcentral DOESNT support the SPLIT dns. e.g. meshcentral.example.com on the internet which points to ur external ip but using meshcentral.example.local for internal use to point to ur internal ip this has something to do with the certurl and the cert values again if i remember

[D4V3M0NK]

@si458 : can I ask - would an agent have issue with a 4096 bit size RSA key, rather than a 2048 bit key? I appreciate that the hash should still match, but that's the only other item (on the cert side of things) that I can see as changing. I would think that the keysize wouldn't matter, maybe take a few milliseconds more to compute, but as I said, I thought I'd ask...

[si458]

@D4V3M0NK sadly im not 100% sure, my speciality is nodejs and web front end things, i dont deal with C but i am trying to learn, but from what im aware @Ylianst wrote everything with RSA 3072 in mind! so i cant see rsa 4096 effecting anything

[D4V3M0NK]

@si458 I can confirm that LE RSA 4096 bit keys work without issue on the agent.