search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
3.67k stars 511 forks source link

Intel AMT Bare-metal Activation Key Mismatch #4901

Open ross-at-ormo opened 1 year ago

ross-at-ormo commented 1 year ago

Describe your issue I encounter an error on my MeshCentral server (which also crashes the service) every time my Intel AMT device attempts to contact the remote activation service. In the mesherrors.txt log file I see this error:

node:internal/tls/secure-context:92
  context.setKey(key, passphrase);
          ^

Error: error:05800074:x509 certificate routines::key values mismatch
    at setKey (node:internal/tls/secure-context:92:11)
    at configSecureContext (node:internal/tls/secure-context:174:7)
    at Object.createSecureContext (node:_tls_common:117:3)
    at Object.connect (node:_tls_wrap:1629:48)
    at CreateWsmanComm.obj.xxConnectHttpSocket (/home/rosu/node_modules/meshcentral/amt/amt-wsman-comm.js:284:38)
    at CreateWsmanComm.obj.PerformAjaxExNodeJS2 (/home/rosu/node_modules/meshcentral/amt/amt-wsman-comm.js:128:41)
    at CreateWsmanComm.obj.PerformAjaxExNodeJS (/home/rosu/node_modules/meshcentral/amt/amt-wsman-comm.js:116:85)
    at CreateWsmanComm.obj.PerformAjaxEx (/home/rosu/node_modules/meshcentral/amt/amt-wsman-comm.js:109:20)
    at CreateWsmanComm.obj.PerformAjax (/home/rosu/node_modules/meshcentral/amt/amt-wsman-comm.js:87:17)
    at Object.PerformAjax (/home/rosu/node_modules/meshcentral/amt/amt-wsman.js:40:18) {
  library: 'x509 certificate routines',
  reason: 'key values mismatch',
  code: 'ERR_OSSL_X509_KEY_VALUES_MISMATCH'
}

A bit of background - I was initially trying to use an AMT activation certificate that I signed using my own internal CA and was using USB setup.bin to add the root CA cert hash to the trusted root store in the management engine. When I first encountered the problem I thought maybe I had done something wrong when I added my custom certs to MeshCentral. I verified that the private and public keys of my custom activation cert were matching (using hash of modulus) so this didn't seem to make sense based on the error.

Then I realised that MeshCentral already has a wildcard certificate for AMT activation so I could rule out my certs being the issue. I deleted my MeshCentral server configuration and created a new one from scratch. The only changes I made to the vanilla install was to create an Intel AMT only device group, updated the amtProvisioningServer node in the config.json file to include the details of my MeshCentral server, and generate the new setup.bin file from the ACM option in the device group. I validated that the setup.bin file contained the correct server details and the hash of the MeshCentral root CA cert. Again, after applying the new setup.bin settings to my client device the MeshCentral server crashed with the same key values mismatch error.

I checked the config in the MEBx and it all looks fine. If I stop/start the remote configuration manually in MEBx this triggers the error in MeshCentral within a few seconds and causes the MeshCentral to crash/restart (the web management interface becomes unavailable and after a few seconds it comes back at the login page). So when the remote device is contacting the server for activation it definitely appears to the be the trigger for the error. Have also done a port scan to validate that port 9971 is open on the MeshCentral server.

I'm stuck for ideas so any help/suggestions would be appreciated.

Server Software (please complete the following information):

Client Device (please complete the following information):

Remote Device (please complete the following information):

Your config.json file

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "settings": {
    "redirPort": 80,
    "port": 9443
    "cert": "meshcentral.mydomain.com.au",
    "amtProvisioningServer": {
      "newMebxPassword": "<SECRET_PW>",
      "ip": "192.168.0.35",
      "trustedFqdn": "meshcentral.mydomain.com.au",
      "deviceGroup": "mesh//iD3DWP6E7t$ZvHZvbrgCD05E3OqjXHhKPj6eXQ2keWYpo15k@c34QQGlWZG52T6w"
    },
  },
  "domains": {
    "": {
      "newAccounts": false,
      "title2": "Sub title",
      "title": "MyTestServer"
    }
  }
}
kmeaw commented 9 months ago

I can reproduce this error on a desktop PC with ASUS Q170T motherboard running ME v11.8.65.

kmeaw commented 9 months ago

Probably there is a problem in the processHelloDataEx1 function of amtprovisioningserver.js - it reverses the cert chain so the MeshCentral root certificate goes to the top. The doc at https://nodejs.org/api/tls.html#tlscreatesecurecontextoptions describing options.cert says:

Each cert chain should consist of the PEM formatted certificate for a provided private key, followed by the PEM formatted intermediate certificates (if any), in order
dinger1986 commented 7 months ago

AMT issue @si458

ggzengel commented 6 months ago

@Ylianst AMT bare-metal activation is still broken. Can somebody fix this?

aggietallboy commented 5 months ago

I just ran into this same issue today. For now I've disabled AmtProvisioningServer

aggietallboy commented 5 months ago

By any chance are you running a PFX for your cert? My suspicion is that we may be extracting the PFX backwards.

I'm going to try to redo mine, specifying the three certs and the key individually to see if that improves things.

aggietallboy commented 5 months ago

I ran a couple variants: 1) separating out the PFX and doing the four certs and the keyfile separately 1 a) ordered as leaf_int1_int2_ca 1 b) ordered as ca_int2_int1_leaf

No luck, still getting Error: error:05800074:x509 certificate routines::key values mismatch at setKey (node:internal/tls/secure-context:93:11) at configSecureContext (node:internal/tls/secure-context:204:7) at Object.createSecureContext (node:_tls_common:116:3) at Object.connect (node:_tls_wrap:1763:48) at CreateWsmanComm.obj.xxConnectHttpSocket (/home/ec2-user/node_modules/meshcentral/amt/amt-wsman-comm.js:284:38) at CreateWsmanComm.obj.PerformAjaxExNodeJS2 (/home/ec2-user/node_modules/meshcentral/amt/amt-wsman-comm.js:128:41) at CreateWsmanComm.obj.PerformAjaxExNodeJS (/home/ec2-user/node_modules/meshcentral/amt/amt-wsman-comm.js:116:85) at CreateWsmanComm.obj.PerformAjaxEx (/home/ec2-user/node_modules/meshcentral/amt/amt-wsman-comm.js:109:20) at CreateWsmanComm.obj.PerformAjax (/home/ec2-user/node_modules/meshcentral/amt/amt-wsman-comm.js:87:17) at Object.PerformAjax (/home/ec2-user/node_modules/meshcentral/amt/amt-wsman.js:40:18) at Object.ExecGet (/home/ec2-user/node_modules/meshcentral/amt/amt-wsman.js:106:13) at Get (/home/ec2-user/node_modules/meshcentral/amt/amt.js:67:66) at _BatchNextEnum (/home/ec2-user/node_modules/meshcentral/amt/amt.js:147:9) at AmtStackCreateService.obj.BatchEnum (/home/ec2-user/node_modules/meshcentral/amt/amt.js:135:9) at processHelloDataEx1 (/home/ec2-user/node_modules/meshcentral/amtprovisioningserver.js:185:22) at QueryReqWrap.dnsReverseLoopup [as callback] (/home/ec2-user/node_modules/meshcentral/amtprovisioningserver.js:140:13) { library: 'x509 certificate routines', reason: 'key values mismatch', code: 'ERR_OSSL_X509_KEY_VALUES_MISMATCH'

The key was extracted from the PFX, and then decrypted and uploaded.

aggietallboy commented 5 months ago

ca | <string[]> | | <Buffer[]> Optionally override the trusted CA certificates. Default is to trust the well-known CAs curated by Mozilla. Mozilla's CAs are completely replaced when CAs are explicitly specified using this option. The value can be a string or Buffer, or an Array of strings and/or Buffers. Any string or Buffer can contain multiple PEM CAs concatenated together. The peer's certificate must be chainable to a CA trusted by the server for the connection to be authenticated. When using certificates that are not chainable to a well-known CA, the certificate's CA must be explicitly specified as a trusted or the connection will fail to authenticate. If the peer uses a certificate that doesn't match or chain to one of the default CAs, use the ca option to provide a CA certificate that the peer's certificate can match or chain to. For self-signed certificates, the certificate is its own CA, and must be provided. For PEM encoded certificates, supported types are "TRUSTED CERTIFICATE", "X509 CERTIFICATE", and "CERTIFICATE". See also tls.rootCertificates. cert | <string[]> | | <Buffer[]> Cert chains in PEM format. One cert chain should be provided per private key. Each cert chain should consist of the PEM formatted certificate for a provided private key, followed by the PEM formatted intermediate certificates (if any), in order, and not including the root CA (the root CA must be pre-known to the peer, see ca). When providing multiple cert chains, they do not have to be in the same order as their private keys in key. If the intermediate certificates are not provided, the peer will not be able to validate the certificate, and the handshake will fail.

Does anyone else read that as we need to pull out the CA and have it as a separate item?

aggietallboy commented 5 months ago

I was reading some additional stuff over on Intel's forum, and they suggested potential problems with the SHA1 Comodo root cert : https://community.intel.com/t5/Intel-vPro-Platform/Your-SHA-1-Root-CA-AMT-provisioning-certificate-will-not-work/td-p/1449260

Going to try to get Sectigo to reissue my cert, and we'll see if we can do anything there.

aggietallboy commented 5 months ago

No luck on having them reissue the cert, their last advice was to build a pfx without the SHA-1 cert in it - I tried, but then MC complains that it can't find the root cert.

aggietallboy commented 5 months ago

So, I was looking at anything else with AMT issues, and came across 2805

What I noticed, and may be the root of our problem, we had the same hash defined on our amtacm config!

I used a sectigo cert, he used a godaddy cert, but both of us are getting a final spot in the chain with the comodo root cert.

amtacm --- Activation Log --- File : amtactivation.log --- Activation Certificate 1 --- Name : test.net SHA1 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX SHA256: d7a7a0fb5d7e2731d771e9484ebcdef71d5f0c3e0a2948782bc83ee0ea699ef4 --- Activation Certificate 2 --- Name : * SHA1 : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX SHA256: 53f55fc1c11c91a7a38f374e2326e58175e5ccf9723d83667271d125097ad104

And then mine:

--- Activation Log ---
File  : amtactivation.log
--- Activation Certificate 1 ---
Name  : <mydomain.com>
SHA1  : d1eb23a46d17d68fd92564c2f1f1601764d8e349
SHA256: d7a7a0fb5d7e2731d771e9484ebcdef71d5f0c3e0a2948782bc83ee0ea699ef4
--- Activation Certificate 2 ---
Name  : *
SHA1  : <REDACTED>
SHA256: <REDACTED>

I think we're on to something about our cert chain processing.

I noted that the setup.bin certificate matched my Activation Certificate 2. I tried adding the SHA1 hash here, but no positive effect.

When I pulled the hash on my cert, it did not match what I'm seeing in the amtacm output, and, again, with the match to his output, and the fact that both of us have the Comodo CA -- I'm leaning to we're not parsing the chain correctly here.

aggietallboy commented 5 months ago

@Ylianst @si458 Just trying to bubble this up to y'all. If we're trying to present the root cert, of course our key won't decode it. will try to dig in the AMT server some more to see if I can see where it's getting built.