search

openbmc / bmcweb

A do everything Redfish, KVM, GUI, and DBus webserver for OpenBMC
Apache License 2.0
165 stars 135 forks source link

Bmcweb stop working after CA signed certificate upload #92

Closed rahulmah closed 5 years ago

rahulmah commented 5 years ago

Bmcweb fails after CA signed certificate upload which in created using CSR.

Journal log

root@witherspoon:~# journalctl -f
-- Logs begin at Thu 2019-06-27 11:00:28 UTC. --
Jun 27 11:11:51 witherspoon systemd[1]: Started Assert bmc_booted LED.
Jun 27 11:11:53 witherspoon systemd[1]: obmc-flash-bmc-setenv@pnor\x2df01ebecf\x3d1.service: Succeeded.
Jun 27 11:11:53 witherspoon systemd[1]: Started Set U-Boot environment variable.
Jun 27 11:11:54 witherspoon systemd[1]: obmc-flash-bmc-setenv@pnor\x2de6348a51\x3d0.service: Succeeded.
Jun 27 11:11:54 witherspoon systemd[1]: Started Set U-Boot environment variable.
Jun 27 11:11:54 witherspoon systemd[1]: Startup finished in 8.161s (kernel) + 2min 6.160s (userspace) = 2min 14.321s.
Jun 27 11:12:12 witherspoon systemd[1]: systemd-timedated.service: Succeeded.
Jun 27 11:12:13 witherspoon systemd[1]: systemd-hostnamed.service: Succeeded.
Jun 27 11:12:26 witherspoon ipmid[1448]: Host control timeout hit!
Jun 27 11:12:26 witherspoon ipmid[1448]: Failed to deliver host command
Jun 27 11:14:12 witherspoon phosphor-certificate-manager[1640]: Given Key pair algorithm
Jun 27 11:14:12 witherspoon phosphor-certificate-manager[1640]: Writing private key to file
Jun 27 11:14:12 witherspoon phosphor-certificate-manager[1640]: Writing CSR to file
Jun 27 11:14:12 witherspoon phosphor-certificate-manager[1640]: Removing the existing file
Jun 27 11:23:14 witherspoon systemd[1]: Created slice system-dropbear.slice.
Jun 27 11:23:14 witherspoon systemd[1]: Started SSH Per-Connection Server (9.202.18.97:58405).
Jun 27 11:23:15 witherspoon dropbear[1644]: Child connection from ::ffff:9.202.18.97:58405
Jun 27 11:23:17 witherspoon dropbear[1644]: PAM password auth succeeded for 'root' from ::ffff:9.202.18.97:58405
Jun 27 11:25:05 witherspoon phosphor-certificate-manager[1177]: Certificate install
Jun 27 11:25:05 witherspoon phosphor-certificate-manager[1177]: Certificate loadCert
Jun 27 11:25:05 witherspoon phosphor-certificate-manager[1177]: Certificate verification failed
Jun 27 11:25:05 witherspoon phosphor-certificate-manager[1177]: Private key not present in file
Jun 27 11:25:05 witherspoon phosphor-certificate-manager[1177]: Certificate compareKeys
Jun 27 11:25:05 witherspoon systemd[1]: Starting Cleanup of Temporary Directories...
Jun 27 11:25:05 witherspoon phosphor-certificate-manager[1177]: Certificate loadCert
Jun 27 11:25:05 witherspoon systemd[1]: Reloading Start bmcweb server.
Jun 27 11:25:05 witherspoon systemd[1]: Reloaded Start bmcweb server.
Jun 27 11:25:05 witherspoon bmcweb[1256]: Checking certs in file /etc/ssl/certs/https/server.pem
Jun 27 11:25:05 witherspoon bmcweb[1256]: Found an RSA key
Jun 27 11:25:05 witherspoon bmcweb[1256]: Certificate verification failed. ErrorCode: 20
Jun 27 11:25:05 witherspoon bmcweb[1256]: terminate called after throwing an instance of 'boost::wrapexcept<boost::system::system_error>'
Jun 27 11:25:05 witherspoon bmcweb[1256]:   what():  use_certificate_file: asio.ssl error
Jun 27 11:25:05 witherspoon systemd[1]: bmcweb.service: Main process exited, code=killed, status=6/ABRT
Jun 27 11:25:05 witherspoon systemd[1]: bmcweb.service: Failed with result 'signal'.
Jun 27 11:25:05 witherspoon systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Jun 27 11:25:05 witherspoon systemd[1]: Started Cleanup of Temporary Directories.
Jun 27 11:25:29 witherspoon systemd[1]: Started Start bmcweb server.
Jun 27 11:25:30 witherspoon bmcweb[1656]: Checking certs in file /etc/ssl/certs/https/server.pem
Jun 27 11:25:30 witherspoon bmcweb[1656]: Found an RSA key
Jun 27 11:25:30 witherspoon bmcweb[1656]: Certificate verification failed. ErrorCode: 20
Jun 27 11:25:30 witherspoon bmcweb[1656]: terminate called after throwing an instance of 'boost::wrapexcept<boost::system::system_error>'
Jun 27 11:25:30 witherspoon bmcweb[1656]:   what():  use_certificate_file: asio.ssl error
Jun 27 11:25:30 witherspoon systemd[1]: bmcweb.service: Main process exited, code=killed, status=6/ABRT
Jun 27 11:25:30 witherspoon systemd[1]: bmcweb.service: Failed with result 'signal'.
Jun 27 11:25:30 witherspoon systemd[1]: Started Start bmcweb server.
Jun 27 11:25:31 witherspoon bmcweb[1657]: Checking certs in file /etc/ssl/certs/https/server.pem
Jun 27 11:25:31 witherspoon bmcweb[1657]: Found an RSA key
Jun 27 11:25:31 witherspoon bmcweb[1657]: Certificate verification failed. ErrorCode: 20
Jun 27 11:25:31 witherspoon bmcweb[1657]: terminate called after throwing an instance of 'boost::wrapexcept<boost::system::system_error>'
Jun 27 11:25:31 witherspoon bmcweb[1657]:   what():  use_certificate_file: asio.ssl error
Jun 27 11:25:31 witherspoon systemd[1]: bmcweb.service: Main process exited, code=killed, status=6/ABRT
Jun 27 11:25:31 witherspoon systemd[1]: bmcweb.service: Failed with result 'signal'.
Jun 27 11:25:31 witherspoon systemd[1]: bmcweb.service: Start request repeated too quickly.
Jun 27 11:25:31 witherspoon systemd[1]: bmcweb.service: Failed with result 'signal'.
Jun 27 11:25:31 witherspoon systemd[1]: Failed to start Start bmcweb server.
Jun 27 11:25:31 witherspoon systemd[1]: bmcweb.socket: Failed with result 'service-start-limit-hit'.

Steps to reproduce:

  1. Generate Root CA private key(rootCA.key): openssl genrsa -des3 -out rootCA.key 204

  2. Generate Root CA certificate(rootCA.pem) using: openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem

  3. Now generate CSR request from BMC using json file generate_csr_wsbmc015.json

{
    "City": "Austin",
    "CertificateCollection": {
        "@odata.id": "/redfish/v1/Managers/bmc/NetworkProtocol/HTTPS/Certificates/"
    },
    "CommonName": "<BMC_IP>",
    "ContactPerson":"myname",
    "AlternativeNames":["wsbmc015.aus.stglabs.ibm.com"],
    "ChallengePassword":"",
    "GivenName":"",
    "Initials":"",
    "Country": "US",
    "KeyCurveId":"",
    "KeyUsage":["KeyAgreement"],
    "KeyBitLength": 512,
    "Organization": "IBM",
    "OrganizationalUnit": "ISL",
    "State": "AU",
    "Surname": "",
    "UnstructuredName": ""
}
{
  "CSRString": "-----BEGIN CERTIFICATE REQUEST-----\nMIIBZzCCARECAQEwgasxJTAjBgNVHREMHHdzYm1jMDE1LmF1cy5zdGdsYWJzLmli\nbS5jb20xDzANBgNVBAcMBkF1c3RpbjESMBAGA1UEA  wwJOS4zLjIxLjU1MQ8wDQYD\nVQQpDAZteW5hbWUxCzAJBgNVBAYTAlVTMQ0wCwYEKw4DAgwDUlNBMRUwEwYDVR0P\nDAxLZXlBZ3JlZW1lbnQxDDAKBgNVBA  oMA0lCTTELMAkGA1UECAwCQVUwXDANBgkq\nhkiG9w0BAQEFAANLADBIAkEAwY9eVEdOobpT646Ssn7QmcxLeoWnCIulyP3hKR2f\n4E8Cy3FdO/j3HlrlKxJ  ijB8eBDmdB0zR8CnVUipUcknj4QIDAQABoAAwDQYJKoZI\nhvcNAQELBQADQQBcKCRdSZxqKoH7h4uta27Qchna88ljrJwX3PLqNES5nyCUaacx\ne8Xqddi9  iG7FcnULE9VLzhpr86UTZV4393+s\n-----END CERTIFICATE REQUEST-----\n",
  "CertificateCollection": {
    "@odata.id": "/redfish/v1/Managers/bmc/NetworkProtocol/HTTPS/Certificates/"
  }
}

  1. Convert response into .csr file(device.csr)

    bash-4.1$ cat device.csr
-----BEGIN CERTIFICATE REQUEST-----
MIIBZzCCARECAQEwgasxJTAjBgNVHREMHHdzYm1jMDE1LmF1cy5zdGdsYWJzLmli
bS5jb20xDzANBgNVBAcMBkF1c3RpbjESMBAGA1UEAwwJOS4zLjIxLjU1MQ8wDQYD
VQQpDAZteW5hbWUxCzAJBgNVBAYTAlVTMQ0wCwYEKw4DAgwDUlNBMRUwEwYDVR0P
DAxLZXlBZ3JlZW1lbnQxDDAKBgNVBAoMA0lCTTELMAkGA1UECAwCQVUwXDANBgkq
hkiG9w0BAQEFAANLADBIAkEAwY9eVEdOobpT646Ssn7QmcxLeoWnCIulyP3hKR2f
4E8Cy3FdO/j3HlrlKxJijB8eBDmdB0zR8CnVUipUcknj4QIDAQABoAAwDQYJKoZI
hvcNAQELBQADQQBcKCRdSZxqKoH7h4uta27Qchna88ljrJwX3PLqNES5nyCUaacx
e8Xqddi9iG7FcnULE9VLzhpr86UTZV4393+s
-----END CERTIFICATE REQUEST-----

  2. Now use BMC generated CSR request(device.csr) to generate CA signed certificate(device.crt) openssl x509 -req -in device.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out device.crt -days 500 -sha256

  3. Now create json file(certificate.json) with above generated CA signed certificate file (device.crt).

bash-4.1$ cat certificate.json
{
    "CertificateString": "-----BEGIN CERTIFICATE-----\nMIIC+TCCAeECCQCk+dNJDXfI1jANBgkqhkiG9w0BAQsFADCBmDELMAkGA1UEBhMC\nSU4xDjAMBgNVBAgMBURFTEhJMQ4wDAYDVQQHD  AVERUxISTEeMBwGA1UECgwVQ0VS\nVElGSUNBVEUgQVVUSE9SSVRZMQswCQYDVQQLDAJJVDEeMBwGA1UEAwwVRGF0YSBD\nZW50ZXIgT3ZlcmxvcmRzMRwwGg  YJKoZIhvcNAQkBFg1ub25lQG5vbmUuY29tMB4X\nDTE5MDYyNzExMTczNloXDTIwMTEwODExMTczNlowgasxJTAjBgNVHREMHHdzYm1j\nMDE1LmF1cy5zdGd  sYWJzLmlibS5jb20xDzANBgNVBAcMBkF1c3RpbjESMBAGA1UE\nAwwJOS4zLjIxLjU1MQ8wDQYDVQQpDAZteW5hbWUxCzAJBgNVBAYTAlVTMQ0wCwYE\nKw4D  AgwDUlNBMRUwEwYDVR0PDAxLZXlBZ3JlZW1lbnQxDDAKBgNVBAoMA0lCTTEL\nMAkGA1UECAwCQVUwXDANBgkqhkiG9w0BAQEFAANLADBIAkEAwY9eVEdOobp  T646S\nsn7QmcxLeoWnCIulyP3hKR2f4E8Cy3FdO/j3HlrlKxJijB8eBDmdB0zR8CnVUipU\ncknj4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQAcYmkbcznF  fOm9bDuIeXHzNSus\nEwOhAberTXWvPMtjbDTmbVH5dRPU+DmgS+LEl2jhYC414R89EUApjrXmk1PzlBrN\nXEnBf9+OHOHOH7H4AIni3diw9PRzEdW5ENHUi  OIVoq7LxWP+RknSHGl8AfOghX/3\n6eRgtpIp+fTYwJkGdZaKb9cI5XXk0Eh1cZZ3W43PNsKbuv1BGLGjJVRRaswF9nb1\ng2M4iZLtVXltdkyHW/Z6TUWvG+  9+TYuKingixv0toaWyRGexjC1CeRORGhyYW8Dz\niGipRCWmVo97MC5sWtQjVAshB1TY6rUqipxzW9SqyjplBD+AHySY/IqGM+wU\n-----END CERTIFICATE-----\n",
    "CertificateType": "PEM",
    "CertificateUri":
    {
        "@odata.id": "/redfish/v1/Managers/bmc/NetworkProtocol/HTTPS/Certificates/1"
    }
}
  1. Replace server certificate using above json file with CA signed certificate details(certificate.json)
bash-4.1$ curl -c cjar -b cjar -k -H "X-Auth-Token: $bmc_token" -X POST https://${BMC_IP}/redfish/v1/CertificateService/Actions/CertificateService.ReplaceCertificate/ -d @certificate.json
{
  "@odata.context": "/redfish/v1/$metadata#Certificate.Certificate",
  "@odata.id": "/redfish/v1/Managers/bmc/NetworkProtocol/HTTPS/Certificates/1",
  "@odata.type": "#Certificate.v1_0_0.Certificate",
  "CertificateString": "-----BEGIN CERTIFICATE-----\nMIIC+TCCAeECCQCk+dNJDXfI1jANBgkqhkiG9w0BAQsFADCBmDELMAkGA1UEBhMC\nSU4xDjAMBgNVBAgMBURFTEhJMQ4wDAYDVQQHD  AVERUxISTEeMBwGA1UECgwVQ0VS\nVElGSUNBVEUgQVVUSE9SSVRZMQswCQYDVQQLDAJJVDEeMBwGA1UEAwwVRGF0YSBD\nZW50ZXIgT3ZlcmxvcmRzMRwwGg  YJKoZIhvcNAQkBFg1ub25lQG5vbmUuY29tMB4X\nDTE5MDYyNzExMTczNloXDTIwMTEwODExMTczNlowgasxJTAjBgNVHREMHHdzYm1j\nMDE1LmF1cy5zdGd  sYWJzLmlibS5jb20xDzANBgNVBAcMBkF1c3RpbjESMBAGA1UE\nAwwJOS4zLjIxLjU1MQ8wDQYDVQQpDAZteW5hbWUxCzAJBgNVBAYTAlVTMQ0wCwYE\nKw4D  AgwDUlNBMRUwEwYDVR0PDAxLZXlBZ3JlZW1lbnQxDDAKBgNVBAoMA0lCTTEL\nMAkGA1UECAwCQVUwXDANBgkqhkiG9w0BAQEFAANLADBIAkEAwY9eVEdOobp  T646S\nsn7QmcxLeoWnCIulyP3hKR2f4E8Cy3FdO/j3HlrlKxJijB8eBDmdB0zR8CnVUipU\ncknj4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQAcYmkbcznF  fOm9bDuIeXHzNSus\nEwOhAberTXWvPMtjbDTmbVH5dRPU+DmgS+LEl2jhYC414R89EUApjrXmk1PzlBrN\nXEnBf9+OHOHOH7H4AIni3diw9PRzEdW5ENHUi  OIVoq7LxWP+RknSHGl8AfOghX/3\n6eRgtpIp+fTYwJkGdZaKb9cI5XXk0Eh1cZZ3W43PNsKbuv1BGLGjJVRRaswF9nb1\ng2M4iZLtVXltdkyHW/Z6TUWvG+  9+TYuKingixv0toaWyRGexjC1CeRORGhyYW8Dz\niGipRCWmVo97MC5sWtQjVAshB1TY6rUqipxzW9SqyjplBD+AHySY/IqGM+wU\n-----END CERTIFICATE-----\n",
  "Description": "HTTPS certificate",
  "Id": "1",
  "Issuer": {
    "City": "DELHI",
    "CommonName": "Data Center Overlords",
    "Country": "IN",
    "Organization": "CERTIFICATE AUTHORITY",
    "OrganizationalUnit": "IT",
    "State": "DELHI"
  },
  "KeyUsage": [],
  "Name": "HTTPS certificate",
  "Subject": {
    "City": "Austin",
    "CommonName": "<BMC_IP>",
    "Country": "US",
    "Organization": "IBM",
    "State": "AU"
  },
  "ValidNotAfter": "2020-11-07T23:17:36+00:00",
  "ValidNotBefore": "2019-06-26T23:17:36+00:00"
}
  1. After the replace, we will see bmcweb server failed to start in journal.
Jun 27 11:25:31 witherspoon systemd[1]: bmcweb.service: Failed with result 'signal'.
Jun 27 11:25:31 witherspoon systemd[1]: bmcweb.service: Start request repeated too quickly.
Jun 27 11:25:31 witherspoon systemd[1]: bmcweb.service: Failed with result 'signal'.
Jun 27 11:25:31 witherspoon systemd[1]: Failed to start Start bmcweb server.
Jun 27 11:25:31 witherspoon systemd[1]: bmcweb.socket: Failed with result 'service-start-limit-hit'.
rahulmah commented 5 years ago

This Issue is only seen only when KeyBitLength in CSR request is set to 512. And it is not seen when KeyBitLength is set to 1024, 2048, 4096. After discussing with @ojayanth, opening this issue in bmcweb to further debug.

rahulmah commented 5 years ago

@edtanous : Please look into this issue and let us know your thoughts on this failure.

gtmills commented 5 years ago

@devenrao @ojayanth Can you have a look?

devenrao commented 5 years ago

We are having discussions on IRC whether to support RSA due to the time taken to generate CSR as it is taking time more than the https request timeout of 10 seconds. We may either remove RSA support or support only RSA with key-bit length of 2048.

rahulmah commented 5 years ago

Sure @devenrao

devenrao commented 5 years ago

At present supporting only 2048 key bit length with pre-generated private key, the changes are pushed through https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/19421

rahulmah commented 5 years ago

Thanks for the update @devenrao . Closing the issue.