Problems issuing Chromebook certificates through SCEP provider

JMyklebust commented 2 years ago

Subject of the issue

We are trying to implement step-ca as the certificate provider for client certifcates to Chromebook devices in our organization. Google Workspace uses SCEP protocol to request certificates. Here is some of Google's documentation on this, they mostly refer to Windows ADCS as SCEP service: https://support.google.com/chrome/a/answer/11053129?hl=en&ref_topic=6330253 https://support.google.com/chrome/a/answer/11338941?hl=en

But there is no technical limitation from Google's side on which CA service to use. I have built a test enviroment that can issue SCEP certficates and testing the basic issuing using the application "sscep".

These parts are all working using the sscep client software, however when we test with a Chromebook we get the error: scep post request failed: pkcs7: Message digest mismatch

Your environment

OS - Windows Server 2016
Version - step-ca 0.18.2

Steps to reproduce

I've attached a Powershell script that i use to build the entire CA (and optionally install as a service using "shawl"). I've also attached a zip containing the exact executables i use for reference. TestEXE.zip WindowsCA.ps1.txt

But specifically these are used in addtion to step 0.18.2: Shawl - https://github.com/mtkennerly/shawl - Pre-compiled version 1.1.0 ( not needed if not installing as a service) SSCEP - https://github.com/certnanny/sscep - Self-compiled binary for Windows based on v0.10.0 OpenSSL - https://slproweb.com/products/Win32OpenSSL.html - using win64 v3.0.2

For testing with Google Workspace and Chromebook, we have set up a SCEP profile with these settings (omitted setting are blank/default):

Profile Name: SCEPTest
Subject name format:
- Common name: ${DEVICE_SERIAL_NUMBER}
SAN: None
SCEP Server URL: <hostname:port>/scep/scepca
Challenge type: Static
- <challenge password>
Certificate Authority: <Intermediate CA certificate>
Device plattforms: Chromebook

Obviously you need a Google Workspace and Chromebook to set this up. The Google Certificate Connection got installed using a local user on the server (because the installer required it), and then the service was changed to use "LocalService" through Windows Services.

Expected behaviour

Expected that the Chromebook would complete the certificate request. Request is passed to step-ca with Google Certificate Connector as a proxy relay (certificate request flow is decribed in Google's documentation "Configure SCEP with ADCS for Chromebooks" as Appendix A). But short version is that Chromebook sends request to Google Workspace, then the Certificate Connector polls Workspace for any pending request. If request are pending they get pulled by the connector for signing and are pushed back to Workspace after signing is complete, afterwards Workspace push the certficate to the Chromebook.

Actual behaviour

Certificate request goes through, but final signing from step-ca seems to fail with error scep post request failed: pkcs7: Message digest mismatch.

Additional context

We managed to extract one of the failed requests. Since the request itelf only contains test-data there is no issue with sharing it here:

Of note a difference with the test performed via OpenSSL made request and the request from the Chromebook is this (after decoding the request via OpenSSL):

        Attributes:
            challengePassword        :secret1234
            1.2.840.113549.1.9.25.3  :unable to print attribute
            Requested Extensions:
                1.3.6.1.4.1.311.20.2:
                    ..

hslatman commented 2 years ago

Hi @JMyklebust,

Thanks for your report. That should help in resolving this issue. I'll have a look at it soon.

I currently don't have a Chromebook, but I should be able to run it in a VM.

Is it possible for you to capture the actual SCEP requests and responses off the wire (e.g. using Wireshark)? Or does Google Workspace require HTTPS, complicating that? A copy of your full CA config could also help, if you're still in the testing phase and no confidential data is in there. Otherwise I'll try to stand up one and try it with Workspace myself.

JMyklebust commented 2 years ago

Thank you for looking into this. The actual full payloads I'm a bit reluctant to share publicly since they do contain some actual hostnames (the Google Connector logs into Windows Application Log). The logged payload is base64 encoded so trying to censor the server hostname might accidentaly remove relevant data as well.

The CA config I can share though:

{
    "root":  "D:/Scripts/PKITest/.step/certs/root_ca.crt",
    "federatedRoots":  null,
    "crt":  "D:/Scripts/PKITest/.step/certs/intermediate_ca.crt",
    "key":  "D:/Scripts/PKITest/.step/secrets/intermediate_ca_key",
    "address":  ":9000",
    "insecureAddress":  ":9001",
    "dnsNames":  [
                     "<REMOVED>"
                 ],
    "logger":  {
                   "format":  "text"
               },
    "db":  {
               "type":  "badgerv2",
               "dataSource":  "D:/Scripts/PKITest/.step/db",
               "badgerFileLoadingMode":  "FileIO"
           },
    "authority":  {
                      "provisioners":  [
                                           {
                                               "type":  "SCEP",
                                               "name":  "scepca",
                                               "forceCN":  true,
                                               "challenge":  "secret1234",
                                               "EncryptionAlgorithmIdentifier":  2,
                                               "encryptionAlgorithm":  3,
                                               "claims":  {
                                                              "minTLSCertDuration":  "24h",
                                                              "maxTLSCertDuration":  "9000h",
                                                              "defaultTLSCertDuration":  "4320h",
                                                              "disableRenewal":  false
                                                          },
                                               "options":  {
                                                               "x509":  {
                                                                            "templateFile":  "D:/Scripts/PKITest/.step/templates/ChromeSCEP.tpl"
                                                                        }
                                                           }
                                           }
                                       ]
                  },
    "tls":  {
                "cipherSuites":  [
                                     "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
                                     "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
                                 ],
                "minVersion":  1.2,
                "maxVersion":  1.3,
                "renegotiation":  false
            }
}

The Powershell script actually build the exact configs i use and I don't really modify them afterwards (rather i rebuild them using the script for consistency).

hslatman commented 2 years ago

No problem regarding not sharing all details. I think we can do without it too if I can set up an environment myself. Actually curious what it works like 🙂.

One thing I noticed about your ca.json: you may want to check the encryption algorithm. We map it from encryptionAlgorithmIdentifier, although I think Go will unmarshal it if it starts with a capital E. Currently it should thus be using algorithm 2 in your case.

JMyklebust commented 2 years ago

Corrected encryptionAlgorithmIdentifier, though it din't change much on the issue with the Chromebook. Did allow me to remove encryptionAlgorithm from the config though and keep things working.

The part of encryptionAlgorithmIdentifier should probably be documented a bit better tbh. Defaulting to DES did not work well together with the SSCEP tool (though that might be a weakness in the tool rather than in step-ca implementation). Had to search the the repository here for the relevant points of the code (https://github.com/smallstep/certificates/blob/master/authority/provisioner/scep.go). And then figuring out what the correct value is. Might help if it is noted in the docs for SCEP provider.

hslatman commented 2 years ago

Didn't expect it to matter much either, but better to be safe 🙂

You're right about the docs lacking a bit in this regard. Being able to change the algorithm is still kind of new and updating the docs is on my list.

Haven't had time to look further into it yet.

hslatman commented 2 years ago

@JMyklebust: wanted to let you know that we're getting things in place to be able to test this. We need a test Workplace environment and Chromebook for this. Using a VM may still be an option, but need different hardware for that.

hslatman commented 2 years ago

@JMyklebust: could you try it with a 1024 bits RSA key for the CA (and maybe also the client)? That will probably not be what you want for your actual deployment, but I found this issue with the same error message: https://github.com/micromdm/scep/issues/79#issuecomment-993338179. Since that's the library we use for PKCS7, I'm curious to know if the same solution works for you and then we might be able to narrow it down.

JMyklebust commented 2 years ago

I'll try to get around to test that next week. Though we had to push ahead a bit, we've set up a Windows NDES service so we could go ahead and test the rest of our configuration for the Chromebooks. But would be nice to have it work with step-ca in the end to decouple the Chromebooks from (most) Windows dependencies in our environment. Luckily the SCEP profiles are pretty flexible so I can just add another one for testing step-ca without interfering with the NDES setup.

I did note that while setting up for the NDES service that Google recommends changing one of the IIS settings like this. c:\windows\system32\inetsrv\appcmd.exe set config -section:system.webServer/security/requestFiltering /requestLimits.maxQueryString:"3072" /commit:apphost That they specifically note that you need to up maxQueryString for IIS (I think the original value is 2048) might be a lead in general? Especially if key size does makes a difference.

hslatman commented 2 years ago

I tried to reason about the flows through which this error can be reached. There don't seem to be many so far. Assuming the correct hashing algorithm is (correctly) encoded in the message (and retrieved by Go), the only things that can differ are the encoded digest and the (parsed) content bytes.

There's another PR open for the PKCS7 library that touches the content that is hashed, so it might be related: https://github.com/mozilla-services/pkcs7/pull/64. I haven't been able to create a reproducer myself yet, though, so I can't easily test it with this change. Is it an option for you to share a full example message using a private communication channel?

JMyklebust commented 2 years ago

Haven't gotten around to test with a Chromebook just yet. But since i get this message while testing with sscep I don't think step-ca likes client keys that are less than 2048 anyway: "error when signing new certificate: error generating certificate for order: certificate request RSA key must be at least 2048 bits (256 bytes)" Though apparently Google won't allow less than 2048 client keys either in a SCEP profile, so it does not really matter:

But the CA keys are 1024 (and signs certs with 2048 keys just fine) so still need to do a propert test with a Chromebook (and can do that tomorrow).

hslatman commented 2 years ago

@JMyklebust: step-ca indeed requires 2048 bits by default, but there's a configuration to change that. But I think we don't need to try it; I don't think that one will help here.

Do you mean you haven't tested with an actual Chromebook so far? Or just this week? If so, how did you do the testing before that? Just with sscep?

Is it an option for you to share a SCEP request in a private channel instead?

JMyklebust commented 2 years ago

I haven't tested using 1024 bits CA keys with a Chromebook yet. But I have tested the initial setup with an actual Chromebook, and that is where I got the error i reported. sscep has been working all along, but I basically just use it as a sort of sanity check that the SCEP provider is working at all.

I'm switch between working from home and from the office, and currently my test Chromebook is at the office. Basically I've found the most reliable way to get the Chromebook to send certificate requests is to reboot it. But that is unfortunately a bit hard to do reliably if you don't have hands-on. Working from the office tomorrow so I can test properly then.

And I can probably share a SCEP request in a private channel yeah. I can prep that tomorrow as well.

hslatman commented 2 years ago

I haven't tested using 1024 bits CA keys with a Chromebook yet. But I have tested the initial setup with an actual Chromebook, and that is where I got the error i reported. sscep has been working all along, but I basically just use it as a sort of sanity check that the SCEP provider is working at all.

I'm switch between working from home and from the office, and currently my test Chromebook is at the office. Basically I've found the most reliable way to get the Chromebook to send certificate requests is to reboot it. But that is unfortunately a bit hard to do reliably if you don't have hands-on. Working from the office tomorrow so I can test properly then.

Ah, gotcha. I was trying to make sure I understood you correctly 🙂

And I can probably share a SCEP request in a private channel yeah. I can prep that tomorrow as well.

That would be great!

JMyklebust commented 2 years ago

I've just tested and it indeed works if the CA-keys are 1024 bit. So the issue seems to be payload size.

I also found out that the SCEP profile accepts localhost as SCEP server address (since the Google Certificate Collector and step-ca are running on the same server and I'm using http). This means I can share it here. So here are the payloads that actually works (this is taken from the logs of the certificate connector):

The initial pull of the request:
[com.google.mdm.certificate.agent.RequestSubscriber]: Received pull response {"receivedMessages":[{"ackId":"RFAGFixdRkhRNxkIaFEOT14jPzUgKEUSCAgUBXx9cF1JdV1Zc2hRDRlyfWB9alIbVQcXUncNURsHaE5tdR_viOrRS0NVbl8SAQVFUX9aXxkJblVZcC-U7pK-2cK9XUAvObzAoN1pe7mTycltZiM9XxJLLD5-KTdFQV5AEkwiAkRJUytDCypYEU4EISE-MD5FUw","message":{"attributes":{"pubsub.googleapis.com/typed-publisher/message-type/protobuf":"ccc.hosted.devicemanagement.proto.emm.certificate.enrollment.Request","pubsub.googleapis.com/typed-publisher/message-type/mime-type/":"application/x-protobuf"},"data":"CpgJCiRjNjIyODM1Ny1lYjllLTRlZDktYmU3ZS0yNTI5ZjJmMWQ1YzYSmAFDZ2xETURNM2EyTjVOeklTRlRFeE5UYzBNVGs0TWpZNE5EWTFORE00TWpJek1Sb2tZell5TWpnek5UY3RaV0k1WlMwMFpXUTVMV0psTjJVdE1qVXlPV1l5WmpGa05XTTJJQWNxSkdabU16UmlZbVkzTFRWaU1qRXROR1V5TlMxaE9XSTRMVFV5TWpoalpqYzVaVGxrTkVnQhqxB01JSUN0ekNDQVo4Q0FRQXdFekVSTUE4R0ExVUVBd3dJVkdWemRFTmxjblF3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUUNURTNSbW14NS9qbjExM3ZBSFVTbjl1ZlRGcFJzTC9BNmx0eEJRN0lpbU1Eam5TV0tIN3lyU08xLzNHVVBXK2VPdAorbVBMVVpIVDhWb0pIczJkWjdrK1VibFVkalkxSEdPdUIwMzNLVnJRUFFxc1J6TW1sbElzTnJEMUJ4d3l2ck5qb1lwcEtPdWJNTmhFCmxuU29XK2lkZHlIS3FPT3NkTFF1alVXckVNdlJuUHM1T3VVS0cxTmljU0lLRjJhSEY2UUUrZDFiT0E1d1NqZTVYK3hmTkxSNnNkTjYKV3YxTmFVQmdBUklXa0tQdmw0TW05ZDM0MlBFWGExZ1h4SnV0ZjRHUm9oWkh4bnd4VURMV3J5NTRkVTkwelpiV1FxaWUxbE9lbG5YZAp6S1llZG1TV1ZLOUp6UjNpdjNBYzg4cElUOW53Ry9nSHUxek9wSmVxNFZHSkFHRnhBZ01CQUFHZ1h6QVpCZ2txaGtpRzl3MEJDUWN4CkRCTUtjMlZqY21WME1USXpOREFnQmdrcWhraUc5dzBCQ1E0eEV6QVJNQThHQ1NzR0FRUUJnamNVQWdRQ0RBQXdJQVlLS29aSWh2Y04KQVFrWkF6RVNCQkJFdlNZZHJ6TzQ3cFNvRFhIV3FmdjRNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUE5VFIwV3FscHVGb0ZHeXpkbQpGYllCNTJIK21RK1Q5YTBDTS84N2x5RHM5VUdodVZtUC9qc2h0YXUrQitvVUhTRFhYelo3SUxpVmRuaDBVNmlSS2g3RjdZMFdMVk9wCnRyTFNnRFFDaGpDcmNhYy9pQ2IrMXBpUDJieUxMYlNyUVZocjhxcVRnTXJTdUZzRkFGK1FsckdUQ2F4VVBHVkhLRkk3OTdFUG8zcUgKSmtRa3pBRU5mNit0T1h3RTI5N2JDZko3RjdBZEFmdUZVb3pwcTBLTkNwN2NuVUU3NU9kUDNkZmwwNzNpNVg0Y0MxOUx0YVIvMlFnZwp6ZURSMkZXRGRzS1g5RzA5WEJoM1dBVUw4cmFleHErb3VyMW9ZZGFQb2dpQVZrWkg5d0RMN3BjYkVDQjRkeDFXaW83UFNCYjhJcG5UCjBFZUdVV0Z5SUNJaHVKeEt3TmYyCiIhaHR0cDovL2xvY2FsaG9zdDo5MDAxL3NjZXAvc2NlcGNh","messageId":"4510702154246955","publishTime":"2022-05-03T07:32:27.782Z"}}]}

I think this is processing the request:
[com.google.mdm.certificate.agent.RequestReceiver]: Received pubsub payload: {"attributes":{"pubsub.googleapis.com/typed-publisher/message-type/protobuf":"ccc.hosted.devicemanagement.proto.emm.certificate.enrollment.Request","pubsub.googleapis.com/typed-publisher/message-type/mime-type/":"application/x-protobuf"},"data":"CpgJCiRjNjIyODM1Ny1lYjllLTRlZDktYmU3ZS0yNTI5ZjJmMWQ1YzYSmAFDZ2xETURNM2EyTjVOeklTRlRFeE5UYzBNVGs0TWpZNE5EWTFORE00TWpJek1Sb2tZell5TWpnek5UY3RaV0k1WlMwMFpXUTVMV0psTjJVdE1qVXlPV1l5WmpGa05XTTJJQWNxSkdabU16UmlZbVkzTFRWaU1qRXROR1V5TlMxaE9XSTRMVFV5TWpoalpqYzVaVGxrTkVnQhqxB01JSUN0ekNDQVo4Q0FRQXdFekVSTUE4R0ExVUVBd3dJVkdWemRFTmxjblF3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUUNURTNSbW14NS9qbjExM3ZBSFVTbjl1ZlRGcFJzTC9BNmx0eEJRN0lpbU1Eam5TV0tIN3lyU08xLzNHVVBXK2VPdAorbVBMVVpIVDhWb0pIczJkWjdrK1VibFVkalkxSEdPdUIwMzNLVnJRUFFxc1J6TW1sbElzTnJEMUJ4d3l2ck5qb1lwcEtPdWJNTmhFCmxuU29XK2lkZHlIS3FPT3NkTFF1alVXckVNdlJuUHM1T3VVS0cxTmljU0lLRjJhSEY2UUUrZDFiT0E1d1NqZTVYK3hmTkxSNnNkTjYKV3YxTmFVQmdBUklXa0tQdmw0TW05ZDM0MlBFWGExZ1h4SnV0ZjRHUm9oWkh4bnd4VURMV3J5NTRkVTkwelpiV1FxaWUxbE9lbG5YZAp6S1llZG1TV1ZLOUp6UjNpdjNBYzg4cElUOW53Ry9nSHUxek9wSmVxNFZHSkFHRnhBZ01CQUFHZ1h6QVpCZ2txaGtpRzl3MEJDUWN4CkRCTUtjMlZqY21WME1USXpOREFnQmdrcWhraUc5dzBCQ1E0eEV6QVJNQThHQ1NzR0FRUUJnamNVQWdRQ0RBQXdJQVlLS29aSWh2Y04KQVFrWkF6RVNCQkJFdlNZZHJ6TzQ3cFNvRFhIV3FmdjRNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUE5VFIwV3FscHVGb0ZHeXpkbQpGYllCNTJIK21RK1Q5YTBDTS84N2x5RHM5VUdodVZtUC9qc2h0YXUrQitvVUhTRFhYelo3SUxpVmRuaDBVNmlSS2g3RjdZMFdMVk9wCnRyTFNnRFFDaGpDcmNhYy9pQ2IrMXBpUDJieUxMYlNyUVZocjhxcVRnTXJTdUZzRkFGK1FsckdUQ2F4VVBHVkhLRkk3OTdFUG8zcUgKSmtRa3pBRU5mNit0T1h3RTI5N2JDZko3RjdBZEFmdUZVb3pwcTBLTkNwN2NuVUU3NU9kUDNkZmwwNzNpNVg0Y0MxOUx0YVIvMlFnZwp6ZURSMkZXRGRzS1g5RzA5WEJoM1dBVUw4cmFleHErb3VyMW9ZZGFQb2dpQVZrWkg5d0RMN3BjYkVDQjRkeDFXaW83UFNCYjhJcG5UCjBFZUdVV0Z5SUNJaHVKeEt3TmYyCiIhaHR0cDovL2xvY2FsaG9zdDo5MDAxL3NjZXAvc2NlcGNh","messageId":"4510702154246955","publishTime":"2022-05-03T07:32:27.782Z"}

The signed certificate:
[com.google.mdm.certificate.agent.EnrollDeviceRequestHandler]: Received certificate -----BEGIN CERTIFICATE-----MIIC2jCCAkOgAwIBAgIQKAXWevhCAML/QvIrxTgfGjANBgkqhkiG9w0BAQsFADAfMR0wGwYDVQQDExRUZXN0UEtJIEludGVybWVkaWF0ZTAeFw0yMjA1MDMwNzMxMzBaFw0yMjEwMzAwNzMyMzBaMBMxETAPBgNVBAMTCFRlc3RDZXJ0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkxN0Zpsef459dd7wB1Ep/bn0xaUbC/wOpbcQUOyIpjA450lih+8q0jtf9xlD1vnjrfpjy1GR0/FaCR7NnWe5PlG5VHY2NRxjrgdN9yla0D0KrEczJpZSLDaw9QccMr6zY6GKaSjrmzDYRJZ0qFvonXchyqjjrHS0Lo1FqxDL0Zz7OTrlChtTYnEiChdmhxekBPndWzgOcEo3uV/sXzS0erHTelr9TWlAYAESFpCj75eDJvXd+NjxF2tYF8SbrX+BkaIWR8Z8MVAy1q8ueHVPdM2W1kKontZTnpZ13cymHnZkllSvSc0d4r9wHPPKSE/Z8Bv4B7tczqSXquFRiQBhcQIDAQABo4GeMIGbMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDAjAdBgNVHQ4EFgQULgtccAZNQekvyc0+EGvbqMxVVkAwHwYDVR0jBBgwFoAUFBm/q3TZuLjm11u8vxT/PFq7h0QwEwYDVR0RBAwwCoIIVGVzdENlcnQwHwYMKwYBBAGCpGTGKEABBA8wDQIBCgQGc2NlcGNhBAAwDQYJKoZIhvcNAQELBQADgYEAGv5GHWRg7qu7CEr3kAWgEh1V/ktSCl23KYIQt4nHQcv1EtLrMQhNAbaAYIRcotnPWyRju4xmsO0H+3zpbOmBIeHmwME8pfviXuTI0FLe73iPjFg8v9EK62sd3J7F6N0Kf6JxU4HTWywA1tnjm82WzbvhbrfaPLxzNrYi5PcqaGA=-----END CERTIFICATE----- for device ID c6228357-eb9e-4ed9-be7e-2529f2f1d5c6

So this is for the successfull request using 1024bit CA-keys.

This is a failed request using 2048bit CA-keys.

[com.google.mdm.certificate.agent.RequestSubscriber]: Received pull response {"receivedMessages":[{"ackId":"RVNEUAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRIICBQFfH1xXlh1W1oaB1ENGXJ8aXViUkYIBkxSeFVbEQ16bVxtrt7-ukRfQXFvWxMJAkdTeltcGAtqWlldwrKG1dbeqUJwYSuojbbwSH_vkcc8ZiA9XxJLLD5-KTdFQV5AEkwiAkRJUytDCypYEU4EISE-MD4","message":{"attributes":{"pubsub.googleapis.com/typed-publisher/message-type/protobuf":"ccc.hosted.devicemanagement.proto.emm.certificate.enrollment.Request","pubsub.googleapis.com/typed-publisher/message-type/mime-type/":"application/x-protobuf"},"data":"CpgJCiRjNjIyODM1Ny1lYjllLTRlZDktYmU3ZS0yNTI5ZjJmMWQ1YzYSmAFDZ2xETURNM2EyTjVOeklTRlRFeE5UYzBNVGs0TWpZNE5EWTFORE00TWpJek1Sb2tZell5TWpnek5UY3RaV0k1WlMwMFpXUTVMV0psTjJVdE1qVXlPV1l5WmpGa05XTTJJQWNxSkRKbVl6QTNaRFExTFRrNU56UXRORGd4TWkwNE1UUmhMV0kwTmpSbFpERXhOamhqT0VnQhqxB01JSUN0ekNDQVo4Q0FRQXdFekVSTUE4R0ExVUVBd3dJVkdWemRFTmxjblF3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUUNibDc3c1VkaTA5aUFCalo1NTF0WG9STlZ0RjZPWTN5cDlmN1o4NVdpYWc1TmIxQzcrK1k5UXlCdE05WUJkZHFvZQoxMllWU3hSMzdMRlFtMTgvcnRESEdINEM4WVloS1EwODJQa3JGc1VRbW1DZnFpN1hZSDJTVGZMKzVJSmowNWlnNkcvbjErL1N1OUswCnIvckNXQk94Ujl3VG5kSU91RzJ5bTV0SHhZSnFuUHdwdWFqL0lHWDJCREF1UGVnbHRvd3Azbk5BYzdqSGMzL0kxblhxdW1TL2FBUnIKZ2hVS2ppcEw0STJ6Z2YrbVVydmJQZjBYTnVIRWdIZjk0QWtvbjZURUdSL1lhSTd3eC9UNlBGY1BuNDdTZkNrcUY1VFBPYmg4Mk1oeApxc1h2cTNDT2V4a2dhWHZERkw5QS93cUVkTzZRc1VOMHJsMUFhNmluWHhIM2MvT1ZBZ01CQUFHZ1h6QVpCZ2txaGtpRzl3MEJDUWN4CkRCTUtjMlZqY21WME1USXpOREFnQmdrcWhraUc5dzBCQ1E0eEV6QVJNQThHQ1NzR0FRUUJnamNVQWdRQ0RBQXdJQVlLS29aSWh2Y04KQVFrWkF6RVNCQkJTN3J4cnlhSmJOa0cxQldLSjBGYmVNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUFmbjg3OEtCQ2luT1pMTjdyYQpEVTdPNjNGUzVYVG5zOEp2a3gwQmk3YTFTY3lINzA4NHU2S054b1BSajlpcnZrTC9PV0liNU9MNDR2NURCenlHa25XdlkxdnUwSjJPCkphRkdINThPaFFtd1l6eVd5UGppQVBwSDZMa1A2RFB1dmFuWllqNDByYUhmbUlRZGFVcFdTY0UxNUFtOXlnbzg5N2FDNW9EeU5QRUMKQkdORUR5aDI2WkFMUGZubENsaWVzS1IvN2JBQ0tpNzJDQUE4dVpjK0RDVkRpZjQ0ejZOcVF3MmUrNUtFbDV6b2lWc0pienYxWWNnYwozazJyN1hGTHZaZXdpZFJ5L0F5QjVCZDNGdnpQSWkwbWNUWC9jN3VTSmc1clBFaGRSN0l4R2Vld3AzY2EvUHgwRkgzTEV5elNrNWNMCjVQR0JjQWhzWWdGdkpJMkNQdFc3CiIhaHR0cDovL2xvY2FsaG9zdDo5MDAxL3NjZXAvc2NlcGNh","messageId":"4510802044736265","publishTime":"2022-05-03T07:53:51.754Z"}}]}

[com.google.mdm.certificate.agent.RequestReceiver]: Received pubsub payload: {"attributes":{"pubsub.googleapis.com/typed-publisher/message-type/protobuf":"ccc.hosted.devicemanagement.proto.emm.certificate.enrollment.Request","pubsub.googleapis.com/typed-publisher/message-type/mime-type/":"application/x-protobuf"},"data":"CpgJCiRjNjIyODM1Ny1lYjllLTRlZDktYmU3ZS0yNTI5ZjJmMWQ1YzYSmAFDZ2xETURNM2EyTjVOeklTRlRFeE5UYzBNVGs0TWpZNE5EWTFORE00TWpJek1Sb2tZell5TWpnek5UY3RaV0k1WlMwMFpXUTVMV0psTjJVdE1qVXlPV1l5WmpGa05XTTJJQWNxSkRKbVl6QTNaRFExTFRrNU56UXRORGd4TWkwNE1UUmhMV0kwTmpSbFpERXhOamhqT0VnQhqxB01JSUN0ekNDQVo4Q0FRQXdFekVSTUE4R0ExVUVBd3dJVkdWemRFTmxjblF3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUUNibDc3c1VkaTA5aUFCalo1NTF0WG9STlZ0RjZPWTN5cDlmN1o4NVdpYWc1TmIxQzcrK1k5UXlCdE05WUJkZHFvZQoxMllWU3hSMzdMRlFtMTgvcnRESEdINEM4WVloS1EwODJQa3JGc1VRbW1DZnFpN1hZSDJTVGZMKzVJSmowNWlnNkcvbjErL1N1OUswCnIvckNXQk94Ujl3VG5kSU91RzJ5bTV0SHhZSnFuUHdwdWFqL0lHWDJCREF1UGVnbHRvd3Azbk5BYzdqSGMzL0kxblhxdW1TL2FBUnIKZ2hVS2ppcEw0STJ6Z2YrbVVydmJQZjBYTnVIRWdIZjk0QWtvbjZURUdSL1lhSTd3eC9UNlBGY1BuNDdTZkNrcUY1VFBPYmg4Mk1oeApxc1h2cTNDT2V4a2dhWHZERkw5QS93cUVkTzZRc1VOMHJsMUFhNmluWHhIM2MvT1ZBZ01CQUFHZ1h6QVpCZ2txaGtpRzl3MEJDUWN4CkRCTUtjMlZqY21WME1USXpOREFnQmdrcWhraUc5dzBCQ1E0eEV6QVJNQThHQ1NzR0FRUUJnamNVQWdRQ0RBQXdJQVlLS29aSWh2Y04KQVFrWkF6RVNCQkJTN3J4cnlhSmJOa0cxQldLSjBGYmVNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUFmbjg3OEtCQ2luT1pMTjdyYQpEVTdPNjNGUzVYVG5zOEp2a3gwQmk3YTFTY3lINzA4NHU2S054b1BSajlpcnZrTC9PV0liNU9MNDR2NURCenlHa25XdlkxdnUwSjJPCkphRkdINThPaFFtd1l6eVd5UGppQVBwSDZMa1A2RFB1dmFuWllqNDByYUhmbUlRZGFVcFdTY0UxNUFtOXlnbzg5N2FDNW9EeU5QRUMKQkdORUR5aDI2WkFMUGZubENsaWVzS1IvN2JBQ0tpNzJDQUE4dVpjK0RDVkRpZjQ0ejZOcVF3MmUrNUtFbDV6b2lWc0pienYxWWNnYwozazJyN1hGTHZaZXdpZFJ5L0F5QjVCZDNGdnpQSWkwbWNUWC9jN3VTSmc1clBFaGRSN0l4R2Vld3AzY2EvUHgwRkgzTEV5elNrNWNMCjVQR0JjQWhzWWdGdkpJMkNQdFc3CiIhaHR0cDovL2xvY2FsaG9zdDo5MDAxL3NjZXAvc2NlcGNh","messageId":"4510802044736265","publishTime":"2022-05-03T07:53:51.754Z"}

The returned error from step-ca to the certificate connector.

[com.google.mdm.certificate.agent.EnrollDeviceRequestHandler]: Unable to read from PEM string: org.jscep.transaction.TransactionException: org.jscep.transport.TransportException: 500 Internal Server Error
    at org.jscep.transaction.Transaction.send(Unknown Source)
    at org.jscep.transaction.EnrollmentTransaction.send(Unknown Source)
    at org.jscep.client.Client.send(Unknown Source)
    at org.jscep.client.Client.enrol(Unknown Source)
    at org.jscep.client.Client.<unknown>(Unknown Source)
    at com.google.mdm.certificate.agent.EnrollDeviceRequestHandler.execute(Unknown Source)
    at com.google.mdm.certificate.agent.RequestReceiver.receiveMessage(Unknown Source)
    at com.google.mdm.certificate.agent.RequestSubscriber.pull(Unknown Source)
    at com.google.mdm.certificate.agent.RequestSubscriber.<unknown>(Unknown Source)
    at com.google.mdm.certificate.agent.RequestSubscriber$$Lambda$25.run(Unknown Source)
    at com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: org.jscep.transport.TransportException: 500 Internal Server Error
    at org.jscep.transport.UrlConnectionPostTransport.sendRequest(Unknown Source)
    ... 18 more

This is the log line from step-ca: time="2022-05-03T09:54:02+02:00" level=error duration=0s duration-ns=0 error="scep post request failed: pkcs7: Message digest mismatch\n\tExpected: BCD6C3FA9F996EAE7F05D64690914FEE8452FF391341F65378385A10C78126AC\n\tActual : 5A607D497FED984E2BCDCFF7CCE3EBE695DB28EC41E64A5152A7B0513354CF33" fields.time="2022-05-03T09:54:02+02:00" method=POST name=ca path="/scep/scepca?operation=PKIOperation" protocol=HTTP/1.1 referer= remote-address=127.0.0.1 request-id=c9odv6k20edg954d0ll0 size=227 status=500 user-agent=Java/1.8.0_181 user-id=

hslatman commented 2 years ago

@JMyklebust: thank you for these logs 😄

So, based on your observation with the 1024 bit key it seems we're on the right track with this issue: https://github.com/micromdm/scep/issues/79. I saw from the logs that the user agent is Java/1.8.0_181, indicating that https://github.com/jscep/jscep may be in use under the hood, which is also mentioned in that issue.

I tried extracting some info from the logs, but those don't seem to contain the actual SCEP requests performed against step-ca, so I can't extract the bytes that are hashed and signed. Could you capture the actual request using Wireshark (or something similar) and share that?

JMyklebust commented 2 years ago

Luckily this is a fresh server so the loopback interface is pretty quiet. Here is a capture of the traffic and the a complete session log for step-ca.

Log:

badger 2022/05/03 12:46:47 INFO: All 1 tables opened in 1ms
badger 2022/05/03 12:46:47 INFO: Replaying file id: 0 at offset: 3645
badger 2022/05/03 12:46:47 INFO: Replay took: 0s
2022/05/03 12:46:47 Serving HTTP on :9001 ...
2022/05/03 12:46:47 Serving HTTPS on :9000 ...
time="2022-05-03T12:54:33+02:00" level=info duration=0s duration-ns=0 fields.time="2022-05-03T12:54:33+02:00" method=GET name=ca path="/scep/scepca?operation=GetCACaps&message=" protocol=HTTP/1.1 referer= remote-address=127.0.0.1 request-id=c9ogjqc20edhuc0lnkjg size=66 status=200 user-agent=Java/1.8.0_181 user-id=
time="2022-05-03T12:54:33+02:00" level=info duration=0s duration-ns=0 fields.time="2022-05-03T12:54:33+02:00" method=GET name=ca path="/scep/scepca?operation=GetCACaps&message=" protocol=HTTP/1.1 referer= remote-address=127.0.0.1 request-id=c9ogjqc20edhuc0lnkk0 size=66 status=200 user-agent=Java/1.8.0_181 user-id=
time="2022-05-03T12:54:33+02:00" level=info duration="22.6µs" duration-ns=22600 fields.time="2022-05-03T12:54:33+02:00" method=GET name=ca path="/scep/scepca?operation=GetCACert&message=" protocol=HTTP/1.1 referer= remote-address=127.0.0.1 request-id=c9ogjqc20edhuc0lnkkg size=806 status=200 user-agent=Java/1.8.0_181 user-id=
time="2022-05-03T12:54:33+02:00" level=info duration=0s duration-ns=0 fields.time="2022-05-03T12:54:33+02:00" method=GET name=ca path="/scep/scepca?operation=GetCACaps&message=" protocol=HTTP/1.1 referer= remote-address=127.0.0.1 request-id=c9ogjqc20edhuc0lnkl0 size=66 status=200 user-agent=Java/1.8.0_181 user-id=
time="2022-05-03T12:54:33+02:00" level=info duration="506.3µs" duration-ns=506300 fields.time="2022-05-03T12:54:33+02:00" method=GET name=ca path="/scep/scepca?operation=GetCACert&message=" protocol=HTTP/1.1 referer= remote-address=127.0.0.1 request-id=c9ogjqc20edhuc0lnklg size=806 status=200 user-agent=Java/1.8.0_181 user-id=
time="2022-05-03T12:54:33+02:00" level=info duration=0s duration-ns=0 fields.time="2022-05-03T12:54:33+02:00" method=GET name=ca path="/scep/scepca?operation=GetCACaps&message=" protocol=HTTP/1.1 referer= remote-address=127.0.0.1 request-id=c9ogjqc20edhuc0lnkm0 size=66 status=200 user-agent=Java/1.8.0_181 user-id=
time="2022-05-03T12:54:33+02:00" level=error duration=10.1361ms duration-ns=10136100 error="scep post request failed: pkcs7: Message digest mismatch\n\tExpected: 9D9D15CE7E583FAFB3685C9C8A3C77E0124ABB8FFF9A47A0F3E85F35EB73E7ED\n\tActual  : CDCE935FDD09A452CAD00570FFE06521C1141B34421F007BC04AA8670B783056" fields.time="2022-05-03T12:54:33+02:00" method=POST name=ca path="/scep/scepca?operation=PKIOperation" protocol=HTTP/1.1 referer= remote-address=127.0.0.1 request-id=c9ogjqc20edhuc0lnkmg size=227 status=500 user-agent=Java/1.8.0_181 user-id=

Wireshark trace: RequestTrace.pcapng.txt

Edit: Also the data payload from the certificate connector is base64 (ascii) encoded. So decode that first and then you see the actual payload, though the PEM headers are missing in the encoded text.

hslatman commented 2 years ago

@JMyklebust: I'm getting closer to a solution. I've created a small POC to read your SCEP request. I've changed the part that reads the contents to this:

    if compound.IsCompound {
        rest := compound.Bytes
        for len(rest) > 0 {
            if rest, err = asn1.Unmarshal(rest, &compound); err != nil {
                return nil, err
            }
            // Don't allow further constructed types.
            if compound.Class != asn1.ClassUniversal || compound.Tag != asn1.TagOctetString || compound.IsCompound {
                return nil, errors.New("bad class or tag")
            }
            content = append(content, compound.Bytes...)
        }
    } else {
        content = compound.Bytes
    }

That is part of what's in https://github.com/omorsi/pkcs7/pull/2. The SCEP request then can be parsed correctly and results in a PKCSReq (19) type message, which step-ca should be able to process further (e.g. decrypt, read CSR, sign the cert, etc).

I'll need to see how we want to fix this going forward. Ideally the fix gets upstreamed (soon), but we could use a fork or create our own too. Either way, you should soon be able to use step-ca 😄

JMyklebust commented 2 years ago

That's great news! 😄

hslatman commented 2 years ago

@JMyklebust: I've opened a PR that uses a fork of the https://github.com/mozilla-services/pkcs7 library that is used to parse the contents of the SCEP request: https://github.com/smallstep/certificates/pull/932. I've tested the fork with the patch applied on your example message and that was successful.

You could try if the change works in your environment by building step-ca from source using the herman/pkcs7-patches branch. Some instructions to build step-ca are available here: https://github.com/smallstep/certificates/blob/master/docs/CONTRIBUTING.md#build-a-standard-step-ca. Let me know if this doesn't work for you, so that we can find a solution for that.

JMyklebust commented 2 years ago

Setting up a build enviroment for Go is a bit more work than what i can put into this just at the moment. I mean I had a quick look over it, and it looks fairly straight forward. But there is always some devil in the details that ends up using a lot of time to figure out . 😅

Might be able to find time to build it myself later, but might take a few days at the least.

hslatman commented 2 years ago

@JMyklebust: are you willing to run a custom build provided by me? Another alternative is to see if we can get an RC out soon.

hslatman commented 2 years ago

I've just merged #932 into master, so that the next release of step-ca will contain the patches.

JMyklebust commented 2 years ago

Sorry a lot of stuff keeping me busy. I could probably test a custom build by you, just to verify yes. Though I'm not sure how long off it is for the next proper release of step-ca (like talking days or weeks)?

hslatman commented 2 years ago

@JMyklebust: I'll ask if we have an ETA for a release (candidate). We don't follow a strict release cadence at the moment, so it's hard to say for sure at this time. An RC should be possible within days, though.

hslatman commented 2 years ago

@JMyklebust: we're aiming for a release today/tomorrow

JMyklebust commented 2 years ago

Took me a bit of time to get around to test, but finally had time. I can confirm that issuing certificates to Chromebooks are working as intended now. 😄 Thanks for the help!

hslatman commented 2 years ago

@JMyklebust: that's great to hear! 🥳

Thank you for testing, too! I'm going to add the example message you provided as a positive test vector, so that we can go forward with our SCEP implementation with (more) confidence regarding backwards compatibility 🙂

smallstep / certificates