Sign macOS enrollment and configuration profiles

[zhumo]

Goal

User story
As an IT admin,
I want my the enrollment profile and configuration profiles installed on my macOS hosts to be signed
so that anyone viewing the enrollment profile in System Settings > Profiles won't see a red "Unsigned" message.

Context

• Product designer: @marko-lisica

Changes

Product

• [ ] Changes:
  • Add mdm.apple​_scep​_cert as a trusted certificate root to the host's keychain
  • Profiles uploaded to Fleet should automatically be signed with SCEP certificate that's added to root ^
  • fleetd should add certificate to the root
  • Profiles should be signed in following scenarios:
    • After automatic enrollment
    • After manual enrollment described in docs (when fleetd is installed prior to enrollment profile)
    • After manual enrollment via enrollment profile (when fleetd is installed after enrollment profile)
    • NOTE: The enrollment profile will appear as "Unverified" before end-user review and install profile and immediately after it's installed will show up as verified as well as a configuration profiles.
  • Profiles installed before the upgrade won't be retroactively signed
• [ ] Outdated documentation changes: TODO Release notes that existing profiles won't be retroactively signed (they must be re-installed). UPDATE: This didn't make it into the release notes. Pulling it into a comment here.

Engineering

• [ ] Database schema migrations: TODO
• [ ] Load testing: TODO

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: TODO
• Risk level: Low / High TODO
• Risk description: TODO

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.

[lukeheath]

@georgekarrv Assigning this to you to break into sub-tasks and spec (or assign to an engineer to spec.)

[zhumo]

Hi @noahtalerman , this story did not make it into the current sprint, so I'm de-prioritizing it. Please bring it back to FF if necessary.

[noahtalerman]

Hey @georgekarrv when you get the chance, can you please work with folks on the MDM team to determine what we need to build to support signing the enrollment profile?

[zhumo]

@noahtalerman does this one need design? Could it go to the next stage?

[noahtalerman]

@zhumo no, it doesn't need design. Good catch. I moved it to the designed column.

[noahtalerman]

@georgekarrv heads up, removed this from the drafting board as part of design sprint review because it didn't get estimated

[noahtalerman]

Hey @zhumo this didn't make it into the current sprint. Please bring it back to FF if you think we should consider it for next sprint.

[johnny-yeng]

Bumping up this FR 🆙🆙

Similar to @zhumo, my colleagues complained about the enrollment profile having a big, scary unsigned and Allow device erase permission.

Requirements

It would be nice if there were some endpoint to upload custom enrollment profile or a flag to set the features of the manual_enrollment_profile.

My Goal

• Sign the profile to prevent modification.
• Edit the default access right from 8191 to 4161.
• Add profile removal password to prevent colleagues from removing the profile by themselves.

[xastherion]

Me too. Would be not possible to put a way to sign the enrollment profile with another tool (like imazing) and then upload it again to fleet?. Or to load the code-sign certificate to sign it?

[roperzh]

potential implementation in this draft PR: https://github.com/fleetdm/fleet/pull/16490/files

[nonpunctual]

https://www.jamf.com/blog/malicious-profiles-come/

[noahtalerman]

potential implementation in this draft PR: https://github.com/fleetdm/fleet/pull/16490/files

Nice! Awesome that we started work on this.

@roperzh I noticed that your PR adds two new config options: mdm.signing_cert_bytes and mdm.signing_key_bytes

Can we instead use the SCEP cert/key to sign profiles?

This way, the IT admin doesn't have to manage another set of keys for macOS MDM features.

[noahtalerman]

@marko-lisica and I updated this user story to cover signing for enrollment profile and configuration profiles.

Closing the "Sign macOS configuration profiles" issue: #9148

[roperzh]

Can we instead use the SCEP cert/key to sign profiles?

unfortunately we can't use it to sign the enrollment profile because the SCEP certificate is not trusted by the device.

quoting the docs on that PR:

To ensure successful installation on a macOS devices, signatures using the certificate must be recognized as trusted. This can be achieved by:

1. Utilizing a publicly trusted signing certificate.
2. Employing an Apple Developer certificate.
3. Using any signing certificate that has been authenticated by a Certificate Authority already trusted by the device.

we could make the SCEP cert work for configuration profiles, but not for the enrollment profile (unless there's a workaround I haven't thought of)

[roperzh]

@noahtalerman sorry forgot to ping you in the message above. I also want to add:

1. We could make signing non-compulsory?
2. We could consider using a certificate provided by us for cloud instances?

This way customers provide a cert if and only if they're self hosted and want the profiles signed.

[nonpunctual]

Maybe adding a CA is the answer if the SCEP cert can't be used?

[noahtalerman]

@roperzh isn't Fleet already a certificate provider? Fleet provides SCEP client certificates to the device. I could be wrong.

@nonpunctual are there other features that would benefit from a "built-in" Fleet certificate provider? (aka CA)

[roperzh]

@noahtalerman @nonpunctual indeed, Fleet is already a CA, but maybe you're onto something there? let's say that instead of using fleetctl generate mdm-apple to generate your SCEP server keypair, we change the flow so you have to provide a keypair issued by a CA that is able to generate certs that are already trusted? (we can't do that easily)

Then, the server SCEP cert will be trusted, and we can use it to sign the enrollment profile and any other profiles we send afterwards.

[noahtalerman]

keypair issued by a CA that is able to generate certs that are already trusted? (we can't do that easily)

@roperzh ah, so Fleet is a CA that's unable to generate certs that are trusted?

How difficult would it be to make it so Fleet can generate certs that are trusted?

[roperzh]

@noahtalerman that's the second part of the message above, happy to clarify anything that's obscure/unclear

but maybe you're onto something there? let's say that instead of using fleetctl generate mdm-apple to generate your SCEP server keypair, we change the flow so you have to provide a keypair issued by a CA that is able to generate certs that are already trusted? (we can't do that easily)

Then, the server SCEP cert will be trusted, and we can use it to sign the enrollment profile and any other profiles we send afterwards.

[nonpunctual]

@noahtalerman @roperzh yes there are benefits from it. This gets confusing pretty fast but I will try to use as few words as possible. :)

https://docs.jamf.com/technical-papers/jamf-pro/integrating-ad-cs/10.6.0/Distribute_Certificates_Using_Configuration_Profiles.html

A lot of Jamf customers use Microsoft Active Directory to generate certificates internally. These certs are signed with a private key from a Windows server & it's secure because it's usually very restricted. Generating keypairs & certs is something that's part of a change mgmt process & usually secrets are managed in a PAM db or dedicated secrets server.

AD CS = Active Directory Certificate Service. This allows Jamf (or actually any server built to use AD CS which is a Windows server thing...) to act as a CA proxy (not a network proxy...) basically meaning:

The org generates all certs internally with AD, therefore, if Jamf is securely integrated with AD in the local environment, the org can allow Jamf to hand out signed certs from AD CS. This is often used to hand out certificates to satisfy 802.1x authentication for automatic join to corporate Wi-Fi (no PSK needed, auth happens via the cert.)

Ok. Roberto is correct to say that these certs are not trusted automatically. Lots of traditional orgs still WANT to deploy their own internal certs which is usually a bad idea because clients & cloud services don't know how to trust them. They are effectively self-signed, meaning, they aren't a leaf or intermediate cert in certificate chain of trust. An example of universally trusted certs would be what you'd see in Keychain Access > System Roots. Windows, Mac, linux all come with a pre-installed set of top-level certs from many different vendors. This is how your computer knows to trust websites out of the box...

Part of what Jamf installs on enrollment is a top-level cert self-signed cert. This means anything it signs with its own CA is trusted because of 2 things:

1. Certs installed via MDM are automatically trusted
2. The top level cert in the chain is already in the keychain because it was installed during enrollment.

I think those 2 things are the magic that allow Jamf to sign profiles. For packages, they aren't signed, but, Jamf Composer lets you access your Apple Developer cert easily (kind of like iMazing Profile Creator) to sign packages.

Finally, the CA in the Jamf GUI lets upload a csr generated (so a private key...) with something like openssl or java keytool. This lets you use your Jamf CA to sign things from your Jamf server related to devvice management that will be trusted on your clients or in other web servers.

whew. sorry. I wish I had a tl;dr of this.

[roperzh]

@nonpunctual thanks so much! that's super interesting.

Part of what Jamf installs on enrollment is a top-level cert self-signed cert. This means anything it signs with its own CA is trusted because of 2 things:

1. Certs installed via MDM are automatically trusted

2. The top level cert in the chain is already in the keychain because it was installed during enrollment.

we evaluated this as an option, and works very well to sign configuration profiles once the host has MDM turned on, but what about the manual enrollment profile?

Finally, the CA in the Jamf GUI lets upload a csr generated (so a private key...) with something like openssl or java keytool. This lets you use your sign things from your Jamf server that will be trusted on your clients or in other web servers.

This makes sense, and I think it's equivalent to the IT admin providing a keypair that's trusted by the device via a config, right?

---

@nonpunctual @noahtalerman what I take from Brock's answer above + my personal understanding is that we know of two groups of options, both involve extra work for the IT admin:

• Option A: IT admin provides an extra keypair via a new configuration that's trusted and Fleet's uses to sign enrollment + configuration profiles
• Option B: Fleet uses one of the keipairs already has (let's say config.mdm.apple​_scep​_*), but IT admin ensures it's trusted by the host by doing ONE of:
  1. Option B1: Adding mdm.apple​_scep​_cert as a trusted certificate root (easy to do once enrolled, hard prior to enrollment which complicates things for manual enrollments)
  2. Option B2: Providing a mdm.apple​_scep​_cert that's already trusted because it was issued by a known CA (this involves changing the fleetctl generate mdm-apple flow)

[noahtalerman]

@nonpunctual and @roperzh thanks for the discussion and options!

I think up to @marko-lisica and @roperzh.

Marko, what are your thoughts on the options presented by Roberto in the above comment?

[marko-lisica]

I had discussion with Roberto to understand better each option, here are notes:

Option B2:

• User will need to get cert from trusted authority
• Configure server with this cert
• fleetctl generate mdm-apple will still generate SCEP, but it won't be able to sign profiles
• If existing customers want to sign profiles, we'll need migration (e.g. we don't check the certificate and we issue a command to renew a certificate)
• If we want to sign other stuff in the future (e.g. Software packages), then it might be tricky or even impossible to do this with SCEP cert

Option B1:

• Manual enrollment, user need to accept enrollment profile installation and that's when they will see that profile is unsigned (red text)
• Idea: maybe we can use fleetd to add SCEP to the user's keychain so Fleet generated SCEP is trusted by device and can be used to sign profiles.
  • Potential problem: If user manually enrolls profile by installing it from the link then we won't be able to use fleetd to add SCEP to keychain

OPTION A:

• It will work for other stuff like software package signing.

Potential problem (for all options): Existing customers in order to sign existing profiles (installed on hosts), will need to reinstall profiles. (Fleet could do this in the background)

[nonpunctual]

I think we need more information before we try to implement this & I'm concerned that means the choice will be made to not do so. Can we try to get some sort of consultation from Apple on the best way to implement this? I know they have an "office hours" model during WWDC but that's not until June. I believe there is some number of hours available for consultation from Apple through the developer certificate membership.

[roperzh]

@nonpunctual do you mind expanding what are your concerns? maybe we can use those as a guidance to investigate ourselves.

[nonpunctual]

1) I guess am not concerned about reinstalling profiles. It should be trivial to do that.

2) I am not sure I understand all the cases in which we are using a manual enrollment workflow as opposed to Automated Device Enrollment (ADE / DEP) I feel in the cases where we are using manual enrollment, signing them before delivery is MORE important than in automated delivery because manually handling unvalidated identity assets is how a malicious enrollment could occur.

3) I am unlclear about the distinctions stated regarding what can & can't be done with the SCEP cert. At the risk of repeating myself, I don't know exactly how the Jamf CA works, but, I know there are assets installed during enrollment which validate the connection from server to client & from the client to the server (bi-directional) & there is a "signing" certificate in the Keychain tied to the Jamf server instance that allow delivery of validated, signed Configuration Profiles.

[roperzh]

@nonpunctual thank you! let me know if this makes sense:

I feel in the cases where we are using manual enrollment, signing them before delivery is MORE important than in automated delivery because the manual handling of what is effectively an identity asset is where a malicious enrollment could occur.

I think we're on the same page there! Manual enrollment is what I mention as the main blocker to use a non-trusted cert (which would be the SCEP cert fleetctl generates today)

I am unlclear about the distinctions stated regarding what can & can't be done with the SCEP cert.

So, the concern there is that, if we go with option B1, the IT admin would provide a root keypair that's trusted, and that will be used by Fleet's CA to issue SCEP certs.

But, since this root keypair is not generated by Fleet anymore, it might have some restrictions to the key usage, and we will have this problem again if we need to sign software (coming up soon)

For more info on key usages:

The key usage extension defines the purpose (e.g., encipherment, signature, certificate signing) of the key contained in the certificate. The usage restriction might be employed when a key that could be used for more than one operation is to be restricted. For example, when an RSA key should be used only to verify signatures on objects other than public key certificates and CRLs, the digitalSignature and/or nonRepudiation bits would be asserted. Likewise, when an RSA key should be used only for key management, the keyEncipherment bit would be asserted.

Source: https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.3

[nonpunctual]

ok. when you are referring to "signing software" (coming up soon) I assume you mean Application features on the road map?

Jamf does not sign software it delivers from distribution points. Munki does not either. App distribution relies on code already signed from the vendors which is "sanctioned" by Apple via WWDR app submission (which is why notarization is important - it verifies the app went through Apple's submission process. I assume Windows & linux validate in a similar way.)

Apologies if this is not what you mean.

The Jamf CA can be used for anything but what it was designed for was to sign certs for horizontal scaling of the on-premises server install. This is the use case where an admin uploads a key pair / csr to the Jamf CA. It can be used in other ways though.

The Jamf "signing cert" is deployed during enrollment. anything Jamf delivers to a client signed by a matching instance of the Jamf server is trusted on the device because the trusted Jamf intermediate "signing cert" exists & is trusted in the keychain. This allows for encrypted, signed profiles to be delivered.

[roperzh]

Jamf does not sign software it delivers from distribution points. Munki does not either. App distribution relies on code already signed from the vendors which is "sanctioned" by Apple via WWDR app submission

IMO this is a reasonable ask: IT admin ensures that the package is signed before it's uploaded to Fleet.

The thing is: I don't know if we'll want to sign the packages for the IT admin (AFAIK this wasn't discussed yet) so I don't want to close any doors without Marko and Noah having an explicit understanding of what that implies.

The Jamf "signing cert" is deployed during enrollment. anything Jamf delivers to a client signed by a matching instance of the Jamf server is trusted on the device because the trusted Jamf intermediate "signing cert" exists & is trusted in the keychain. This allows for encrypted, signed profiles to be delivered.

I feel like we're talking over each other on this point :(. This makes sense for automatic enrollments. How would you do it for manual enrollments? where the manual enrollment profile is "sent" to the user prior to enrollment.

We could consider (noted above) adding the "signing cert" to Keychain with fleetd, but that won't work for flows like the one customer-preston has.

[nonpunctual]

Sorry! Not meaning to be redundant @roperzh :)

customer-Preston enrollment workflow can hardly be described as "manual". Please see chat for further details

[noahtalerman]

@marko-lisica the changes specified in this issue LGTM.

I think we can move it over to "Settled."

@roperzh please let us know if we're missing anything.

[georgekarrv]

Hey team! Please add your planning poker estimate with Zenhub @dantecatalfamo @ghernandez345 @gillespi314 @jahzielv @mna @roperzh

[noahtalerman]

@nonpunctual heads up, this story are settled and estimated.

[nonpunctual]

PS. a little color on this randomly I found the Intune replacement for internal AD certificate authority stuff today:

https://www.intuneirl.com/demystifying-cloud-pki-in-intune-an-advanced-guide-for-intune-administrators/

[roperzh]

@marko-lisica what should we do about the enrollment profile downloaded in this endpoint?

https://github.com/fleetdm/fleet/blob/71ae862b94de4f36e63b9a128173344cc4f35a04/docs/REST%20API/rest-api.md?plain=1#L4666-L4674

if we sign it, we'll break the flow of customers that sign the profile with a different certificate (customer-preston). Is it okay to not sign and add a note to the docs? should we add a query parameter that allows to skip signing?

[marko-lisica]

Based on today's discussion during design review and slack conversation, we decided to make SCEP certificate trusted via MDM (instead of using fleetd agent).

[noahtalerman]

today's discussion during design review and slack conversation

Moving the Slack thread here so we don't lose it:

Profile signing

1. Adding a certificate to the keychain doesn't make it trusted

2. Documented ways to make a root certificate trusted:

   • Deliver a profile via MDM (no user action)
   • Ask the user to install a profile (requires user action)
   • Set the trust programatically (requires user action)
     • https://opensource.apple.com/source/Security/Security-55471/sec/Security/SecTrustSettings.h.auto.html
   • Ask the user to add the certificate to the keychain via the ui

3. Undocumented ways:

   • Modify authdb (sqlite db) temporarily

      $ sudo security authorizationdb read com.apple.trust-settings.admin > rights
      $ sudo security authorizationdb write com.apple.trust-settings.admin allow
      $ security add-trusted-cert -d -r trustRoot -k ~/Library/Keychains/login.keychain-db ~/Downloads/fleet-mdm-apple-scep.crt
      $ sudo security authorizationdb write com.apple.trust-settings.admin < rights

   • From Brock: there might be a special/undocumented entitlement given by Apple to vendors

Jamf

• jamf trustJSS adds a trust for the JAMF CA into the System keychain.
• Ask the user/IT admin to manually install a config profile with the certificate.
• Add a trusted cert that Jamf uses to sign

Our options

1. Allow users to give us a signing cert

   • Simple, but after chatting with Brock and looking at docs, sometimes people want to use their own, untrusted certificate anyways.

2. Sign everything using the SCEP certificate, deliver a configuration profile making it trusted.

   • This will show "Verified" in green for all scenarios except for manual enrollment profile.

3. Build some flow for manual enrollments that requires user interaction to add the cert

   • Per 1, also solves scenarios of people wanting to use a different CA
   • Per 2, this flow is only required for manual enrollment

4. Do the authdb hack

   • Might get flagged by security auditors. Passed the Brock smell test
   • Undocumented

[noahtalerman]

Sign everything using the SCEP certificate, deliver a configuration profile making it trusted.

Marko: We learned that we can't do this unless we're a "supported MDM vendor." We're unsure what this means and how to get this status.

More info from Roberto:

This is what I see:

• I have a profile (attached) of type com.apple.security.root
• If I install that profile manually (double clicking on it) the cert shows up as trusted in Keychain
• If I send that exact same profile via MDM, the cert shows up as untrusted
• If I bundle the certificate to the enrollment profile, the cert shows up as untrusted

I have two theories:

• There are undocumented requirements that my certificate doesn't met to be trusted via MDM You need special MDM powers from Apple (based on this)

A certificate has automatic full trust if it is:

• Installed by an Apple Configurator instance that has the same supervision identity as the device
• Automatically installed from a supported MDM solution
• Manually installed by a payload attached to an enrollment profile from a supported MDM solution

[nonpunctual]

Our Apple Developer enterprise cert I believe allows us to have a small number of consulations with  per year. I asked Luke about this & he said the business-operations controls the certs & the team? I am not sure I understand this since the developer certs are for signing the software we create, but, if so, we should try to use one of these consultations to understand what it would take to become a supported vendor.

[noahtalerman]

Do the authdb method

Hey @roperzh, @marko-lisica and I chatted and we think we should go w/ the fleetd programmatic method. Why?

• We can achieve the goal of making profiles appear green w/o having to wait for Apple.
• Sounds like the risk is Apple removes support for the authdb commands. If this happens, profiles will be unsigned (they'll still get delivered).

Let's air guitar becoming a "supported MDM solution" so that we know what work is involved. This way, we can later decide if we want to switch from fleetd programmatic method to delivering the certificate via MDM profile.

If you think this is the wrong approach let's hop on a call. Thanks!

cc @nonpunctual @georgekarrv

[nonpunctual]

Doing whatever it takes to get 's blessing for all of this stuff is the only way it will get fully resolved. Please let me know what I can do to help.

[roperzh]

@noahtalerman I believe I figured out a few minutes ago what was the problem we had installing certs via MDM, so I don't believe we need to contact Apple anymore.

Do you still want to go with the fleetd solution? I would vouch not to do it that way but up to you and Marko

[roperzh]

@noahtalerman @marko-lisica to save us a meeting: https://www.loom.com/share/dfb9c77b5af44888a11f4680a583655e

[marko-lisica]

@roperzh We wanted to go with fleetd method because we thought it is the only one that's possible. Since you figured out how to manage this via MDM configuration profile, we should stick to that approach.

cc @noahtalerman

[roperzh]

@marko-lisica thanks so much, do you have any preference for the copy of this new profile?

[nonpunctual]

I still am going to make an Apple Developer support case because I think we need to close the loop on what Apple means exaclty by being a "supported MDM vendor". Thanks.

[marko-lisica]

@marko-lisica thanks so much, do you have any preference for the copy of this new profile?

@roperzh We always display the full term together with the abbreviation. Let's go with: Fleet root certificate authority (CA)

cc @noahtalerman

[noahtalerman]

Hey @roperzh are we stuck with "FleetDM" for the organization name?

We want to use "Fleet" in the docs in product.

This is looking really good.

cc @marko-lisica

[roperzh]

@noahtalerman unfortunately is the name we currently put in the SCEP cert, we can absolutely change that for new certs we generate tho.