search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
2.97k stars 413 forks source link

Enforce latest OS when macOS, iOS, and iPadOS hosts automatically enroll #19674

Closed nonpunctual closed 2 days ago

nonpunctual commented 3 months ago

Goal

User story
As an IT admin,
I want to require end-users to have a minimum OS version installed on the host before they enroll to MDM (with ADE)
so that I can make my new hosts compliant before they are enrolled into production.

Context

Research

During MDM enrollment, the device will respond with MachineInfo to the server when fetching the enrollment profile. The server needs to check if the device is on specified minimum OS version, if not server sends 403 error

More info here (Create a New Device Management Connection > step 3)

Changes

Product

Engineering

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

Manual testing steps

  1. Step 1
  2. Step 2
  3. Step 3

Testing notes

Confirmation

  1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
  2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.
noahtalerman commented 3 months ago

Thanks @nonpunctual. Really helpful link to Filewave docs.

We'll weight this at the next feature fest on 2024-06-20.

nonpunctual commented 3 months ago

Apple guide for "Managing software updates for IT" updated June 2024 per WWDC managing-software-updates-v1-0.pdf

lashomb commented 3 months ago

This would be really nice to replace some problematic lagacy policies we have to do this now.

nonpunctual commented 3 months ago

@lashomb added customer label to this issue. Thanks!

marko-lisica commented 3 months ago

@nonpunctual I converted the original issue description to user story format. Moving original here.

Apple has added a feature for denying enrollment if a device is not on a specified, minimum OS version.

Property from the schema documented at https://developer.apple.com/documentation/devicemanagement/machineinfo -

Property: MDM_CAN_REQUEST_SOFTWARE_UPDATE Type: boolean Value: If TRUE indicates that the (MDM) server can trigger the (enrolling) device to do a required software update. Available on iOS 17 and later, and macOS 14 and later. Default value: false

From https://github.com/apple/device-management/blob/release/mdm/errors/softwareupdate.required.yaml -

The schema for a JSON or property list XML document returned in an MDM server's 403 response body. The response headers must include a "Content-Type" header indicating whether JSON or XML is being returned.

This response is returned when a device is enrolling with an MDM server during Setup Assistant, and the MDM server requires the device to perform a software update before enrollment is allowed and setup can proceed.

In order for users to take advantage of this feature Fleet must be enabled to:

Problem

Fleet is not currently enabled to deny MDM enrollment based on a specified minimum OS version.

Potential solutions

  1. Add the ability to recognize the MachineInfo properties to Fleet MDM
  2. Enable the response for devices that do not match a specified minimum OS version

Documentation for this feature in Filewave https://kb.filewave.com/books/apple-school-business-manager/page/minimum-os-version-for-enrolling-apple-devices-via-ade

noahtalerman commented 3 months ago

Hey @nonpunctual I pulled the ~feature fest off because this story is in the current design sprint :) You can tell if it has :product

Please check before adding requests to feature fest. Thanks!

georgekarrv commented 2 months ago

@roperzh when you get some time can you review this and make sure we have a good understanding of all of the steps needed to both identify and deliver this specific case? When that's done go ahead and pull this over to specified.

roperzh commented 2 months ago

This should apply only on hosts running macOS 14 and above or iOS/iPadOS 17 and above that are enrolling through ADE

~as I understand this flow is available for manual enrollments too, any reason to limit it to ADE?~

edit: I was wrong, seems like it's ADE only, sorry!

@marko-lisica or @nonpunctual (not sure who added this section)

georgekarrv commented 2 months ago

Hey team! Please add your planning poker estimate with Zenhub @dantecatalfamo @ghernandez345 @gillespi314 @jahzielv @mna @roperzh

PezHub commented 2 months ago

QA Notes: Identified a blocker - Summary: we can't force any version, it has to be one of the versions defined here. If you specify other version, the user can't continue through setup assistant (screenshot in the thread). This is a problem because we currently enforce the version configured by the IT admin for software updates.

Error - SoftwareUpdateError

We will work on a fix and delay release.

Good news - setting the minimum version to Apple's latest release (14.6) and hardcoding the BundleVersion (separate PR) does indeed work. 14 6 update 14 6 install

noahtalerman commented 1 month ago

@gillespi314, @marko-lisica, and I met during design review and we decided to pause on this feature and not ship it in the upcoming Fleet release.

Why? Currently, if the IT admin specifies a minimum macOS version that's not included in this public list here (ex. macOS 14.5), the end user won't be able to enroll (see error in the comment here). We don't want to IT admins to shoot themselves in the foot.

What's the plan?

Any thoughts/concerns w/ this plan?

cc @lukeheath @georgekarrv @PezHub

nonpunctual commented 1 month ago

@noahtalerman @gillespi314 @marko-lisica @lukeheath @georgekarrv @PezHub @spokanemac @zayhanlon

This is a very important feature.

Why? Because organizations that have not yet built complex Device Health / Conditional Access workflows will get to take advantage of one of the most important checks at the most important time, i.e., they get to validate the minimum OS prior to enrollment. In many orgs, minimum OS is the only attribute that's strictly enforced.

This feature will temporarily take pressure off of design + eng on some of the more complicated FRs related to Conditional Access workflows.

It is part of an admin's job to validate the up-to-date OS versions on that list. There is a Slack RSS feed that delivers them in Mac Admins Slack or to email. They are now in the SOFA feed. Apple publishes them. Other 3rd party sources publish them. This is something most macOS admins check daily.

When we talk about "admins shooting themselves in the foot" I believe our standard should be something like "prevent admins from doing something that will impact the performance overhead of Fleet on the Hosts & on the server." This does not fall in that category of error.

I can also see using this in a postive way: Maybe you want to block all enrollments for some reason today? Well, you could set the minimum version higher than an actual extant version. That would be a lot easier to do than to disable all of your enrollment policies & workflows.

We should work under the assumption that admins using Fleet are being intentional & know what they are doing.

If you give someone a toolbox & they decide to hammer in nails with the monkey wrench, I don't think it's a good solution to that problem to take away the monkey wrench (which they still might need.) Give them all the tools & make it clear how they work.

roperzh commented 1 month ago

@noahtalerman, thanks to @PezHub and @gillespi314, we tried if we really can't provide any version by grabbing a bundle version for macOS 14.5 and trying the whole flow again and 100% confirmed that we can't use any other version except the ones listed under the PublicAssetSets key.

roperzh commented 1 month ago

@nonpunctual if an update is released and the IT admin doesn't catches it on time, then nobody can enroll via ADE until they're back again to fix it , this can be an update released just while the IT admin is in a long meeting, or (for global teams) in a different timezone, or maybe OoO.

I agree with your general premise, but this seems like a scenario where we're setting the IT admin for failure.

lashomb commented 1 month ago

There have and always will be gaps between OOBE provisioning and enforced settings by administrators. When new hardware comes out, it may not be on the version of the OS that you’ve set for the rest of the fleet. But that’s fine, we’ve dealt with that before. It’s not like we’re going to erase things back to an older version like the old days. Newly imaged devices always represent in someway an accumulation of improvements to our process over time, and it’s ok by me if a percentage of the environment is on a later OS than mandated by the org.

nonpunctual commented 1 month ago

Appreciate that feedback @lashomb !! Certainly the intent is not to set up for failure. I think we could build safeguards or even API checks against non-extant OS versions. My emphasis is only on getting the feature out. Thanks!

noahtalerman commented 1 month ago

I hear you @lashomb and @nonpunctual! We're working to ship an improvement ASAP but we don't want to ship something that doesn't work as we intended.

noahtalerman commented 1 month ago

Heads up @lukeheath, @georgekarrv, and @gillespi314, I unassigned Sarah and moved this issue from the release board to the drafting board.

lukeheath commented 1 month ago

@noahtalerman Since we're so close, it would be nice to get this over the finish line next release so we can deliver value to our users. My questions/thoughts:

  1. I think this is by design and not an Apple bug. If this feature was introduced in 14.6, it likely doesn't work on any version prior to that. It's worth checking with Apple, but I think it's safe to operate under this assumption in order to get it out next release.

  2. I'm not sure where the OS version is being set in this workflow. Presumably at some point it goes through our API. Could our API endpoint enforce the minimum version requirement of 14.6? If you try to set it to 14.5, it returns 400 Bad Request.

  3. But what if someone wants 14.5? Can this feature be toggled? In other words, if the feature is enabled, the API enforces a minimum version of 14.6. If it's not enabled, it doesn't enforce any minimum.

noahtalerman commented 1 month ago

it would be nice to get this over the finish line next release

@lukeheath agreed.

The plan is to spend 30 mins on a design review call to decide on an approach. Thanks for the questions/thoughts. Super helpful for starting the conversation.

noahtalerman commented 1 month ago

100% confirmed that we can't use any other version except the ones listed under the PublicAssetSets key

Thanks @roperzh and team!

@PezHub, do we know if/how this affects the OS updates via DDM feature?

That is, if I set 14.5 as my minimum version via Fleet today, will an end user that's below 14.5 see OS update notifications and get updated to 14.5?

Will they get updated to 14.6? (available in PublicAssetSets)

Will they get updated at all?

noahtalerman commented 1 month ago

Brock: Not sure what other MDM solutions are doing b/c they may or not have this feature yet.

noahtalerman commented 1 month ago

Sarah: Here's the link I mentioned where Apple says enforcing a specific software update version no longer fails if it’s not the most recent available update (starting with devices on macOS 14.6)

gillespi314 commented 1 month ago

do we know if/how this affects the OS updates via DDM feature?

@noahtalerman, DDM works as expected. The DDM updates aren't limited by the PublicAssetSets issue. So today, if the admin sets the minimum version to 14.5, a Sonoma device using ADE will enroll to Fleet successfully, and the minimum version will be enforced after enrollment and they'll see the OS update notifications and get updated to 14.5 according to the specified deadline.

noahtalerman commented 1 month ago

UPDATE:

New plan

The plan for this features (v1) is for Fleet to always, by default, enforce the latest macOS, iOS, and iPadOS during Setup Assistant.

Fleet will get the latest OS from Apple here (Public AssetSets). If the request to Apple fails, Fleet will try again up to 3 times. If all attempts fail, Fleet will let the end user through Setup Assistant w/o an OS update.

prospect-numa and customer-faltona confirmed that this is the desired behavior.

Later, we can get fancier and allow the IT admin to disable this behavior, enable it for specific platforms (ex. macOS only) and/or present the user with a list of supported versions.

Hey @georgekarrv I updated the issue description to reflect this, moved the story to ready for spec and assigned you.

Can you please work w/ @gillespi314 and the team to get this one re-estimated and pulled into the current sprint? Please let me know if including iOS/iPadOS changed the estimate drastically.

Old plan summary

The old plan was to use the existing "Minimum version" option and allow the IT admin to specify a specific macOS version that's enforced during Setup Assistant.

However, currently, Apple only supports versions included under PublicAssetSets here. This means the IT admin can shoot themselves in the foot and block end user from enrolling if they specify a version that isn't supported.

If folks have concerns/thoughts on the new approach please let us know!

cc @nonpunctual @lukeheath @lashomb @roperzh

nonpunctual commented 1 month ago

@noahtalerman I have already expressed my concern on this but I will again. We are hard-coding what versions of macOS admins can use by having built the Nudge implementation the way we have & now with this. We should allow admins to set a version. Thanks.

lukeheath commented 1 month ago

@nonpunctual Can you help me understand the use case where choosing a less-than-latest version during migration would be desirable?

@noahtalerman Have any users expressed concern about being forced into the latest version?

noahtalerman commented 1 month ago

@nonpunctual Can you help me understand the use case where choosing a less-than-latest version during migration would be desirable?

Yes please!

Have any users expressed concern about being forced into the latest version?

None so far. Thumbs up from prospect-numa and customer-faltona

nonpunctual commented 1 month ago

@noahtalerman it might not be the strongest case, but, Apple allows 90d from the offical OS release day to defer install. During that 90d period orgs might want to hold off on deploying the latest, greatest to test. Once the testing is satisfied, then the up-to-date version could be released.

dherder commented 1 month ago

@noahtalerman I agree with Brock's assessment here. While many enterprise orgs will want the latest and greatest, some will still want the option to defer the upgrade, especially if they are in highly regulated industries or if they have many users with legacy third party software applications that may break with the latest release.

noahtalerman commented 1 month ago

@nonpunctual and @dherder I hear you.

To move quickly, and because the customers/prospects we asked always want the latest, I think move forward with always enforcing the latest by default for everyone.

Can we please track a feature request to add the ability to turn if off? We can follow up quickly w/ a config option!

nonpunctual commented 1 month ago
Screenshot 2024-08-12 at 1 44 19 PM
noahtalerman commented 1 month ago

Hey @gillespi314, when you get the chance, does the OS update happen before Fleet sends any MDM commands? (InstallApplication, InstallProfile, etc.)

customer-eponym is wondering...

gillespi314 commented 1 month ago

Hey @gillespi314, when you get the chance, does the OS update happen before Fleet sends any MDM commands? (InstallApplication, InstallProfile, etc.)

customer-eponym is wondering...

Yes, it is enforced by Apple before the device is allowed to download the enrollment profile.

noahtalerman commented 1 month ago

Hey @gillespi314 we just learned this from customer-rosner:

This means that our current approach (always updating to latest no matter what) would make folks nervous (at the least) or create a poor onboarding experience (at the worst).

I met with @georgekarrv and we think it makes sense to make the following change this sprint:

Enforce the latest macOS, iOS, or iPadOS during Setup assistant, if the host is below the minimum version set in Fleet. Let the end user through setup assistant w/o an upgrade if their host is above the minimum version set in Fleet.

I updated the issue description (specs) to reflect that^

If you have questions or concerns please feel free to schedule some time w/ me! Thanks :)

lashomb commented 1 month ago

Enforce the latest macOS, iOS, or iPadOS during Setup assistant, if the host is below the minimum version set in Fleet. Let the end user through setup assistant w/o an upgrade if their host is above the minimum version set in Fleet.

This would probably be the safest, I like that approach as well.

PezHub commented 1 month ago

QA Notes:

Tests:

This is the screen the user is presented with prior to enrollment ForcedOSupgrade

Additional note: @noahtalerman after re-installing 14.4.1 on my device, I moved it to another team that had a minimum OS set to 14.5 and ensured DDM worked as expected. I received the prompt that a scheduled update was set for 14.5 (not 14.6 latest)

noahtalerman commented 1 month ago

Thanks @PezHub!

@noahtalerman after re-installing 14.4.1 on my device, I moved it to another team that had a minimum OS set to 14.5 and ensured DDM worked as expected. I received the prompt that a scheduled update was set for 14.5 (not 14.6 latest)

Is this the same for iOS and iPadOS?

Assuming yes, @georgekarrv I think we want to call out this difference in behavior in the UI so that the IT admin knows.

I think this can be a quick tooltip update as designed in Figma here: https://www.figma.com/design/8anmUdAjxircEGfraBTiBu/%2319674-Enforce-latest-OS-when-macOS%2C-iOS%2C-and-iPadOS-hosts-automatically-enroll?node-id=0-1&t=9iYPY3Ej8krZCVGP-1

Do you think we can squeeze that copy update in before we ship the release?

I added the above Figma wireframes to the Product section in the GitHub issue.

georgekarrv commented 1 month ago

From what I have seen my ios devices could only update to the latest. I may have missed this case specifically though

noahtalerman commented 2 weeks ago

@georgekarrv I think we want to call out this difference in behavior in the UI so that the IT admin knows.

I think this can be a quick tooltip update as designed in Figma here: https://www.figma.com/design/8anmUdAjxircEGfraBTiBu/%2319674-Enforce-latest-OS-when-macOS%2C-iOS%2C-and-iPadOS-hosts-automatically-enroll?node-id=0-1&t=9iYPY3Ej8krZCVGP-1

Do you think we can squeeze that copy update in before we ship the release?

Hey @georgekarrv, I forgot to follow up on this so we didn't get the testing/tooltip copy in before shipping this story.

I filed a bug here for testing and tooltip copy update: https://github.com/fleetdm/fleet/issues/21976

I think let's address that bug as soon as we can (top priority below priority tickets) so we can close this story out.

cc @gillespi314 @lukeheath

noahtalerman commented 2 weeks ago

Hey @georgekarrv just giving you an extra ping! re this comment.

noahtalerman commented 2 weeks ago

Hey @zayhanlon and @dherder heads up that this customer/prospect request was shipped!

We're leaving it open until we ship a copy change (bug) but the core functionality is ready to use.

noahtalerman commented 1 week ago

Pulling this out of Slack:

Sarah: For OS updates post-enrollment, we’re trying to understand how to describe how DDM OS updates work. Can we accurately say that “If an already enrolled host is below the minimum version [specified in Fleet], the host is updated to exactly the minimum version [specified in Fleet].” My understanding is that we can’t guarantee that it will be exactly the version specified in Fleet because there are some not very well documented limitations based on what is available in AssetSets from gdmf .

Roberto: this document (shared a couple of times in this channel) is great. I think it answers all our questions my reading is that as Sarah suspected, and Gabe confirmed, you can only pick versions from AssetSets in gdmf also looks like they expect MDM vendors to query this (see under "Using the Apple Software Lookup Service")

managing-software-updates-v1-0 (2).pdf

Sarah: Yep, I referenced this too. I think our current task is to confirm this documentation can be trusted and sus out edge cases as best we can. Then try to describe it all to our end users in a way that makes sense with our current UX where we offer users a free text field (where we assumed that old versions aren’t being removed) while Apple has designed this with ever-changing additions and deletions. For MDM providers like us, we need to provide UX for two paths: When an IT admin wants to set the minimum version, we need to offer them a list that matches the current AssetSets and limit their choices to that list When Apple changes the AssetSets to remove a version that was previously set in Fleet, we need some kind of UX to surface that issue to the IT admin and let them take action (or at least explicitly define whatever action Fleet will take by default in that scenario).

George: I think instead of the perfect UX or the perfect tooltip we need to just figure out how to explain this behavior for this ticket and open a feature request to improve it to get this closed in this release. If an already enrolled host is below the minimum version, the host will try to be updated to exactly the minimum version if available from Apples hosted versions.

If the minimum version is no longer available from Apple the update will not be scheduled.

If a new or wiped host is below the minimum version and automatically enrolls (ADE), the host is updated to Apple's lastest version during Setup Assistant.

George: The problem is it is wrong because we cannot got to exactly the minimum version when it's not on gdmf anymore

Noah: Got it! Then, I think let’s go w/ a simpler version of George’s copy. This link doesn’t tell me the versions available.

If an already enrolled host is below the minimum version, the host is updated to exactly the minimum version if it’s available from Apple.

If a new or wiped host is below the minimum version and automatically enrolls (ADE), the host is updated to Apple’s lastest version during Setup Assistant.

noahtalerman commented 4 days ago

Waiting on this bug fix before we close this story: https://github.com/fleetdm/fleet/issues/21976#issuecomment-2369392870

cc @ghernandez345

ghernandez345 commented 4 days ago

@noahtalerman this is now done and was merged in with this PR #22337

noahtalerman commented 4 days ago

@ghernandez345 thanks!

Do you think we should re-open the bug and bring it back onto the release board? So it gets QA'd.

cc @georgekarrv

noahtalerman commented 2 days ago

UPDATE: Closing this user story. I checked to make sure that the redirect in the UI works: https://fleetdm.com/learn-more-about/available-os-update-versions

fleet-release commented 2 days ago

In cloud city's glow, Fleet ensures OS up-to-date, Smooth enrollment bestows.