Fleet + Open Interpreter demo: MDM migration

noahtalerman commented 3 months ago

Demo video: https://www.loom.com/share/e5f733b92773476690b8d4f38592b35d?sid=e777a0c6-c0a1-4906-981c-26c58b4ff883

Goal

User story
As an IT admin,
I want to see a demo of Open Interpreter (OI) migrating a macOS host to Fleet
so that I can see how OI + Fleet can automate IT workflows that require end user action.

Context

Requestor(s): @noahtalerman
Product designer: @noahtalerman

Demo

[x] Show Fleet creating a maintenance window (calendar event) for MDM migration
[x] During the maintenance window, show OI clicking through Fleet's end user migration workflow: https://fleetdm.com/docs/using-fleet/mdm-migration-guide#how-to-turn-on-mdm-end-user
- It's ok if we don't perform a migration from one MDM solution (ex. Workspace ONE) to Fleet. This requires hooking up Fleet and WS1 w/ Tines via a webhook. Instead, for demo purposes, we can start w/ a host that already has MDM turned off.
- End user's local pwd will love in a separate keychain.
- In the demo the Keychain pwd, to retrieve the local pwd, will be hardcoded.
- In production the Keychain pwd will be unique per device. Open Interpreter will talk to Fleet (via API) to get the Keychain pwd.

noahtalerman commented 3 months ago

Hey @getvictor and @sharon-fdm I filed this story to track what Victor and I decided during our call today.

Up to you on where the issue lives. I threw it on the :release board for now.

Show Fleet creating a maintenance window (calendar event) for MDM migration

noahtalerman commented 3 months ago

@getvictor I totally goofed and forgot to mention that we want Fleet's maintenance windows to be in the demo.

That way we can paint the full picture for IT admins and end users: My organization is migrating to Fleet => calendar event shows up when I'm (end user) free => when calendar event starts, OI migrates me to Fleet.

I think we can still use dogfood + Tines for this:

The mdm table answers what MDM solution in the server_url column. Could use this to write a policy.
Killian mentioned that they can export OI skills as bash/shell script. Tines can trigger running that skill.

Let me know if you run into a brick wall w/ either of those. We can audible to demoing a "fake" calendar event and then asking OI to run the skill.

getvictor commented 2 months ago

Video demo: https://www.loom.com/share/e5f733b92773476690b8d4f38592b35d?sid=e777a0c6-c0a1-4906-981c-26c58b4ff883

Prerequisites

Orbit needs access to control the computer (Accessibility), Screen & System Audio Recording, and access to System Events. We could request these permissions at installation. Otherwise, pop-ups will appear during script runtime and mess up the flow.

In development, we need to use a signed version of orbit to have access permission stick from run to run.

Policy to see if device is in Fleet MDM

SELECT 1 FROM mdm WHERE server_url = 'https://dogfood.fleetdm.com/mdm/apple/mdm';

Tines workflow

https://ancient-dust-5769.tines.com/stories/30620

Issues

Open Interpreter issue found: https://github.com/OpenInterpreter/open-interpreter/issues/1292

getvictor commented 2 months ago

Reassigning to Noah for next steps.

noahtalerman commented 2 months ago

Thanks @getvictor!

We could request these permissions at installation.

What does this mean for the end user? Can fleetd silently request these permissions w/o the end user taking any action?

noahtalerman commented 2 months ago

Moved the following conversation from Slack so we don't lose it.

Noah: How’s the Open Interpreter API looking? Do you think we should build on top of it v. building our own OCR using open source model? (ex. llama3)

Victor: My demo flow is flaky, and there are many hard-coded coordinates. It took me 5 takes to make that video. It would take ~1 week to make a flow that works most of the time. The OCR isn't great -- sometimes, it couldn't find the text on the screen.

Victor: Open Interpreter seems to be just a wrapper/integrator of other libraries/APIs. Currently, it doesn't seem to provide sufficient value for our use case. Their primary use case is chatting with an LLM and running the code it provides. Our use case of creating a reliable, repeatable flow is not their primary mission.

I think we should first eliminate Tines from the flow and create something similar to it (Tines-lite?) in our product.

Victor: We can try to make a robust "Migrate to Fleet MDM" script, but it probably won't work 100% of the time. There are lots of things that can mess it up that we can't even predict. Some obvious ones are screen resolutions, browsers, color palates, multiple screens, custom keyboard shortcuts, etc. And end users must be in front of their computer and not messing with it.