Webhooks for global activity feed

Patagonia121 commented 10 months ago

Goal

User story
As an IT admin,
I want to bind a webhook to fire for any activity from Fleet's global activity feed
so that I can trigger a workflow in my automation tool (ex. Tines) when a specific activity occurs.

Context

Product designer: @rachaelshaw

What else should contributors keep in mind when working on this change?

One example use case: As an IT admin, I want to receive a webhook when a host has MDM features turned on so that I can add a Munki configuration file to that host. This way, the Munki agent on the host can communicate w/ my S3 bucket.

Changes

Product

[x] UI changes: Figma
[x] REST API changes: Draft PR
[x] Permissions changes:
- 19863
[x] Outdated documentation changes:
- 19863
[x] Changes to paid features or tiers: Available in Fleet Free and Premium

Engineering

[ ] Database schema migrations: TODO
[ ] Load testing: TODO

ℹ️ Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

Requires load testing: TODO
Risk level: Low / High TODO
Risk description: TODO

Manual testing steps

Step 1
Step 2
Step 3

Testing notes

Confirmation

[ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
[ ] QA (@____): Added comment to user story confirming successful completion of QA.

noahtalerman commented 9 months ago

Feature fest: This is core. I don't think we have the capacity to take it on immediately. Please bring it back.

Patagonia121 commented 8 months ago

This came up again today with customer-reedtimmer - they mentioned this is one of the things they are using heavily with Jamf and something they'd need us to support in terms of feature parity. Bringing back to the next feature fest.

noahtalerman commented 8 months ago

Hey @Patagonia121 this issue looks like a duplicate of the following: #15567

Closing #15567

noahtalerman commented 7 months ago

Brock: Jamf has a policy trigger called something like "enrollment complete." It works sometimes.

Used for...

Reporting. Keep track of # of devices enrolled v. employees onboarded. Who didn't enroll but was supposed to?

noahtalerman commented 7 months ago

Heads up @Patagonia121 this request was discussed during feature fest last week and didn't make it into the current design sprint.

dherder commented 6 months ago

prospect-rosner is also looking for similar webhooks, specifically around re-enrollment events

noahtalerman commented 6 months ago

Heads up @Patagonia121 and @n8felton, this feature request was brought to feature fest on 2024-02-15 and wasn't prioritized for the current design sprint.

This is core to Fleet. Please bring it back to the next feature fest so we can weigh it again.

noahtalerman commented 4 months ago

Hey @Patagonia121, because this issue is in the current design sprint, I updated the issue to use the user story format.

I moved your original issue description here for safekeeping:

Problem

Customer is looking to Fleet to build more webhooks beyond the existing webhooks/automations found here https://fleetdm.com/docs/using-fleet/automations

The specific use case:

The customer is currently performing internal validation that a newly-enrolled host meets their provisioning requirements and is updated in all systems. They are trying to expand on this for user outreach when they receive a new (or replacement) device to send automated messages or surveys to their end user so they can get feedback on their logistics practices as well. They are hoping for Fleet to publish a "new device enrolled" event-based webhook so they can incorporate into their internal validation process and leverage with their current MDM tool (Jamf Pro).

rachaelshaw commented 3 months ago

I went ahead and designed this feature based on this spec, but during the design review, it came up that there's an existing feature that sends audit logs to a log destination. (Documentation for how to enable it here.)

@nonpunctual we were wondering if you could help us get a better idea of some of the use cases for this feature request, and whether sending activities to a log destination might solve them. Here's the call recording for context.

nonpunctual commented 3 months ago

@rachaelshaw @noahtalerman @marko-lisica I am unclear about the word "global" in this title (maybe it just means beyond Fleet Policies) but the use case described is very specific & maybe masks the importance.

Tracking the "enrollment complete" event is extremely important in most orgs. Having a new enrollment send a webhook would be super cool for a number of reasons mostly due to easy integration:

security, i.e., a new device now has access to enterprise resources, needs VPN / proxy access
hardware asset management, i.e., new device for onboarded employee X & a record must be created in HAM system
software provisioning, i.e., a series of installations is kicked off based on role / identity / device type, etc.
notification, i.e., Slack or email because a physical handoff may need to occur between IT setup team & manager or HR
testing, i.e., an automated test build (i.e., enrollment > provisioning) has completed & needs to be checked

Those are the ones I can think of off the top of my head. @dherder @spokanemac Can you think of others?

Another event that Jamf can send as a webhook is "Smart Group Membership Change" so:

device matches some attribute A, device is in smart group X
device at some point in the future no longer matches attribute A, device falls out of group X.

Sending a webhook on that event is extremely powerful. This is very similar to Policy failure in Fleet, but, more flexible & maybe could be associated to Labels or Team membership. Recent conversations with customer-sarahwu lead me to believe that something like this is critical for their future plans.

I can see webhooks being sent to acknowledge changes made in Fleet via gitops, i.e., a new configuration profile was added, a new script was added, a new team was created, etc. These things could be triggered from the Github Actions or whatever themselves, but, acknowledging the actual change in Fleet UI might notify someone that something bad happened if so.

nonpunctual commented 3 months ago

@rachaelshaw Another use case: customer-preston has multiple customers & automated workflows for MDM enrollment.

customer deployments are distributed
waiting for fleetd to process queries post-enrollment causes automation based on enrollment to break
an "enrollment complete" webhook would allow customer-preston to trigger Fleet actions to satisfy post-enrollment workflows

noahtalerman commented 3 months ago

customer-preston has multiple customers & automated workflows for MDM enrollment.

an "enrollment complete" webhook would allow customer-preston to trigger Fleet actions to satisfy post-enrollment workflows

@nonpunctual do you know what actions exactly the customer wants to trigger?

@roperzh do you know when exactly Fleet creates this "turned on MDM features" activity item? (example from dogfood)

Is it when Fleet gets updated host vitals for that particular host? Different timing?

@rachaelshaw I think we can use this existing "turned on MDM features" activity feed item to enable the customers actions. Instead of adding a new activity type.

However, let's make sure that the timing (when webhook will be fired) works for the actions the customer wants to trigger.

roperzh commented 3 months ago

@roperzh do you know when exactly Fleet creates this "turned on MDM features" activity item? (example from dogfood)

@noahtalerman we do it right when the host turns on MDM features, in one of the API calls that are part of the MDM protocol.

noahtalerman commented 3 months ago

we do it right when the host turns on MDM features, in one of the API calls that are part of the MDM protocol.

Great. So we don't have to worry about there being a time lag between when the host turns on MDM and when this is reflected in host vitals.

nonpunctual commented 3 months ago

@noahtalerman are the comments above unclear about "what actions exactly the customer wants to trigger?" Those are all real-world events that organizations use the Jamf "enrollment complete" policy trigger for. The use case at the top of this issue, starting a provisioning workflow that installs apps is probably the most common.

lucasmrod commented 3 months ago

@nonpunctual How much real-time does this need to be?

E.g. can a global activity item be streamed to a webhook 5 minutes after it happened?

(Asking to re-use the functionality we have for streaming these activities to the audit log destination)

sharon-fdm commented 3 months ago

@nonpunctual Would you be able to describe what we should do in case the webhook fails? (Speaking more about 4xx errors of not a server issue but rather a bad payload etc)

sharon-fdm commented 3 months ago

TODO: @rachaelshaw set a discovery meeting next week.

sharon-fdm commented 3 months ago

BE: 8 FE: 3

sharon-fdm commented 3 months ago

@noahtalerman, per our discussion, @getvictor will take the BE part this sprint.

nonpunctual commented 3 months ago

@lucasmrod I think anything between 0s-5m is acceptable. Anything outside that range starts to become a problem for things like notifications in Slack, kicking off workflows in other systems, etc...

@sharon-fdm I know this isn't how it works but maybe I would be expecting something like the TCP protocol which is very redundant. Like, over 5m, try every 1m, or over 10m try every 2m. I am not sure I have a better answer than that because this is meant to start another distributed service or send a notification that something happened in Fleet.

marko-lisica commented 2 months ago

TODO: Docs and permissions changes. @noahtalerman

marko-lisica commented 2 months ago

Hey @zayhanlon, @Patagonia121, @pintomi1989, @nonpunctual this customer request has shipped.

martinpannier commented 2 months ago

Oh we know 😍

On Thu, Jun 13, 2024 at 5:36 PM Marko Lisica @.***> wrote:

Hey @zayhanlon https://github.com/zayhanlon, @Patagonia121 https://github.com/Patagonia121, @pintomi1989 https://github.com/pintomi1989, @nonpunctual https://github.com/nonpunctual this customer request has shipped.

— Reply to this email directly, view it on GitHub https://github.com/fleetdm/fleet/issues/14722#issuecomment-2166027137, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADACUW5HIRPP23QST7TI33ZHG36XAVCNFSM6AAAAAA6OELRXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRWGAZDOMJTG4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Martin Pannier

Founder & CEO

+33 6 15 97 00 07

noahtalerman commented 2 months ago

Hey @martinpannier! Please let us know if you have any feedback :)

noahtalerman commented 2 months ago

Merging in docs is still TODO: #19863

After the docs are merged we can close this user story.

noahtalerman commented 1 month ago

The docs are merged! https://github.com/fleetdm/fleet/pull/19863

fleet-release commented 1 month ago

Webhook's gentle pull, Automation flows like stream, Fleet dances, tasks eased.

fleetdm / fleet