search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
186 stars 391 forks source link

[Auth0] Update Auth0 Integration to use Management API for Log Ingestion #10139

Open terrancedejesus opened 3 weeks ago

terrancedejesus commented 3 weeks ago

Summary

The Auth0 integration seems to use an outdated approach to retrieving and ingesting event logs. Rather the management API can be used to retrieve these logs instead as done by companies like DataDog and Splunk.

We should look to update this integration to use these same APIs, making this integration setup more feasible. TRADE is looking to add a prebuilt threat detection ruleset for this data, however, have ran into several hurdles and blockers with setting up this integration that has halted progress.

Please let us know if there is anything we can help with.

cc @andrewkroh @jamiehynds

References:

elasticmachine commented 3 weeks ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

andrewkroh commented 1 week ago

seems to use an outdated approach

@terrancedejesus, Can you please clarify what is outdated about auth0 custom webhooks?

terrancedejesus commented 1 week ago

@andrewkroh - Thank you for the reply! If I understand correctly, this integration was built to rely on an Auth0 stream that pushes logs via HTTP POST requests to a custom web server that stores locally. Then the Elastic Agent reads these logs and ingests them into the stack, is that correct? My assumption is that at the time this integration was built (~2 years ago) the management API may have not been available.

If so, this issue was opened to adjust how logs are retrieved, using the management API where logs can be pulled by the agent and then ingested as typically done with SaaS integrations. From the small blurb Retrieve logs by checkpoint, this appears to be how DataDog and Splunk ingest Auth0 logs as well.

cpascale43 commented 1 week ago

Hi @terrancedejesus! Looking to understand this one a bit more, are the events exposed via the management API the same as the ones we currently collect?

terrancedejesus commented 1 week ago

@cpascale43 hello! I am not sure but worth the investigation. Their notes suggest that this is how Splunk and DataDog ingest logs for monitoring.

Digging a bit, I see that the DataDog integration sets up the Auth0 log stream to point directly to the DataDog API which requires an API key to their instance. If management API does not seem viable, then we could explore this route and contact Auth0 to add an OOTB Elastic stream option, which is missing.

Screenshot 2024-06-25 at 8 52 28 AM

At the moment, it seems that we can update the integration or at least documentation to setup this integration with Elastic. At the moment, setting up a custom public web server just to accept POST requests and store logs for the agent to push to the stack seems like it could be updated.

The agent running this integration must be able to accept requests from the Internet in order for Auth0 to be able connect. Auth0 requires that the webhook accept requests over HTTPS. So you must either configure the integration with a valid TLS certificate or use a reverse proxy in front of the integration.

I could also be misunderstanding the steps to set this up if someone is willing to help guide me through setup.

jamiehynds commented 2 days ago

@narph to simplify the integration, a move to a typical API based integration makes sense for Auth0. The Management API didn't exist when we built the integration, hence the unusual workflow in place currently.

The endpoint to hit is: https://auth0.com/docs/deploy-monitor/logs/retrieve-log-events-using-mgmt-api An Auth0 account can be created here.

Ok to remove tech clarification and add to the next iteration?