search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.33k stars 1.74k forks source link

Audit Log page shows a timed out error on high-activity Firestore backends #42934

Open TeleLos opened 3 months ago

TeleLos commented 3 months ago

What would you like Teleport to do?

In self-hosted environments with a GCP Firestore backend, The Teleport UI Audit activity does not render successfully if the cluster has high event activity. This was observed when the events exceeded 1 million Firestore documents per day. With Firestore, the query will time out because too much data is being requested.

The feature request is to improve the Teleport UI offering selection options that allow for smaller queries to the backend. Less data will be requested. Grafana for example offers query options for the last 5, 15, or 30 minutes.

What problem does this solve? Currently, the Teleport UI does not render any data for the customer and displaces an error.

Screenshot 2024-05-21 at 3 29 15 PM (1)

If a workaround exists, please include it. Customers can implement the event handler and ship audit events to a SIEM or logging system.

rosstimothy commented 2 months ago

This issue is not unique to the Firestore events backend. Similar problems may arise when using Athena depending on the limits configured in the workgroup: https://github.com/gravitational/teleport/issues/41544.

tigrato commented 1 month ago

PR https://github.com/gravitational/teleport/pull/42902 improved the query and now doesn't pull all the events in the [from, to] window.