search

jmpsec / osctrl

Fast and efficient osquery management
https://osctrl.net
MIT License
384 stars 51 forks source link

Refactor ingested data metrics #547

Open zhuoyuan-liu opened 1 week ago

zhuoyuan-liu commented 1 week ago

During our test, we found that the osctrl logs the size of each successful request to the database. I would suggest switching to Prometheus metrics with a good overview and much less operational costs.

If we want to keep the current mode that we log every request with the environment and device id, I would suggest moving this to each of the log plugins. This is because some of the users may not want to use the same database to store the query results and these statistical logs. It also reduces the write request to the main/shared database.

javuto commented 4 days ago

I think it makes total sense to track that using Prometheus, which can inspect the received request via its middleware, and without any impact to the backend with read or writes.