search

kolide / fleet

A flexible control server for osquery fleets
https://kolide.com/fleet
MIT License
1.1k stars 261 forks source link

feature: add ability to filter additional host info #2329

Open billcobbler opened 4 years ago

billcobbler commented 4 years ago

I know the recommendation is to keep the additional_queries to a minimum (to limit performance impacts). However, we have a potential need to be able to filter the additional fields returned from the /api/v1/kolide/hosts endpoint.

I have proof of concept PR I'm about to submit, but I wanted to open this issue for feedback on the idea in general.

cc: @nyanshak

zwass commented 4 years ago

The recommendation exists because the implementation of additional_queries is not optimized or profiled for high volumes of data.

Are you using it successfully with a large volume of data/hosts? It would be interesting to understand (1) what your use case is and (2) how your infrastructure is configured and how it is performing.

Overall I'm not opposed to the idea of allowing filtering on the additional info, but I think it may be necessary to continue iterating on the concept of additional_queries based on what we learn of real-world uses.

billcobbler commented 4 years ago

We're considering a use case to integrate Fleet with a vuln scanner and provide the vuln scanner with detailed host info such as packages installed on hosts. The goal is to remove the need to run another security agent on our hosts. We haven't enabled additional queries on our real fleet, but we did a load test with 20k ubuntu clients to get feel for performance impact. We used additional queries to pull deb packages with deb_packages: SELECT * FROM deb_packages.

Rough performance we saw at 20k clients (5m config refresh) including API calls to pull ~1k hosts at a time (without this filtering feature): EC2: 2x c5.large, each at ~15% cpu at steady state RDS: db.m5.large, at ~22% cpu at steady state Redis: cache.t2.micro at < 5% cpu at steady state

That lead us to feel it was feasible to use additional queries to gather this data, but that filtering the results would be nice to limit the amount of data sent/received by the frontend (or anything else that doesn't care about additional info).