Separate file size options for query results vs. agent logs when using filesystem storage

[zhumo]

Goal

User story
As an endpoint operator,
I want to separate the retention settings for query results and status logs
so that I can compress result logs and not compress status logs.

✅ Choose different file sizes for automated query results and agent logs.

Context

• Requestor(s): @ksatter
• Product designer: @rachaelshaw

Changes

Product

• [ ] Config changes: Figma
• [ ] Outdated documentation changes: Update Fleet server configuration docs
• [ ] Changes to paid features or tiers: Available in Fleet Free and Fleet Premium

Engineering

• [ ] Database schema migrations: TODO
• [ ] Load testing: TODO

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: TODO
• Risk level: Low / High TODO
• Risk description: TODO

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.

[zayhanlon]

Hey @ksatter , we’re de-prioritizing this issue because it’s not likely to be worked on in the next 6 weeks. Please feel free to bring it back in a future product feature requests meeting. Can you please remember to surface this at PFR again in like 6 weeks so we remember to bring it back on the board?

[noahtalerman]

Zay: #2 on the list

[noahtalerman]

Feature fest: Let's take this as an airguitar.

[noahtalerman]

TODO: Bring this into the next design sprint (kicks off ~Dec 15)

[noahtalerman]

From the customer:

• Result logs are 100s of MB. We don't have to use up as much disk space so if there's downtime we won't lose data.
• Right now, neither are compressed.
• If agent logs get compressed, we have uncompress them and move them before they reach the system that ingests the logs.

1. Relevant config settings
   1. FLEET_FILESYSTEM_MAX_SIZE
   2. FLEET_FILESYSTEM_MAX_AGE
   3. FLEET_FILESYSTEM_MAX_BACKUPS
   4. FLEET_FILESYSTEM_ENABLE_LOG_COMPRESSION
2. Update these to have a log version and a results version
3. What to do about backwards compatibility? Potential solution: The existing options remain applying to both, but the new more-specific set of options would override the old general option.

[noahtalerman]

@zayhanlon heads up, this didn't make the 3 week drafting timeline so we're removing it from the drafting board. Bringing back to feature fest.

[noahtalerman]

Gong call w/ the customer: https://us-65885.app.gong.io/call?id=2408209438823448017

At some point during this call we chatted about what they expect to use this feature for.

cc @rachaelshaw

[noahtalerman]

Noah: What’s happening today?

Customer: Neither are compressed. Which means we added a lot of hard drive space and complex infra on our Fleet instances. Result logs take up most of the space.

Customer:

• Result logs go to custom python code that decides whether to send to Splunk or Kafka
• ebpf goes to Kafka
• Everything else goes to Splunk
• Status logs go straight Splunk

Noah: Why not compress agent logs too?

Customer:

• Logs get processed by different agents.
• The agent processing these logs doesn’t know how to compress
• We get more hardware space if we have this freedom and can compress query results
• If we have a bug in our pipeline or Kafka goes down we can hold more data while we resolve the issue w/o having to worry about losing data
• In addition it would be easier to move to a different infra if we have less complex hardware

[noahtalerman]

@pintomi1989 heads up, we didn't get to this feature request in the current design sprint.

Bringing to feature fest.

[ksatter]

@nonpunctual, bringing this request back.

[ksatter]

@nonpunctual belay that, I didn't look closely enough.

[nonpunctual]

Okiedokie. 🥇