search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
3.13k stars 432 forks source link

Add `parse_json`, `parse_jsonl`, `parse_xml`, and `parse_ini` tables to Fleet's agent (fleetd) #17577

Closed nonpunctual closed 7 months ago

nonpunctual commented 8 months ago

Goal

User story
As an endpoint operator,
I want to write a query that pulls content from JSON files on my hosts
so that I can understand the contents of specific JSON files.

Context

Changes

Product

Engineering

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

Manual testing steps

  1. Step 1
  2. Step 2
  3. Step 3

Testing notes

Confirmation

  1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
  2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.
noahtalerman commented 8 months ago

Thanks Brock! Let's bring this to feature fest.

FYI @erikng I think you might also be interested in this table.

erikng commented 8 months ago

I very much am. There's are many json files, including for tools like Nudge, where we want to understand the contents of these files.

zwass commented 8 months ago

@erikng you can use the file_lines table and sqlite json_extract to do this today, but these tables might make things a bit easier once we ship them.

erikng commented 8 months ago

Yeah I find that too cumbersome which is why we use kolide's json table at this time.

nonpunctual commented 8 months ago

@noahtalerman will we have to add a comment like this when this feature is implemented?

Screenshot 2024-03-18 at 5 15 30 PM
noahtalerman commented 8 months ago

@nonpunctual I don't think we have to add that note. I think we can remove it from other tables.

nonpunctual commented 8 months ago

Noted: as I go through all of them I will clean up these references. Thanks!

nonpunctual commented 7 months ago

@noahtalerman this issue can't be assigned to me just for the task of cleaning up these references. We still need to implement the json table as it is implemented in kolide osquery. Zach said this change would be trivial. Please see the go file referenced above. Thanks.

noahtalerman commented 7 months ago

this issue can't be assigned to me just for the task of cleaning up these references.

Hey @nonpunctual, I assigned the issue to you so that I know that you're the requester during feature fest.

I will unassign you when I clean up the feature fest board.

nonpunctual commented 7 months ago

Summarizing the case for this FR. Please consider for prioritization in the next sprint:

zwass commented 7 months ago

FWIW if there's agreement to do this I could probably implement it in under an hour.

noahtalerman commented 7 months ago

@zwass let's ship it!

I added the user story template, assigned you, and moved it to the drafting board (:product label.)

I moved the original issue description here:

I discussed https://github.com/fleetdm/fleet/issues/12008 with Zach at his office hours today. The main reason to reopen is to add the capability to expose arbitrary json as tables with fleetd osquery.

seph from Kolide said in the osquery Slack Workspace that they have implemented a kolide_json and kolide_jsonl table.

@zwass believes this would be trivial to implement (but please correct that understanding if it's wrong...) Here is the file in the repo he was looking at: https://github.com/fleetdm/fleet/edit/main/orbit/pkg/table/extension_darwin.go

From @noahtalerman At the time, we thought kolide_json and kolide_jsonl tables were just dependencies for other tables. Agree w/ Luke that opening a new story and brining to feature fest is the way to go!

This feature would be simliar in terms of use cases to the files table & plist table & would be useful on all platforms (eg, plist is an Apple format...)

There are limited native tools for parsing json on macOS. sqlite3 & plutil binaries are available but this feature would allow Fleet to handle json data in a way that other MDM vendors do not offer & would theoretically prevent customers from having to deploy something like Joel Brunner's ljt or the jq binary which they may not be allowed to do.

Converting data to other formats or deploying 3rd party tools.

Problem

TODO

Enhance platforms capabilities to include fleetd osquery json parsing.

Potential solutions

  1. add serialization formats at the following spot in the repo? https://github.com/fleetdm/fleet/edit/main/orbit/pkg/table/extension_darwin.go
  2. @zwass also mentioned that other serializations could possibly also be added there like xml?

Thanks!

nonpunctual commented 7 months ago

Thanks everyone. If it's as easy as @zwass says (fingers crossed) this is a simple win.

noahtalerman commented 7 months ago

We moved this story to "Awaiting QA" on the release board since we're preparing to release a new fleetd (1.23) w/ the changes for this story.

cc @lucasmrod @lukeheath @zwass

lucasmrod commented 7 months ago

Smoke tested the four tables on Ubuntu 22.04, Windows 10 and macOS 14.4.1. Now proceeding with pushing the new fleetd update to the edge channel.

lukeheath commented 7 months ago

@noahtalerman Because this was a feature, per our documentation, it should not have merged during freeze without approval. This merged without approval from the EM, QA, and release ritual DRI, which are all required. Please ensure any code changes are tracked on the board and follow our standard Kanban and approval processes regardless of who is working on the ticket.

zwass commented 7 months ago

Hey @lukeheath thanks for commenting on this and I completely agree with the sentiment. I'm not sure, but IIRC I didn't skip the freeze to merge it. Perhaps it was accidentally turned off at the time?

zwass commented 7 months ago

@lukeheath I filed an issue to investigate: https://github.com/fleetdm/fleet/issues/18178

Should I assign to you?

lukeheath commented 7 months ago

@zwass thanks for the heads up! It's possible that the Merge Freeze workflow failed to disable merge. It did the same thing for one of Gabe's PRs last week, which resulted in an accidental merge. If you don't recall unfreezing, I expect that's what happened.

@roperzh put together a POC to block merges using our label system. We have an engineering-initiated story on the sprint board to track the effort, but it's been getting pushed the last few weeks due to higher priority work.

@georgekarrv since merge freeze has been causing some operational pain recently, will you please work with @roperzh to clear some time for him to complete this story?

lukeheath commented 7 months ago

In light of the above, I realize my initial comment came across as a bit presumptive. My apologies, y'all!

fleet-release commented 7 months ago

Parsing JSON, XML, INI files bring clarity, Knowledge unfurls like dawn.