Osquery Hunting notebook

[anelshaer]

Goal

One of the use cases for osquery is to interrogate remote systems and do a hunt for bad signals.

Having a notebook feature like jupyter notebook but for osquery

• provide a playbook for specific investigation with all queries for that case and markdown to explain them
• document your investigation process
• Non experienced users can just run that notebook and learn from it
• Share with the community single notebook with all queries
• Ability to run that notebook against one or group of hosts
• save lots of time jumping between queries to run because you cant run a query pack with a single button on specific machine or host label/group

  Solution

Fleet notebook; Osquery queries that is able to run against one or group of hosts.

[noahtalerman]

@anelshaer I like your proposed community notebook idea.

I think what I describe below shares some similarities with your proposal.

Over the last several weeks, Fleet has been developing what we're currently calling our Standard Query Library. The goal of this library is to provide a central location for what we think are common and useful queries for organizations deploying Fleet and osquery. The Standard Query Library can be found in the fleetdm/fleet repo here in /handbook/queries.

Like I called out earlier, your proposal has some differences with Fleet's current solution. I think a good way to potentially reach an even better solution would be to see what you've envisioned first.

It would be awesome to review a pull request from you that:

• Adds a section to /handbook/queries. Please use whatever formatting you like
• Includes an example playbook, queries, and investigation process

@anelshaer please let me know if this is something you're interested in doing.

[anelshaer]

@noahtalerman I have actually seen the standard library i have couple points on that actually.

• its good as start and give examples
• Queries included in the markdown which means there are no easy import into fleet
• they are not included into fleet as well

The notebook on the other hand or how i envisioned it;

• very similar to jupyter notebook where you have markdown cells to and cells where you write osquery queries.
• Notebook can run on host or group of hosts; this is kind of running a pack of queries not just single query and you get results back
• export and import notebooks into fleet
• Share notebooks with community

Because you already have created something, i will be opening couple PR for queries but in yaml format so they can be imported by users along with md file similar to what you have. Then i can build notebook mock up just markdown to show case the idea.

[anelshaer]

here is an on PR, i suggest we create sub directories for queries and markdowns, so all queries are importable. another idea is to have the key info in yaml as well.

[noahtalerman]

@anelshaer first, thank you for the feedback on the Standard Query Library and thank you for contributing!

• Queries included in the markdown which means there are no easy import into fleet
• they are not included into fleet as well

Making the queries easy to import into Fleet is something we'd like to improve. The next step after this would be including the queries automatically in Fleet.

• Notebook can run on host or group of hosts; this is kind of running a pack of queries not just single query and you get results back

Are these notebooks meant to be run ad-hoc as live queries?

I'm less familiar with Jupyter Notebook and it would be great to review a PR with a notebook mockup.

[anelshaer]

notebook is mean to be ad-hoc like live queries, for example you have suspicious activity on a machine, you want to run set of queries, like what are the user created, process running, cron jobs running etc on that machine. this notebook will help you with all that.

i will write something and push a PR.

[noahtalerman]

for example you have suspicious activity on a machine, you want to run set of queries, like what are the user created, process running, cron jobs running etc on that machine. this notebook will help you with all that.

Got it. I'm guessing the playbook will also answer the "when is it important to run these queries?" and the "what are the next steps I take after I run these queries?"

@anelshaer looking forward to this PR!

[anelshaer]

yes, your are totally right.

its basically contains Markdown section to describe everything like objective, and your analysis steps and next remediation steps like any markdowns, and other cells that contains actual code (osquery queries in our case) that runs agains the host and prints the data under.

searched for online notebooks and here is an example till we create a mockup https://github.com/grokcode/ipython-notebooks/blob/master/nginx-log-analysis.ipynb

[noahtalerman]

A Fleet customer described an investigation workflow that reminded me of this issue.

My understanding of the investigation workflow is the following:

1. In a Jupyter Notebooks, analyst defines several osquery queries. When an alert in Panther fires, these queries will be used to answer "Is this a valid alert or is the alert a false positive or false negative?"
2. In Panther, alter fires
3. In Jupyter Notebooks, analyst runs the queries using an API-only user
4. Analyst answers the above question

The customer is concerned with the level of access given to the API-only user. They only want this API only user to be able to run queries.

Immediately, the customer can update the API only user's role from admin to maintainer: https://fleetdm.com/docs/using-fleet/permissions. This way, the API-only user cannot modify Fleet users and settings.

Next, to further reduce the API only user's permissions, the customer can update the investigation workflow to the following:

1. In Fleet, analyst saves several osquery queries. These queries are marked as Observer can run.
2. In Jupyter Notebooks, analyst references the saved queries from (1). When an alert in Panther fires, these queries will be used to answer "Is this a valid alert or is the alert a false positive or false negative?"
3. In Panther, alert fires
4. In Jupyter Notebooks, analyst runs the queries using an API-only user
5. Analyst answers the above question

This way, the API-only user's role can be updated to observer. Observers cannot modify users, settings, or existing queries.

If the analyst has the maintainer or admin role in Fleet, they own the responsibility of pre-defining the queries. Another way to do this is to add queries to a git repository review them, and add them to Fleet via CI/CD (GitOps).

[anelshaer]

Thanks Noah for reminding me 😅 i actually mentioned this to Mike McNeil.

I’m glad that my feature request years ago are still valid and coming to you from other customers.

A made something specifically for that: https://github.com/anelshaer/Remote-Linux-Triage-Collection-using-OSquery

On Tue 18. Oct 2022 at 6:32 PM Noah Talerman @.***> wrote:

A Fleet customer described an investigation workflow that reminded me of this issue.

My understanding of the investigation workflow is the following:

1. Analyst defines several osquery queries in a Jupyter Notebooks. When an alert in Panther fires, these queries will be used to answer "Is this a valid alert or is the alert a false positive or false negative?"
2. Alert fires in Panther
3. Analyst runs the queries defined in Jupyter Notebooks using an API-only user
4. Analyst answers the above question.

The customer is concerned with the level of access given to the API-only user. They only want this API only user to be able to run queries.

Immediately, the customer can update the API only user's role from admin to maintainer: https://fleetdm.com/docs/using-fleet/permissions. This way, the API-only user cannot modify Fleet users and settings.

Next, to further reduce the API only user's permissions, the customer can update the investigation workflow to the following:

1. Analyst defines several osquery queries in Fleet. These queries are marked as Observer can run.
2. Analyst references these queries in Jupyter Notebooks. When an alert in Panther fires, these queries will be used to answer "Is this a valid alert or is the alert a false positive or false negative?"
3. Alert fires in Panther
4. Analyst runs the queries defined in Jupyter Notebooks using an API-only user
5. Analyst answers the above question.

This way, the API-only user's role can be updated to observer. Observers cannot modify users, settings, and existing queries.

— Reply to this email directly, view it on GitHub https://github.com/fleetdm/fleet/issues/722#issuecomment-1282675515, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7F5JJ6BDNY24VCCN5GKYTWD3GJXANCNFSM44DZAV7A . You are receiving this because you were mentioned.Message ID: @.***>

[noahtalerman]

A made something specifically for that: https://github.com/anelshaer/Remote-Linux-Triage-Collection-using-OSquery

Awesome! Thank you for sharing. Very cool that Ansible and osquery are the only dependencies.