Deploy YARA rules remotely and privately

zayhanlon commented 1 year ago

User story
As a detection & response engineers,
I want to deploy YARA rules to agents remotely and privately from a server I host myself (separate from Fleet)
so don't have to write rules to disk (too large of scope and too slow) or host rules on a non-private webserver (what osquery supports today).

Too large of scope: If they wrote the rules to disk, any engineering (including outside detection & response team) could read these rules because they'd live in the organization's monorepo. If an engineer's account gets compromised then they can read these rules.
Too slow: It takes months to get these rules reviewed.
Privately: Viewing rules must require authentication. Currently, osquery doesn't provide a way to authenticate YARA rules.

Context

Product designer: @zwass

Changes

Product

[x] No Fleet UI/CLI/API or fleetd changes
[x] osquery changes: Support deploying YARA rules to agents remotely behind authenticated endpoints

Engineering

@lucasmrod, please determine if any of the following is needed/done and update accordingly.

[ ] Reference documentation changes: TODO
[ ] Usage documentation changes: TODO
[ ] Database schema migrations: TODO
[ ] Load testing: TODO

ℹ️ Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

Requires load testing: TODO
Risk level: Low / High TODO
Risk description: TODO

Manual testing steps

Step 1
Step 2
Step 3

Testing notes

Confirmation

[ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
[ ] QA (@____): Added comment to user story confirming successful completion of QA.

noahtalerman commented 1 year ago

Feature fest: We have a yara table in Fleet: https://fleetdm.com/tables/yara

Does that address the need?

zayhanlon commented 5 months ago

Related to an ask from customer-domon re: #19553

JoStableford commented 5 months ago

Related to a Slack conversation

nonpunctual commented 5 months ago

The yara & yara_events tables do not have the capabilities to access sigrules from a remote server securely.

zayhanlon commented 4 months ago

@noahtalerman let me know if you or design team would like to chat with the customer during or after this air guitar. thanks!

noahtalerman commented 4 months ago

Hey @zayhanlon! Would love some help setting up a call w/ the customer.

noahtalerman commented 4 months ago

Containers supported by YARA (Specifically Kubernetes): Brock: Need context → Adding namespace column by PID would allow you to use YARA rules in containers kubequery in the repo may be helpful May be a blocker since they don’t have direct Kubernetes API Brock: Could pass along to infra team to use Ben: Potentially, could be hard though Concern was if they could use osquery to run YARA rules in containers that are Docker → Now use Kubernetes so want that same ability for Kube osquery does not support containerd → this would resolve the issue TODO Brock: Bring this to eng to discuss Found article about osquery support for containerd https://developer.ibm.com/articles/monitoring-containers-osquery/

nonpunctual commented 4 months ago

Hey @noahtalerman I did reach out to IBM on their GitHub to see if they had open-sourced anything in relation to this. Got no response. That said, I think we could duplicate what was built based on the article.

noahtalerman commented 4 months ago

Hey @zayhanlon, I updated this air guitar to the user story format and moved our original issue description below.

Bringing this one back to feature fest to weigh whether we'll commit to solving this one next sprint or in a later sprint.

Original issue description:

Ideal workflow: Customer would have a repo of YARA rules that only detection & response team can access. Deploy these per team (production servers and workstations).

Some tables are blocked by not having YARA

Future: How to displace CrowdStrike? Event monitoring. osquery evented tables aren't as good and performant as CrowdStrike.

Problem

Background:

Currently not using YARA scanning at all today
One option - put the rules on disk, it would take forever. If there was an incident and they needed to add new rules, that timeline wouldn’t work. Not a tenable option.
Another option - pull from a web server, but then it would need to be publicly accessible. Then attackers could then see what their rules are.
DART team currently has a task to build a YARA scanning capability. Osquery provides this capability but the methods to implement are not ideal for our environment

Potential solutions

Build the feature for Fleet to be given a set of YARA rules like Osquery query packs. Next, build the feature for Osquery to pull those Osquery rules from the TLS server.
Where would this live in Fleet?
- Same as query packs
- Would be able to be configured as ‘YARA events’ scheduled to run at a recurring configurable interval of frequency
Benefit to them: YARA rules would be the method of distributing out the rule and detecting the presence of new malware signatures out in their environment
- Crowdstrike doesn’t support YARA rules today
- Would make it easier for other DART teams to use this feature (on a moments notice, be able to write a rule for new

zayhanlon commented 4 months ago

Thank you @noahtalerman

noahtalerman commented 3 months ago

Hey @zwass, I'm passing this story to you and tagged you as the product designer.

Please bring this story + your proposed changes to the design review ritual so that we can provide a second pair of eyes on changes and give feedback.

Thanks!

cc @zayhanlon

noahtalerman commented 2 months ago

Hey @zayhanlon because this is being worked on by customer success I think it makes sense to move this issue to the #g-customer-success board. I moved it.

When @zwass brings changes through design review and we decide that it's ready to estimate, I think at that point we add the issue back to the drafting board (:product) so that we can bring it through a product group's estimation session.

Please let me know if you disagree!

zayhanlon commented 2 months ago

sounds good! i was tracking it separately, but we can follow your guidance. this will be zach's next issue after he's back from leave.

zwass commented 1 month ago

Osquery support

I put up a PR to allow osquery to authenticate yara requests: https://github.com/osquery/osquery/pull/8437. I'm working to get this into the 5.14 release this week. With this flag enabled, osquery will send the node key when retrieving yara rules, which will allow the Fleet server to authenticate the request before responding.

UX in Fleet

A very minimal PoC has been implemented so far and this UX will be fully implemented once it has been reviewed.

Configure agent options:

config:
  options:
    # other options ommitted
    yara_sigurl_authenticate: true # "on" switch for using YARA rules in Fleet
  yara:
    signature_urls:
    - https://localhost:8080/api/osquery/yara/.*  # (Fleet server URL) Also required for using YARA rules in Fleet

Configure yara URLS in default.yml:

org_settings:
  yara_rules:
    - path: ../lib/test1.yar
    - path: ../lib/test2.yar

Note: The filenames must be unique, as the filename becomes the last part of the path used in the sigurl. Fleet will throw an error if the user attempts to use multiple files with the same name.

(For now the yara rules will be configured at the global level. For targeting by team, target the live or scheduled queries to the appropriate team.)

Reference the rules in queries (live or scheduled):

SELECT * FROM yara WHERE path="/bin/ls" AND sigurl='https://localhost:8080/api/osquery/yara/test1.yar'

Osquery will then send an authenticated request. If Fleet considers the node key valid it will respond with the appropriate rule file.

noahtalerman commented 1 month ago

@zwass looks good! When you get the chance, can you please open a PR to reference docs branch w/ the proposed YAML file changes? Here's the YAML reference docs: https://fleetdm.com/docs/configuration/yaml-files

Also, can you please include the proposed API changes to /config endpoints (I think PATCH and GET) in that PR?

noahtalerman commented 1 month ago

Hey @sharon-fdm, @zwass is helping out the Endpoint ops team by building this user story.

I added this story to the release board so that we can treat it like any other story (in progress => in review => ready for QA => ready for release).

cc @lukeheath @zayhanlon

noahtalerman commented 1 month ago

As a user writing a query, if the YARA rule doesn't exist, the query will return no results and a helpful error message will be logged in the osquery status logs.

As a user hosting YARA rules outside of Fleet, to migrate to YARA rules with Fleet, they would have to add yara_sigurl_authenticate and signature_urls to their agent options. This would turn on the feature.

If they forget to add the right URL to signature_urls, then they will run into the no results + error in status logs (scenario above).

If they forget to add the yara_sigurl_authenticate, the user will get a 403 error.

@noahtalerman: validation for signature URLs?

Hey @zwass, I think this^ would be a good thing to document in the guide.

sharon-fdm commented 1 month ago

@noahtalerman, @lukeheath, this has a long history and is currently on our release board. Should we give it a P2 / reprioritize / other ?

lukeheath commented 1 month ago

@sharon-fdm It looks like @zwass is owning this story and tracking on the customer success board. Noah left it on the endpoint ops board so that it could go through the normal QA and release process. But, nothing for you to track on this until it's ready for QA.

noahtalerman commented 1 month ago

Hey @zwass can you please open a PR against the reference docs branch (assuming this story will be shipped in 4.59) w/ the proposed YAML file and REST API changes?

YAML reference docs: https://fleetdm.com/docs/configuration/yaml-files

API reference docs: https://fleetdm.com/docs/rest-api/rest-api

fleetdm / fleet