Add new tables for threat hunting and detection & response

[dherder]

• prospect-seidel: Queries they run today:
  • https://drive.google.com/file/d/1ZVxLuOF3zGibMK-B5qwhGIIWctorB1Nx/edit
  • https://drive.google.com/file/d/1djtl0p-OBfjEAdGbHrsfpG4wy2Y-rvEQ/edit
• @noahtalerman: User requested this b/c they want to use Fleet but they can't run all the queries they run today in their current solution. Their current solution is used for threat hunting and detection & response.
• @noahtalerman: What tables is Fleet missing?
  • dns_lookup_events
  • ranked_addresses
  • upt_asset_interfaces
  • upt_assets
  • ranked_extensions
  • http_events
  • chrome_download_history
  • edge_url_history
  • firefox_url_history
  • windows_recent_files
  • chrome_url_history
  • edge_download_history
  • firefox_download_history
• @zwass: I'm not certain that ranked_addresses, upt_asset_interfaces, and upt_assets, are tables that need to be built. We need to get a better understanding of what these queries are doing as they may be able to be satisfied with interface_details and interface_addresses.

• @zwass: ranked_extensions is not actually a table it's just created in the query that uses it. I think that the existing extension tables should work? Not 100% sure.

User stories

• 24198

[noahtalerman]

Thanks @dherder. This is a request from security right? (not IT) Just double checking.

[dherder]

If we can't add all the tables, these are the most important: see https://github.com/shawnhank/osq-ext-bin/blob/master/tables-schema/win_dns_events.table https://github.com/shawnhank/osq-ext-bin/blob/master/tables-schema/win_dns_response_events.table https://github.com/shawnhank/osq-ext-bin/blob/master/tables-schema/win_file_events.table https://github.com/shawnhank/osq-ext-bin/blob/master/tables-schema/win_http_events.table https://github.com/shawnhank/osq-ext-bin/blob/master/tables-schema/win_registry_events.table https://github.com/shawnhank/osq-ext-bin/blob/master/tables-schema/win_removable_media_events.table

[noahtalerman]

What tables is Fleet missing?

• dns_lookup_events
• ranked_addresses
• upt_asset_interfaces
• upt_assets
• ranked_extensions
• http_events

[noahtalerman]

Problem

File auditing events are supported on Linux and macOS. Recently, we added process auditing on Windows. Additional work is required to add file, http, and DNS auditing on Windows. See https://docs.google.com/document/d/18HKASG9x6YY68ACp4RJR-wBKaXXtz4Et6h1yOWjLVSE/edit?usp=sharing

Ideally, we would add all of the Windows evented tables from https://github.com/shawnhank/osq-ext-bin/tree/master/tables-schema

[zwass]

I just heard a bit about this from @ambrusps. Some thoughts:

1) Unfortunately we can't use that "osq-ext-bin" as it's not open source. That was built by https://www.linkedin.com/in/atulkabra/ and his team at his former startup, Polylogyx.

2) File, DNS, and HTTP auditing on Windows are likely pretty significant work (potentially a week or more each, but I'm not super familiar with the domain). The good news is that I think the work that Marcos did to make it easier to build ETW-event based tables in osquery should be a help on each of these. He also provided some more context on the possibilities.

3) I'm not certain that ranked_addresses, upt_asset_interfaces, and upt_assets, are tables that need to be built. We need to get a better understanding of what these queries are doing as they may be able to be satisfied with interface_details and interface_addresses.

4) ranked_extensions is not actually a table it's just created in the query that uses it. I think that the existing extension tables should work? Not 100% sure.