jamiehynds commented 7 months ago

Description

Endace is a leading provider of high-speed network packet recording, playback, and analytics solutions. Specializing in network visibility and security, Endace empowers organizations to confidently manage, secure, and optimize their networks. By capturing and analyzing vast amounts of network data in real-time, Endace enables businesses to swiftly detect and respond to security threats, troubleshoot network performance issues, and comply with regulatory requirements.

Architecture

This will differ slightly from our typical ingest integrations. Endance provides full network packet capture, but using Packetbeat's flow support on their probes, allows us to ingest data from a span port on the Endance probes. The goal of the integration is to allow users to pivot from a document with the flow information within Elastic to Endance, in order to deeply investigate the packet data in Endace.

@jamesagarside has started an implementation whereby an Endace integration, based on the Network Packet Capture integration's flow support, can be used as the basis for the Endance integration. The integration will then contain an option to provide the URL for the Endance UI to ensure certain fields (like source/destination IP) can be hyperlinked to Endace's UI.

In the future we can explore additional use cases and data from Endace, but this is a great starting point to add value to Endace adn Elastic's mutual customers. Endace have also agreed to collaborate on dashboards, detection rules, etc.

Endace has similar integrations which can be viewed here: Palo Alto and Splunk

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency when creating or updating a Package, Module or Dataset for an Integration.

All changes

[x] Change follows the contributing guidelines
[x] Supported versions of the monitoring target are documented
[x] Supported operating systems are documented (if applicable)
[x] Integration or System tests exist
[x] Documentation exists
[x] Fields follow ECS and naming conventions
[x] At least a manual test with ES / Kibana / Agent has been performed.
[x] Required Kibana version set to:

New Package

[ ] Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

[ ] Dashboards exists
[ ] Screenshots added or updated
[ ] Datastream filters added to visualizations

Log dataset changes

[x] Pipeline tests exist (if applicable)
[x] Generated output for at least 1 log file exists
[x] Sample event (sample_event.json) exists

elasticmachine commented 7 months ago

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

elasticmachine commented 7 months ago

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

jamiehynds commented 7 months ago

@norrietaylor - @jamesagarside has built an initial integration package with Endace (based on the flow support we have in the Network Packet Capture integration). Is there someone from the Linux platform team that could assist with creating the PR and reviewing James's package?

norrietaylor commented 7 months ago

The Linux team is incredibly resource-constrained right now. To keep this moving, I will add this task to the Deployment and Devices team backlog. We can discuss it during this week's team meeting. Once an engineer is assigned, they can work directly with @jamesagarside to review and merge the PR.

I believe this should be owned by the Linux team moving forward for SDHs and maintenance due to the dependency on packetbeat and similarities to the Network Packet Capture integration. cc/ @nfritts

mjwolf commented 7 months ago

@jamesagarside I'll help you out with this

jamesagarside commented 5 months ago

Update on this, we have an MVP, Im just finalising tests https://github.com/elastic/integrations/tree/endace

jamesagarside commented 5 months ago

@jamiehynds @mjwolf https://github.com/elastic/integrations/pull/10308

PR in however I cant view the Sonarqube test results

mjwolf commented 4 months ago

@jamiehynds @mjwolf #10308

PR in however I cant view the Sonarqube test results

I'll send you the details of the sonarqube report, and review the PR

elastic / integrations

[New Integration] Endace #9579