[Sandbox] Substation - Githubissues

jshlbrd commented 2 months ago

Application contact emails

jliburdi@brex.com

Project Summary

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.

Project Description

Substation provides three core capabilities:

Deploy cloud native, serverless data pipelines in minutes
Evaluate and transform event logs in real-time
Create custom data processing applications written in Go

Substation is inspired by older data pipeline systems like Logstash and Fluentd, but takes a cloud native approach to deployment and maintenance by deploying directly on cloud services (such as AWS Lambda) with a focus on being highly configurable by end users with features not seen in other systems (robust condition logic, support for external key-value stores, and configurations as code to name a few).

Org repo URL (provide if all repos under the org are in scope of the application)

N/A

Project repo URL in scope of application

https://github.com/brexhq/substation

Additional repos in scope of the application

No response

Website URL

https://substation.readme.io/

Roadmap

https://github.com/brexhq/substation/issues

Roadmap context

The project uses GitHub issues for tracking roadmap progress. In 2022 the project had defined a 2023 roadmap that was implemented by mid 2024. As of now there is no defined roadmap for 2024 (or beyond), but with acceptance to CNCF we can revisit this. The project strictly follows SemVer 2.0 and produces iterative releases (up to once per week) and we also have open discussions about future releases (v1, v2).

IP Policy

[X] If the project is accepted, I agree the project will follow the CNCF IP Policy

Trademark and accounts

[X] If the project is accepted, I agree to donate all project trademarks and accounts to the CNCF

Why CNCF?

The CNCF can provide the project with more exposure to future contributors and cross-organization governance (beyond Brex). There are a number of features that would be beneficial to add to the project (such as support for additional cloud service providers, like GCP or Azure, and out of the box data transform configurations, such as AWS CloudTrail), but the team at Brex can't dedicate time to these additional features.

Without additional support the project will continue to be focused on providing core capabilities built for AWS services.

Benefit to the Landscape

Cloud native event log routing, filtering, and normalization is a challenge faced by many organizations; and interestingly, many organizations that I speak with aren't familiar with the current "state of the art" in this space and are relying on vendors to solve the problem. Today CNCF has one project that partially solves similar problems (Fluentd, mentioned in more detail below), but I think there is room for more than one solution in the CNCF, especially given the differences between these two projects.

Brex has been successfully using Substation in production for more than three years to reduce cost and improve the usefulness of our event logs (and are aware of a few other companies using it for the same use case), more information can be gleaned from blog posts and presentations over the years:

This is a "production-grade" project that has data processing functions not seen in other tools (mentioned above) and that was built with reliability, maintenance, and low-costs as built-in features.

Cloud Native 'Fit'

The project supports many cloud native features and, out of the box, is intended to be deployed on AWS as serverless functions connected to serverless services. This includes:

Deploying compute on Lambda with service support for Kinesis, DynamoDB, S3 (and many more).
Deploying as containers (on Lambda, but can also be built for orchestration platforms).
Designed to run anywhere -- with some effort, new Go applications that utilize the Substation package can be written to run on any cloud service provider.

Cloud Native 'Integration'

Substation does not have any dependencies on existing CNCF projects, but it can complement projects like Cilium and Falco by acting as an event log router, filter, and normalization system for the logs they produce. It can also format audit logs according to the OpenTelemetry SemConv standard.

Cloud Native Overlap

This project overlaps with Fluentd in the sense that both projects can solve similar problems (event log routing and filtering). Where the two projects differ is in their underlying design (Substation is a package written in Go with out of the box support for AWS serverless cloud deployments, and can be extended for deployment in the cloud, on-prem, or locally; Fluentd is an application written in Ruby with out of the box support for deploying on Kubernetes).

Substation is more of a "toolkit" that can be used to implement an event log routing and filtering solution, among other solutions (more info here). Substation was primarily built for security use cases, which is often a superset of observability use cases due to their increased complexity, and was designed for ease of deployment and maintenance ("zero ops").

Similar projects

Landscape

No.

Business Product or Service to Project separation

N/A

Project presentations

TAG Security (9/18/2024): https://github.com/cncf/tag-security/issues/1356#issuecomment-2359844921

Project champions

N/A

Additional information

No response

angellk commented 2 months ago

@jshlbrd please submit a presentation issue to give an overview of the project to TAG Security