search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
197 stars 427 forks source link

[Meta][Amazon Security Lake] Supporting OCSF v1.1 #9607

Open jamiehynds opened 5 months ago

jamiehynds commented 5 months ago

Our current Amazon Security Lake supports OCSF v1.0, which was the latest version of the schema when we initially shipped the integration. The OCSF schema has evolved since, and is now at v1.1.

Our Security Lake pipelines need to be adjusted to ensure we're inline with v1.1, including new event classes, objects and categories. Backward compatibility is not a significant concern in this case, as an example our security findings pipeline can be deprecated, as security findings were deprecated in OCSF v1.1. Related dashboards will also need to be removed, and new ones added to account for new classes introduced in v1.1

Once we have updated pipelines to support the latest OCSF version, we'll create an issue to build a generic OCSF to ECS package, for users who would like to ingest OCSF formatted data outside of Security Lake.

For a full list of v1.1 changes we need to adhere to, please see here: https://github.com/ocsf/ocsf-schema/blob/main/CHANGELOG.md#v110---january-25th-2024

Update: OCSF v1.2 is now available - https://github.com/ocsf/ocsf-schema/blob/main/CHANGELOG.md#v120---april-23rd-2024 - will let @ShourieG decide if we make the necessary changes for both 1.1 and 1.2 in this issue, or create a separate isseu for v1.2 

### Tasks
- [x] Add support for system tests
- [x] Add 3 new events types in the same data stream (with separate mappings)
- [ ] https://github.com/elastic/integrations/issues/10219
- [ ] https://github.com/elastic/integrations/issues/10266
- [ ] https://github.com/elastic/kibana/issues/187951
- [ ] https://github.com/elastic/elastic-package/issues/1917
- [ ] https://github.com/elastic/integrations/issues/10740
elasticmachine commented 5 months ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

ShourieG commented 5 months ago

@jamiehynds, as discussed we are initially starting the upgrade to OCSF v1.1 and targeting the release by 8.15. Post the upgrade to v1.1 we will start with the upgrades for supporting v1.2.

ShourieG commented 3 months ago

We are hitting the upper limit of 2048 fields per data stream while performing the ocsf v1.1 upgrade due to addition of new object types to base objects. I've discussed with the ecosystem team and an issue has been opened for addressing this limitation in future. Right now we might have to keep these new objects in a flattened state to bypass these limitations. cc: @jamiehynds @andrewkroh