This pull request introduces support for real-time data enrichment in Matano during ingest, addressing #99 and #21. The new get_enrichment_table_record function has been added to the VRL log transform pipeline, enabling retrieval of enrichment data and adding it to the incoming data stream in real-time, before the detection / lake writing steps.
For many use cases, this feature means users no longer need to perform manual JOINS in their queries or do manual lookups in their detection rules and improves downstream analytics performance by providing pre-joined/enriched records in the data lake and detection engine.
Up next
Next step, will be to add extend support to GeoIP enrichment tables (MaxMind), which will require special handling logic.
Summary
This pull request introduces support for real-time data enrichment in Matano during ingest, addressing #99 and #21. The new
get_enrichment_table_recordfunction has been added to the VRL log transform pipeline, enabling retrieval of enrichment data and adding it to the incoming data stream in real-time, before the detection / lake writing steps.For many use cases, this feature means users no longer need to perform manual JOINS in their queries or do manual lookups in their detection rules and improves downstream analytics performance by providing pre-joined/enriched records in the data lake and detection engine.
Up next
Next step, will be to add extend support to GeoIP enrichment tables (MaxMind), which will require special handling logic.