Adds an HTTP processor that fulfills a variety of GET and POST request use cases
Adds two HTTP processing patterns for interacting with GreyNoise
Adds functionality to retrieve and interpolate secrets from environment variables or AWS Secrets Manager
Adds capability to use a KV Store outside of the internal package
Refactors the HTTP and Sumo Logic sinks
Motivation and Context
Similar to the DNS processor, the HTTP processor lets users enrich data by making HTTP requests (GET or POST). This one is more complex than DNS for two reasons:
HTTP requests to external services usually require authentication
HTTP can host more diverse content than DNS can
We're addressing authentication through a feature referred to as secrets retrieval. Components across the system (conditions, processors, sinks) can opt in to secrets retrieval and interpolate secrets with configuration files during runtime. Here's a summary of how this works:
If the templated variable is found, then it is parsed into a secrets location and secrets name
The location (either env var or Secrets Manager) is queried to retrieve the secret associated with the name
The secret is interpolated with the configuration using string replacement and stored in memory for up to 15 minutes
Secrets retrieval is designed similar to internal/file, so new secrets locations can be added as needed.
We're addressing the diversity of content and use cases by allowing more flexibility in the processor's data and object handling patterns compared to other processors. These are the known use cases it supports, though there may be more:
Integration with any external enrichment service that supports GET and POST requests (intended use case)
Download (GET) text files and insert their value into objects
Download (GET) non-text files (e.g., binary) and overwrite the contents of a capsule
Send (POST) data from objects to an HTTP server without modifying the capsule
How Has This Been Tested?
The changes were integration tested using the two new examples files. The HTTP processor use cases was tested with these external services:
Description
Motivation and Context
Similar to the DNS processor, the HTTP processor lets users enrich data by making HTTP requests (GET or POST). This one is more complex than DNS for two reasons:
We're addressing authentication through a feature referred to as secrets retrieval. Components across the system (conditions, processors, sinks) can opt in to secrets retrieval and interpolate secrets with configuration files during runtime. Here's a summary of how this works:
${SECRETS ...}(the same design is used in other regexp-based libraries) to identify secrets inside of configurationsSecrets retrieval is designed similar to internal/file, so new secrets locations can be added as needed.
We're addressing the diversity of content and use cases by allowing more flexibility in the processor's data and object handling patterns compared to other processors. These are the known use cases it supports, though there may be more:
How Has This Been Tested?
The changes were integration tested using the two new examples files. The HTTP processor use cases was tested with these external services:
Types of changes
Checklist: