search

opensearch-project / data-prepper

Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
238 stars 176 forks source link

Add option to shut down and fail data prepper when any sub-pipelines are invalid #4530

Open graytaylor0 opened 1 month ago

graytaylor0 commented 1 month ago

Is your feature request related to a problem? Please describe. The default behavior of Data Prepper core is to only shut down when no sub-pipelines are valid. This can lead to a confusing user experience, as someone who wants all sub-pipelines to be running correctly for data prepper to start.

Describe the solution you'd like An optional parameter in the data-prepper-config.yaml to change this behavior to fail to start Data Prepper when any of the sub-pipelines are not configured properly. This would then throw an exception that stops Data Prepper when any sub-pipelines are misconfigured.

fail_on_any_sub_pipeline_failure: true // defaults to false

Describe alternatives you've considered (Optional) A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

dlvenable commented 1 month ago

@graytaylor0 , Data Prepper has an option for controlling the shutdown behavior already.

pipeline_shutdown: on-any-pipeline-failure

When any pipeline fails, Data Prepper will shutdown when the following is set (it is also the default).

It appears there may be a bug where Data Prepper doesn't honor this on start-up. We should fix that.

Also, Data Prepper doesn't have "sub-pipelines." That is an OSI concept. All pipelines are generally equal and we can have pipelines have no relationship to each other.