search

opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
267 stars 206 forks source link

[BUG] Routes: regex doesn't work #4763

Closed vasily-yanov-draivn closed 3 months ago

vasily-yanov-draivn commented 4 months ago

Describe the bug

create pipeline with a route:

            - dev-nginx-logs: '/kubernetes/pod_name =~ ".*nginx.*" and /k8scluster == "dev-cluster"'
            - dev-app-logs: '/kubernetes/namespace_name == "my-app-dev"'

but when DP starts I see:

2024-07-24T14:52:59,685 [logs-pipeline-processor-worker-1-thread-3] ERROR org.opensearch.dataprepper.pipeline.router.RouteEventEvaluator - Failed to evaluate route. This route will not be applied to any events.
org.opensearch.dataprepper.expression.ExpressionEvaluationException: Unable to evaluate statement "".*nginx.*" and /k8scluster == "dev-cluster""
        at org.opensearch.dataprepper.expression.GenericExpressionEvaluator.evaluate(GenericExpressionEvaluator.java:44) ~[data-prepper-expression-2.8.0.jar:?]
        at org.opensearch.dataprepper.expression.ExpressionEvaluator.evaluateConditional(ExpressionEvaluator.java:30) ~[data-prepper-api-2.8.0.jar:?]
        at org.opensearch.dataprepper.pipeline.router.RouteEventEvaluator.findMatchedRoutes(RouteEventEvaluator.java:64) [data-prepper-core-2.8.0.jar:?]
        at org.opensearch.dataprepper.pipeline.router.RouteEventEvaluator.evaluateEventRoutes(RouteEventEvaluator.java:45) [data-prepper-core-2.8.0.jar:?]
        at org.opensearch.dataprepper.pipeline.router.Router.route(Router.java:44) [data-prepper-core-2.8.0.jar:?]
        at org.opensearch.dataprepper.pipeline.Pipeline.publishToSinks(Pipeline.java:346) [data-prepper-core-2.8.0.jar:?]
        at org.opensearch.dataprepper.pipeline.ProcessWorker.postToSink(ProcessWorker.java:168) [data-prepper-core-2.8.0.jar:?]
        at org.opensearch.dataprepper.pipeline.ProcessWorker.doRun(ProcessWorker.java:150) [data-prepper-core-2.8.0.jar:?]
        at org.opensearch.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:61) [data-prepper-core-2.8.0.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.opensearch.dataprepper.expression.ExpressionEvaluationException: Unable to evaluate the part of input statement: /kubernetes/pod_name =~ ".*nginx.*"
        at org.opensearch.dataprepper.expression.ParseTreeEvaluator.evaluate(ParseTreeEvaluator.java:41) ~[data-prepper-expression-2.8.0.jar:?]
        at org.opensearch.dataprepper.expression.ParseTreeEvaluator.evaluate(ParseTreeEvaluator.java:17) ~[data-prepper-expression-2.8.0.jar:?]
        at org.opensearch.dataprepper.expression.GenericExpressionEvaluator.evaluate(GenericExpressionEvaluator.java:41) ~[data-prepper-expression-2.8.0.jar:?]
        ... 13 more
.......
Caused by: java.lang.IllegalArgumentException: '=~' requires left operand to be String.

To Reproduce

  1. Create pipeline with route provided above
  2. Check

Expected behavior route events to indexes according route rules

Environment (please complete the following information):

dlvenable commented 4 months ago

From my investigation of this, the problem is that the regex operation fails when the key is not found in the event. Basically, when /kubernetes/pod_name is null, this throws a failure. I will create a PR for this.