Testing Observability Pipeline

[francisan]

Hi,

How I can test below pipeline using some sample payload. Want to test/ensure connector routes the request to corresponding pipeline based on the field 'name'.

receivers:
  webhookevent:

      endpoint: 0.0.0.0:1111
      path: /get/info

processors:
  transform/tr:

       error_mode: ignore
       log_statements:
        - context: log
          statements:
           - set(body, ParseJSON(body)) where body != nil
           - set(resource.attributes["name"], body["name"])

connectors:
  routing :
      table:
        - statement: route() where attributes["name"] == "abc"
           pipelines:
            - logs/abc
        - statement: route() where attributes["name"] == "xyz"
           pipelines:
            - logs/xyz            

exporters:
 logging:
   loglevel: debug

service:
  pipelines:

  logs/receivers:
    receivers: [webhookevent]
    processors: [transform/tr]
    exporters: [routing,logging]

  logs/abc:
    receivers: [routing]
    processors: []
    exporters: [logging]

  logs/xyz:
    receivers: [routing]
    processors: []
    exporters: [logging]

Want to build something as shown below. Any existing test frameworks available?

type: Test
spec:
  id: 19L__G7IR
  name: Call API
  trigger:
    type: http
    httpRequest:
      url: http://host.docker.internal:1111/get/info
      method: POST
      headers:
      - key: Content-Type
        value: application/json
  specs:

    assertions:
    - attr:http.status_code  =  200

[danielbdias]

hi @francisan , how are you?

Are you planning to use OTel Collector to test the routing? As far as I see in the configurations, you are testing only with logs.

You can do with Tracetest, in a mixed approach, by running a test that will trigger an HTTP endpoint and validating its response, like this example:

type: Test
spec:
  id: PrLoRxqIg
  name: Http Bin test
  trigger:
    type: http
    httpRequest:
      method: GET
      url: https://httpbin.org/get?hello=world
      headers:
      - key: Content-Type
        value: application/json
  skipTraceCollection: true

Then, manually inspect the OTel Collector logs to see if they are working.

Another option is to make your webhook emit traces and then validate it against a Tracing Backend.

[francisan]

@danielbdias I am good. Thanks for your reply.

Suppose if I am exporting logs to Kafka or Splunk, how can I test whether corresponding logs are successfully ingested to corresponding Kafka Topic or Splunk HEC. Need to do the assertion/verification in an automated way, without any manual intervention.

[francisan]

@danielbdias Any thoughts regarding above question?

[mathnogueira]

Hello @francisan

If I understood it correctly, you want to test your collector configuration regarding the routing of the logs exporters. Unfortunately, it's not possible to use Tracetest for that at the current state of the project because we only handle traces. We plan to support other signals in the future, but for now, only traces are being supported.

Unfortunately, I'm not aware of any framework to test this scenario specifically. I suppose you could do that automatically by using file exporters for abc and xyz and check the content of the files after running an operation to generate those logs and send them to the collector.

Let me know if I misunderstood your problem.

[francisan]

Hi @mathnogueira

Thanks for your reply. Yes, you are right.

What I am thinking is, I will create some Junit Testcases which will invoke the web hook API, and after routing the logs are written to corresponding files (abc.txt or xyz.txt). Through Junit itself I will verify the files contents and if I found name=xyz/abc then I will mark the test case as success.

Please share your thoughts.

Also, I am facing one issue in webhookevent receiver. If I put any new line in between Json, then rather than considering it as a single log statement, it will consider it as two different log statements.

For example,

If we use below Json, then it will consider it as a single log. We are good here. {"name" : "francis" ,"city" : "newyork"}

But if we use below Json, then it will consider it as 4 different logs. I think, ideally it should consider it as a single log statement.

{ "name": "francis", "city": "newyork" }

that is, { --->log1 "name": "francis", --->log2 "city": "newyork" --->log3 } --->log4

Why is it behaving in this way?

[mathnogueira]

I think I would use the same approach for testing your case and I think it works.

About the log issue, I don't know webhookevent deeply enough. I recommend you to post your issue at the opentelemetry collector chat in Slack (get invite here)

As this is specific problem is not related to tracetest, I'll close this issue. But feel free to open another issue if you want to discuss how Tracetest can help you test your observability pipeline or if you face any issues while creating tracetests. We will be more than happy to assist you.