search

elastic / logstash-filter-elastic_integration

The Elastic Integrations filter for Logstash, which enables running Elastic Integrations inside of Logstash pipelines
Other
5 stars 8 forks source link

E2E tests for the plugin. #122

Closed mashhurs closed 7 months ago

mashhurs commented 7 months ago

Description

Adds integration tests for ["apache", "m365_defender", "nginx", "tomcat"] integrations, can be easily changeable in main.py

High level description: E2E framework

How to test

Integrate with BK CI

Author's checklist

Logs

mashhurs commented 7 months ago

Functionally this looks good. I left some optional comments.

When running the tests locally, using the instructions in the README file, I noticed there were some errors in the output:

Error: one or more test cases failed
Internal failure happened with nginx, process return code: 1.
--- Test results for package: nginx - START ---
FAILURE DETAILS:
nginx/access default (variant: v1):
[0] field "message" is undefined

╭─────────┬─────────────┬───────────┬───────────────────────┬────────────────────────────────────────────────────────────────────────────────────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME             │ RESULT                                                                                 │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ nginx   │ access      │ system    │ default (variant: v1) │ FAIL: one or more errors found in documents stored in logs-nginx.access-ep data stream │ 48.090519211s │
│ nginx   │ error       │ system    │ default (variant: v1) │ PASS                                                                                   │ 36.341966511s │
│ nginx   │ stubstatus  │ system    │ default (variant: v1) │ PASS                                                                                   │ 31.301999294s │
╰─────────┴─────────────┴───────────┴───────────────────────┴────────────────────────────────────────────────────────────────────────────────────────┴───────────────╯
--- Test results for package: nginx - END   ---
Done

however, the script finished successfully:

...
Stopping elastic-package stack...
Take down the Elastic stack
Done
$ echo $?
0

Shouldn't the above test failure have caused the script to fail too?

This result is coming from internal elastic-package, not from elastic_integration or es-output plugins. If you take a look more logs, you will be able to see how many events plugins processed. Some of events may get rejected by ES and that is caught by elastic-package. Every integration behavior is different and may require different setting especially in es-output side (ECS compatibility, ILM, datastream, etc...). So, with current E2E tests, we are more focusing on running integrations with elastic_integration plugin, may extend to output plugins. Current failure looks because ECS is enabled by default and the data is in event.original. If any failure happens with this, tests fail, otherwise prints out every possible failures (elastic-package, es-output, etc...) for better visibility.

dliappis commented 7 months ago

This result is coming from internal elastic-package, not from elastic_integration or es-output plugins. If you take a look more logs, you will be able to see how many events plugins processed. Some of events may get rejected by ES and that is caught by elastic-package. Every integration behavior is different and may require different setting especially in es-output side (ECS compatibility, ILM, datastream, etc...). So, with current E2E tests, we are more focusing on running integrations with elastic_integration plugin, may extend to output plugins. Current failure looks because ECS is enabled by default and the data is in event.original. If any failure happens with this, tests fail, otherwise prints out every possible failures (elastic-package, es-output, etc...) for better visibility.

Thanks for the clarification. This is alright for now, but I fear it could be trappy behaviour for someone not familiar with the test troubleshooting a failed CI job where there is a genuine error like the one you described above, plus the above nginx error which should be ignored (but how would the person triaging know about it? should we address this in a troubleshooting guide here or at a later PR?)

elasticmachine commented 7 months ago

:green_heart: Build Succeeded

History

cc @mashhurs

mashhurs commented 7 months ago

🎉 First 🟢 actual CI