feat(quickwit): add quickwit integration

idrissneumann commented 6 months ago

Description

Hi. For now it's still draft but I'm exploring how elastalert2 can comply with Quickwit.

To provide a bit of context:

I use to enjoy working with elastalert several years ago and even rebuild version for ARM32, I'm pretty convinced that it's still a relevant tool for alerting used as an external microservice
Quickwit is one of the best log and traces search engine on the market and I'm use to contribute in order to provide integration to other opensource project (around the CNCF ecosystem) such as Falcosidekick, Odigos, and Grafana datasource for quickwit
Quickwit has a level of interoperability with the Elastic/Opensearch API for the read query, that been said, we have to handle the mapping creation of the indexes, add a /api/v1/_elastic suffix to the URL, etc

Checklist

[x] I have reviewed the contributing guidelines.
[x] I have included unit tests for my changes or additions.
[x] I have successfully run make test-docker with my changes.
[ ] I have manually tested all relevant modes of the change in this PR.
[x] I have updated the documentation.
[x] I have updated the changelog.

Questions or Comments

jertel commented 6 months ago

This sounds great! I'm looking forward to trying it out.

idrissneumann commented 6 months ago

I know this is a work in progress, but I figured I'd review it as you go to help me follow along.

No problem at all. I'm still figuring out if it's really compliant or not. For example with real test, I have this:

qw_tests-elastalert-1  | ERROR:elastalert:Error finding recent pending alerts: RequestError(400, '{\n  "message": "OneOrMany could not deserialize any variant:\\n  One: unknown field `from`, expected one of `gt`, `gte`, `lt`, `lte`, `boost`\\n  Many: invalid type: map, expected a sequence at line 1 column 206"\n}') {'query': {'bool': {'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2024-03-26T17:09:24.709242Z', 'to': '2024-03-28T17:09:24.709947Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}

Maybe @fmassot can help to see if it's something we can fix on quickwit side or not.

For the rest, I'll take care of all your feedbacks, thanks for thoses.

github-actions[bot] commented 5 months ago

This PR is stale because it has been open for 30 days with no activity. The longer a PR remains stale the more out of date with the main branch it becomes.

github-actions[bot] commented 4 months ago

This PR was closed because it has been inactive for 30 days since being marked as stale. It will be automatically locked after an additional 30 days. If there is still a commitment to finishing this PR please re-open it, or request that a project maintainer re-open it before it becomes locked.

rahmat-multiplier commented 2 days ago

@idrissneumann may I know if this is already completed or anything left? I really want to use quickwit but currently we are actively using elastalert to manage log based alert. anyway, love your work!

jertel / elastalert2