search

jertel / elastalert2

ElastAlert 2 is a continuation of the original yelp/elastalert project. Pull requests are appreciated!
https://elastalert2.readthedocs.org
Apache License 2.0
895 stars 282 forks source link

Opensearch discover link support #1310

Closed luffynextgen closed 9 months ago

luffynextgen commented 9 months ago

Description

This pull request aim to add the possibility to create opensearch discover url from the elastalert query, the same way it is done for the kibana discover url.

I tried to minimize the change, so I only added one key in the yaml schema ( generate_opensearch_discover_url ). This feature reuse all the current kibana related variable.

Checklist

Questions or Comments

I have tested this in my environment making elastalert run on an Opensearch SIEM in 2.11. The alert triggered with this feature were sent to thehive and slack. The link in thehive was added to the description of the alert and worked. In slack the "Discover in Kibana" button also worked.

This feature for now was only tested with opensearch 2.11

jertel commented 9 months ago

I like the new feature! I can definitely see the value here for OpenSearch users.

Instead of referencing kibana_* vars within the OpenSearch logic, it would be cleaner to replace the kibana_ references with something more generic, so that it makes sense to share those vars with both OpenSearch and Elasticsearch. The drawback to doing this is it would still have to support the kibana_* as well, so that existing users didn't break after upgrading. An alternative is to simply create the corresponding vars using the opensearch name. That is simpler, but means a little more work since the schema and docs need to be updated. It would be less work than the other option though. I know you are trying to minimize changes but not doing this will cause confusion to future developers, when they're scratching their heads trying to figure out why OpenSearch logic is reference Kibana URLs, etc.

The new Python code will also need unit tests coverage before this can be merged in. It's a lot of work to add the coverage but it's very valuable for future developers so they don't break your changes when they add new functionality.

Thanks for helping to keep the project growing!

nsano-rururu commented 9 months ago

If you added the settings to slack, please add the settings below as well.

nsano-rururu commented 9 months ago

Does opensearch have a Shorten URL API like kibana? . If not, I think there is no need to add any related settings.

luffynextgen commented 9 months ago

Hello @nsano-rururu ,

I'm currently writing the test code, I'll add the documentation for the variable afterward.

It does have a shorten API but for now, I let it aside. For now the code call to kibana external formater url only to finish the url, the base url add happen in this file for kibana and opensearch

luffynextgen commented 9 months ago

I have put the tests and the documentation up to date

nsano-rururu commented 9 months ago

I haven't checked the operation of the test code.

nsano-rururu commented 9 months ago

Please check the error details and correct the test code.

luffynextgen commented 9 months ago

Please check the error details and correct the test code.

I'm working on the issues printed

luffynextgen commented 9 months ago

All the test succeeded, but the final is this: 

The HTML pages are in build/html.
  py311: FAIL code 1 (100.76=setup[27.84]+cmd[71.69,1.24] seconds)
  docs: OK (28.14=setup[26.26]+cmd[1.88] seconds)
  evaluation failed :( (128.96 seconds)
make: *** [Makefile:30: test-docker] Error 255
Error: Process completed with exit code 2.
jertel commented 9 months ago

The linter is failing due to the new code not following the Python development guidelines.

 ./alerters/mattermost_test.py:1145:1: E302 expected 2 blank lines, found 1
./alerters/mattermost_test.py:1344:1: E302 expected 2 blank lines, found 1
./alerters/slack_test.py:458:1: E302 expected 2 blank lines, found 1
./alerters/slack_test.py:657:1: E302 expected 2 blank lines, found 1
./alerters/teams_test.py:392:1: E302 expected 2 blank lines, found 1
./alerters/teams_test.py:528:1: E302 expected 2 blank lines, found 0
./loaders_test.py:458:1: E302 expected 2 blank lines, found 1
./loaders_test.py:477:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:23:1: W293 blank line contains whitespace
./opensearch_discover_test.py:26:20: E261 at least two spaces before inline comment
./opensearch_discover_test.py:29:23: E261 at least two spaces before inline comment
./opensearch_discover_test.py:32:16: E261 at least two spaces before inline comment
./opensearch_discover_test.py:34:20: E261 at least two spaces before inline comment
./opensearch_discover_test.py:39:42: E261 at least two spaces before inline comment
./opensearch_discover_test.py:39:42: E262 inline comment should start with '# '
./opensearch_discover_test.py:39:65: W291 trailing whitespace
./opensearch_discover_test.py:40:64: E261 at least two spaces before inline comment
./opensearch_discover_test.py:40:65: E262 inline comment should start with '# '
./opensearch_discover_test.py:40:88: W291 trailing whitespace
./opensearch_discover_test.py:42:1: W293 blank line contains whitespace
./opensearch_discover_test.py:45:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:57:1: W293 blank line contains whitespace
./opensearch_discover_test.py:77:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:136:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:150:1: W293 blank line contains whitespace
./opensearch_discover_test.py:168:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:199:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:247:1: W293 blank line contains whitespace
./opensearch_discover_test.py:249:5: E303 too many blank lines (3)
./opensearch_discover_test.py:267:141: E501 line too long (160 > 140 characters)
./opensearch_discover_test.py:268:141: E501 line too long (158 > 140 characters)
./opensearch_discover_test.py:272:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:304:141: E501 line too long (150 > 140 characters)
./opensearch_discover_test.py:310:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:348:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:387:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:421:1: W293 blank line contains whitespace
./opensearch_discover_test.py:422:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:466:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:498:141: E501 line too long (145 > 140 characters)
./opensearch_discover_test.py:508:1: E302 expected 2 blank lines, found 1
./opensearch_discover_test.py:544:141: E501 line too long (159 > 140 characters)
luffynextgen commented 9 months ago

Hello @nsano-rururu ,

I finally got the build right :tada:

nsano-rururu commented 9 months ago

kibana_external_url_formatter.py

create_opensearch_external_url_formatter

to

opensearchexternal_url_formatter.py

create_opensearch_external_url_formatter

nsano-rururu commented 9 months ago

ruletyles.rst

Please add the following to each design of mattermost, rocket_chat, slack, and ms_teams.

``mattermost_attach_opensearch_discover_url``: Enables the attachment of the ``opensearch_discover_url`` to the mattermost notification. The config ``generate_opensearch_discover_url`` must also be ``True`` in order to generate the url. Defaults to ``False``.

``mattermost_opensearch_discover_color``: The color of the Opensearch Discover url attachment. Defaults to ``#ec4b98``.

``mattermost_opensearch_discover_title``: The title of the Opensearch Discover url attachment. Defaults to ``Discover in opensearch``.
``rocket_chat_attach_opensearch_discover_url``: Enables the attachment of the ``opensearch_discover_url`` to the Rocket.Chat notification. The config ``generate_opensearch_discover_url`` must also be ``True`` in order to generate the url. Defaults to ``False``.

``rocket_chat_opensearch_discover_color``: The color of the Opensearch Discover url attachment. Defaults to ``#ec4b98``.

``rocket_chat_opensearch_discover_title``: The title of the Opensearch Discover url attachment. Defaults to ``Discover in opensearch``.
``slack_attach_opensearch_discover_url``: Enables the attachment of the ``opensearch_discover_url`` to the slack notification. The config ``generate_opensearch_discover_url`` must also be ``True`` in order to generate the url. Defaults to ``False``.

``slack_opensearch_discover_color``: The color of the Opensearch Discover url attachment. Defaults to ``#ec4b98``.

``slack_opensearch_discover_title``: The title of the Opensearch Discover url attachment. Defaults to ``Discover in opensearch``.
``ms_teams_attach_opensearch_discover_url``: Enables the attachment of the ``opensearch_discover_url`` to the MS Teams notification. The config ``generate_opensearch_discover_url`` must also be ``True`` in order to generate the url. Defaults to ``False``.

``ms_teams_opensearch_discover_title``: The title of the Opensearch Discover url attachment. Defaults to ``Discover in opensearch``.
luffynextgen commented 9 months ago

ruletyles.rst

Please add the following to each design of mattermost, rocket_chat, slack, and ms_teams.

``mattermost_attach_opensearch_discover_url``: Enables the attachment of the ``opensearch_discover_url`` to the mattermost notification. The config ``generate_opensearch_discover_url`` must also be ``True`` in order to generate the url. Defaults to ``False``.

``mattermost_opensearch_discover_color``: The color of the Opensearch Discover url attachment. Defaults to ``#ec4b98``.

``mattermost_opensearch_discover_title``: The title of the Opensearch Discover url attachment. Defaults to ``Discover in opensearch``.
``rocket_chat_attach_opensearch_discover_url``: Enables the attachment of the ``opensearch_discover_url`` to the Rocket.Chat notification. The config ``generate_opensearch_discover_url`` must also be ``True`` in order to generate the url. Defaults to ``False``.

``rocket_chat_opensearch_discover_color``: The color of the Opensearch Discover url attachment. Defaults to ``#ec4b98``.

``rocket_chat_opensearch_discover_title``: The title of the Opensearch Discover url attachment. Defaults to ``Discover in opensearch``.
``slack_attach_opensearch_discover_url``: Enables the attachment of the ``opensearch_discover_url`` to the slack notification. The config ``generate_opensearch_discover_url`` must also be ``True`` in order to generate the url. Defaults to ``False``.

``slack_opensearch_discover_color``: The color of the Opensearch Discover url attachment. Defaults to ``#ec4b98``.

``slack_opensearch_discover_title``: The title of the Opensearch Discover url attachment. Defaults to ``Discover in opensearch``.
``ms_teams_attach_opensearch_discover_url``: Enables the attachment of the ``opensearch_discover_url`` to the MS Teams notification. The config ``generate_opensearch_discover_url`` must also be ``True`` in order to generate the url. Defaults to ``False``.

``ms_teams_opensearch_discover_title``: The title of the Opensearch Discover url attachment. Defaults to ``Discover in opensearch``.

Done

nsano-rururu commented 9 months ago

@jertel I can't find anything else to point out. is there anything else?