search

jertel / elastalert2

ElastAlert 2 is a continuation of the original yelp/elastalert project. Pull requests are appreciated!
https://elastalert2.readthedocs.org
Apache License 2.0
895 stars 282 forks source link

Update telegram.py to be able to send alerts to Telegram in Markdown format #1214

Closed villeparamio closed 1 year ago

villeparamio commented 1 year ago

Hi

Description

I've been trying to send elastalert alerts in markdown format, although I specify in the alert the parameter telegram_parse_mode: "markdown", this does not work, because in the code of the telegram.py file, it is established that if the text is not html, three quotation marks must be added to the body of the alert. When receiving it in telegram it interprets it as monospaced code, ignoring the markdown format.

Removing the 3 quotes from the function allows elastalert to be able to send the alerts to Telegram in Markdown format.

Thank you for your attention. Regards, David Paramio.

imagen

Checklist

Questions or Comments

jertel commented 1 year ago

Hello! I think the idea is a good addition but I'm concerned that you removed the template checklist from your PR description. That checklist outlines the requirements for all incoming PRs. One of the requirements is ensuring there is unit test coverage for all code changes. Also, I think it would be better to have the code support a specific markdown type.

villeparamio commented 1 year ago

Hello! I think the idea is a good addition but I'm concerned that you removed the template checklist from your PR description. That checklist outlines the requirements for all incoming PRs. One of the requirements is ensuring there is unit test coverage for all code changes. Also, I think it would be better to have the code support a specific markdown type.

Hello I have edited the PR with the checklist

jertel commented 1 year ago

Hello, do you still intended to finish this PR?

nsano-rururu commented 1 year ago

@villeparamio the testcode fails. please fix.

================================================================================================================== FAILURES ==================================================================================================================
___________________________________________________________________________________________________________ test_telegram_markdown ___________________________________________________________________________________________________________
[gw3] linux -- Python 3.11.1 /home/elastalert/tests/.tox/py311/bin/python

caplog = <_pytest.logging.LogCaptureFixture object at 0x7f2aa7fa3350>

    def test_telegram_markdown(caplog):
        caplog.set_level(logging.INFO)
        rule = {
            'name': 'Test Telegram Rule',
            'type': 'any',
            'telegram_bot_token': 'xxxxx1',
            'telegram_room_id': 'xxxxx2',
            'alert': []
        }
        rules_loader = FileRulesLoader({})
        rules_loader.load_modules(rule)
        alert = TelegramAlerter(rule)
        match = {
            '@timestamp': '2021-01-01T00:00:00',
            'somefield': 'foobarbaz'
        }
        with mock.patch('requests.post') as mock_post_request:
            alert.alert([match])
        expected_data = {
            'chat_id': rule['telegram_room_id'],
            'text': '⚠ *Test Telegram Rule* ⚠ ```\nTest Telegram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n ```',
            'parse_mode': 'markdown',
            'disable_web_page_preview': True
        }

        mock_post_request.assert_called_once_with(
            'https://api.telegram.org/botxxxxx1/sendMessage',
            data=mock.ANY,
            headers={'content-type': 'application/json'},
            proxies=None,
            auth=None
        )

        actual_data = json.loads(mock_post_request.call_args_list[0][1]['data'])
>       assert expected_data == actual_data
E       AssertionError: assert {'chat_id': '...barbaz\n ```'} == {'chat_id': '...foobarbaz\n '}
E         Omitting 3 identical items, use -vv to show
E         Differing items:
E         {'text': '⚠ *Test Telegram Rule* ⚠ ```\nTest Telegram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n ```'} != {'text': '⚠ *Test Telegram Rule* ⚠ \nTest Telegram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n '}
E         Use -v to get more diff

alerters/telegram_test.py:49: AssertionError
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------
INFO     elastalert:telegram.py:62 Alert sent to Telegram room xxxxx2
____________________________________________________________________________________________________________ test_telegram_proxy _____________________________________________________________________________________________________________
[gw3] linux -- Python 3.11.1 /home/elastalert/tests/.tox/py311/bin/python

    def test_telegram_proxy():
        rule = {
            'name': 'Test Telegram Rule',
            'type': 'any',
            'telegram_bot_token': 'xxxxx1',
            'telegram_room_id': 'xxxxx2',
            'telegram_proxy': 'http://proxy.url',
            'telegram_proxy_login': 'admin',
            'telegram_proxy_pass': 'password',
            'alert': []
        }
        rules_loader = FileRulesLoader({})
        rules_loader.load_modules(rule)
        alert = TelegramAlerter(rule)
        match = {
            '@timestamp': '2021-01-01T00:00:00',
            'somefield': 'foobarbaz'
        }
        with mock.patch('requests.post') as mock_post_request:
            alert.alert([match])
        expected_data = {
            'chat_id': rule['telegram_room_id'],
            'text': '⚠ *Test Telegram Rule* ⚠ ```\nTest Telegram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n ```',
            'parse_mode': 'markdown',
            'disable_web_page_preview': True
        }

        mock_post_request.assert_called_once_with(
            'https://api.telegram.org/botxxxxx1/sendMessage',
            data=mock.ANY,
            headers={'content-type': 'application/json'},
            proxies={'https': 'http://proxy.url'},
            auth=HTTPProxyAuth('admin', 'password')
        )

        actual_data = json.loads(mock_post_request.call_args_list[0][1]['data'])
>       assert expected_data == actual_data
E       AssertionError: assert {'chat_id': '...barbaz\n ```'} == {'chat_id': '...foobarbaz\n '}
E         Omitting 3 identical items, use -vv to show
E         Differing items:
E         {'text': '⚠ *Test Telegram Rule* ⚠ ```\nTest Telegram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n ```'} != {'text': '⚠ *Test Telegram Rule* ⚠ \nTest Telegram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n '}
E         Use -v to get more diff

alerters/telegram_test.py:128: AssertionError
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------
INFO     elastalert:telegram.py:62 Alert sent to Telegram room xxxxx2
________________________________________________________________________________________________________ test_telegram_text_maxlength ________________________________________________________________________________________________________
[gw3] linux -- Python 3.11.1 /home/elastalert/tests/.tox/py311/bin/python

    def test_telegram_text_maxlength():
        rule = {
            'name': 'Test Telegram Rule' + ('a' * 3985),
            'type': 'any',
            'telegram_bot_token': 'xxxxx1',
            'telegram_room_id': 'xxxxx2',
            'alert': []
        }
        rules_loader = FileRulesLoader({})
        rules_loader.load_modules(rule)
        alert = TelegramAlerter(rule)
        match = {
            '@timestamp': '2021-01-01T00:00:00',
            'somefield': 'foobarbaz'
        }
        with mock.patch('requests.post') as mock_post_request:
            alert.alert([match])
        expected_data = {
            'chat_id': rule['telegram_room_id'],
            'text': '⚠ *Test Telegram Rule' + ('a' * 3979) +
                    '\n⚠ *message was cropped according to telegram limits!* ⚠ ```',
            'parse_mode': 'markdown',
            'disable_web_page_preview': True
        }

        mock_post_request.assert_called_once_with(
            'https://api.telegram.org/botxxxxx1/sendMessage',
            data=mock.ANY,
            headers={'content-type': 'application/json'},
            proxies=None,
            auth=None
        )

        actual_data = json.loads(mock_post_request.call_args_list[0][1]['data'])
>       assert expected_data == actual_data
E       AssertionError: assert {'chat_id': '...mits!* ⚠ ```'} == {'chat_id': '... limits!* ⚠ '}
E         Omitting 3 identical items, use -vv to show
E         Differing items:
E         {'text': '⚠ *Test Telegram Ruleaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n⚠ *message was cropped according to telegram limits!* ⚠ ```'} != {'text': '⚠ *Test Telegram Ruleaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n⚠ *message was cropped according to telegram limits!* ⚠ '}
E         Use -v to get more diff

alerters/telegram_test.py:165: AssertionError
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------
INFO     elastalert:telegram.py:62 Alert sent to Telegram room xxxxx2
____________________________________________________________________________________________________________ test_telegram_matchs ____________________________________________________________________________________________________________
[gw2] linux -- Python 3.11.1 /home/elastalert/tests/.tox/py311/bin/python

    def test_telegram_matchs():
        rule = {
            'name': 'Test Telegram Rule',
            'type': 'any',
            'telegram_bot_token': 'xxxxx1',
            'telegram_room_id': 'xxxxx2',
            'alert': []
        }
        rules_loader = FileRulesLoader({})
        rules_loader.load_modules(rule)
        alert = TelegramAlerter(rule)
        match = {
            '@timestamp': '2021-01-01T00:00:00',
            'somefield': 'foobarbaz'
        }
        with mock.patch('requests.post') as mock_post_request:
            alert.alert([match, match])
        expected_data = {
            'chat_id': rule['telegram_room_id'],
            'text': '⚠ *Test Telegram Rule* ⚠ ```\n' +
                    'Test Telegram Rule\n' +
                    '\n' +
WARNING:py.warnings:/home/elastalert/tests/.tox/py311/lib/python3.11/site-packages/jira/client.py:14: DeprecationWarning: 'imghdr' is deprecated and slated for removal in Python 3.13
  import imghdr

                    '@timestamp: 2021-01-01T00:00:00\n' +
                    'somefield: foobarbaz\n' +
                    '\n' +
                    '----------------------------------------\n' +
                    'Test Telegram Rule\n' +
                    '\n' +
                    '@timestamp: 2021-01-01T00:00:00\n' +
                    'somefield: foobarbaz\n' +
                    '\n' +
                    '----------------------------------------\n' +
                    ' ```',
            'parse_mode': 'markdown',
            'disable_web_page_preview': True
        }

        mock_post_request.assert_called_once_with(
            'https://api.telegram.org/botxxxxx1/sendMessage',
            data=mock.ANY,
            headers={'content-type': 'application/json'},
            proxies=None,
            auth=None
        )

        actual_data = json.loads(mock_post_request.call_args_list[0][1]['data'])
>       assert expected_data == actual_data
E       AssertionError: assert {'chat_id': '...------\n ```'} == {'chat_id': '...---------\n '}
E         Omitting 3 identical items, use -vv to show
E         Differing items:
E         {'text': '⚠ *Test Telegram Rule* ⚠ ```\nTest Telegram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n\...egram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n\n----------------------------------------\n ```'} != {'text': '⚠ *Test Telegram Rule* ⚠ \nTest Telegram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n\n--...Telegram Rule\n\n@timestamp: 2021-01-01T00:00:00\nsomefield: foobarbaz\n\n----------------------------------------\n '}
E         Use -v to get more diff

alerters/telegram_test.py:290: AssertionError
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------
INFO     elastalert:telegram.py:62 Alert sent to Telegram room xxxxx2
============================================================================================================== warnings summary ==============================================================================================================
.tox/py311/lib/python3.11/site-packages/pkg_resources/__init__.py:121
.tox/py311/lib/python3.11/site-packages/pkg_resources/__init__.py:121
.tox/py311/lib/python3.11/site-packages/pkg_resources/__init__.py:121
.tox/py311/lib/python3.11/site-packages/pkg_resources/__init__.py:121
.tox/py311/lib/python3.11/site-packages/pkg_resources/__init__.py:121
  /home/elastalert/tests/.tox/py311/lib/python3.11/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
    warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)

.tox/py311/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: 20 warnings
  /home/elastalert/tests/.tox/py311/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform linux, python 3.11.1-final-0 -----------
Name                                                           Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------------------------------------------
/home/elastalert/elastalert/__init__.py                           52     12     26      5    68%   42, 49->52, 58, 64, 68-76, 132->135, 140->142, 142->144, 144->147
/home/elastalert/elastalert/alerters/__init__.py                   0      0      0      0   100%
/home/elastalert/elastalert/alerters/alerta.py                    69      0     14      0   100%
/home/elastalert/elastalert/alerters/alertmanager.py              54      1     12      1    97%   40
/home/elastalert/elastalert/alerters/chatwork.py                  36      0      6      0   100%
/home/elastalert/elastalert/alerters/command.py                   36      0     12      0   100%
/home/elastalert/elastalert/alerters/datadog.py                   23      0      0      0   100%
/home/elastalert/elastalert/alerters/debug.py                     12      0      4      0   100%
/home/elastalert/elastalert/alerters/dingtalk.py                  48      0     12      0   100%
/home/elastalert/elastalert/alerters/discord.py                   52      0      8      0   100%
/home/elastalert/elastalert/alerters/email.py                    104      9     50      6    90%   53-54, 62->68, 66->68, 75-79, 107->109, 113-114
/home/elastalert/elastalert/alerters/exotel.py                    26      0      2      0   100%
/home/elastalert/elastalert/alerters/gelf.py                      67      6     24      4    89%   23, 24->27, 26, 55-56, 73-74, 102->exit
/home/elastalert/elastalert/alerters/gitter.py                    25      0      0      0   100%
/home/elastalert/elastalert/alerters/googlechat.py                55      0     14      0   100%
/home/elastalert/elastalert/alerters/httppost2.py                 79      1     34      2    96%   21, 22->24
/home/elastalert/elastalert/alerters/httppost.py                  44      0     12      0   100%
/home/elastalert/elastalert/alerters/jira.py                     254     44    142     26    78%   96-103, 117, 137, 167, 182, 189, 202, 209, 237, 252, 255, 273-276, 295-296, 302-303, 304->299, 308-309, 311-315, 318-319, 322, 335->350, 349, 353-354, 359->364, 363, 369, 376, 381, 384->388, 390, 404, 409
/home/elastalert/elastalert/alerters/line.py                      27      0      6      0   100%
/home/elastalert/elastalert/alerters/mattermost.py               109      2     50      2    96%   59, 72
/home/elastalert/elastalert/alerters/opsgenie.py                 143      0     80      0   100%
/home/elastalert/elastalert/alerters/pagerduty.py                 82      6     42      2    90%   118-120, 135-137
/home/elastalert/elastalert/alerters/pagertree.py                 24      0      0      0   100%
/home/elastalert/elastalert/alerters/rocketchat.py                77      1     24      1    98%   50
/home/elastalert/elastalert/alerters/servicenow.py                31      0      6      0   100%
/home/elastalert/elastalert/alerters/ses.py                       59     16     30      6    62%   29, 34, 39, 43, 50-59, 63, 96
/home/elastalert/elastalert/alerters/slack.py                    116      1     46      1    99%   63
/home/elastalert/elastalert/alerters/sns.py                       28      4      2      1    83%   20-21, 34, 44
/home/elastalert/elastalert/alerters/stomp.py                     47      8      6      2    81%   24, 29-35, 70-72, 75
/home/elastalert/elastalert/alerters/teams.py                     65      0     22      0   100%
/home/elastalert/elastalert/alerters/telegram.py                  44      0     10      0   100%
/home/elastalert/elastalert/alerters/tencentsms.py                64      0      8      0   100%
/home/elastalert/elastalert/alerters/thehive.py                  108      3     50      4    94%   35->34, 36->51, 91-93, 125->145
/home/elastalert/elastalert/alerters/twilio.py                    29      0      6      0   100%
/home/elastalert/elastalert/alerters/victorops.py                 33      0      2      0   100%
/home/elastalert/elastalert/alerters/zabbix.py                    64     23     14      2    63%   13-20, 26-41, 72, 96
/home/elastalert/elastalert/alerts.py                            207      9    126      8    94%   16-19, 84, 113, 155->exit, 167, 226->232, 252, 287->302, 320, 325
/home/elastalert/elastalert/auth.py                               26      0     10      0   100%
/home/elastalert/elastalert/config.py                             73      5     34      3    93%   105, 135, 138-140
/home/elastalert/elastalert/create_index.py                      154    122     42      1    18%   24-95, 124-242, 246
/home/elastalert/elastalert/elastalert.py                       1156    349    551     92    67%   104, 109, 112, 117->120, 121-123, 164, 177-178, 180, 184, 224-227, 230->233, 242, 251-279, 287-297, 316, 321, 330->329, 338-339, 371, 382, 387-393, 400, 414, 452, 456-462, 467-482, 498-504, 519-544, 582, 584, 587-589, 600, 615, 617, 623-627, 630-634, 653->exit, 661-662, 709-712, 718-728, 747, 763-766, 781, 796, 833, 844-845, 864, 867-868, 873, 906, 935-952, 965-972, 1022-1030, 1036-1037, 1039-1063, 1068->1067, 1075->1019, 1079->1100, 1084-1085, 1087, 1094->1080, 1097, 1105-1112, 1128->exit, 1133-1138, 1141, 1144->1148, 1187-1192, 1195-1197, 1201-1203, 1212, 1219, 1223-1236, 1242, 1246-1263, 1280-1285, 1298-1299, 1305-1306, 1315, 1322-1340, 1343-1346, 1358-1361, 1364, 1368-1370, 1382-1384, 1410, 1413-1416, 1443, 1453-1454, 1484-1486, 1497-1499, 1503->1502, 1507, 1513->1490, 1534-1535, 1540-1549, 1568-1569, 1585-1589, 1604-1608, 1624-1625, 1674-1675, 1678-1679, 1684, 1690-1692, 1695-1696, 1717, 1728-1731, 1733-1740, 1770-1771, 1773-1781, 1784, 1800-1801, 1807->1809, 1854, 1859-1861, 1865-1875, 1879
/home/elastalert/elastalert/enhancements.py                       11      0      0      0   100%
/home/elastalert/elastalert/eql.py                                37      0     20      0   100%
/home/elastalert/elastalert/kibana_discover.py                    61      0     20      0   100%
/home/elastalert/elastalert/kibana_external_url_formatter.py      90      0     22      0   100%
/home/elastalert/elastalert/loaders.py                           371     63    174     40    78%   171-172, 174, 270, 305->307, 308, 313, 314->316, 317, 319, 321, 323, 325, 330-331, 358-359, 361-362, 381, 385-388, 392, 395, 398, 411-412, 415-416, 417->420, 426, 427->429, 429->431, 432, 438-439, 444-446, 452, 460, 469-475, 484, 490, 494-495, 498->exit, 514, 519, 549, 564->561, 569, 577-581, 586-587, 609-612
/home/elastalert/elastalert/prometheus_wrapper.py                 38     32     10      0    12%   9-24, 27, 31-34, 39-55
/home/elastalert/elastalert/ruletypes.py                         768    181    410     36    73%   99-102, 196->200, 251->exit, 261, 269->268, 274-282, 325, 334-345, 349-352, 356-359, 362, 367-387, 413, 432, 434-442, 454-459, 490-496, 513-514, 529-530, 544-560, 565-575, 624-633, 658-660, 664, 666, 669, 671-675, 679-681, 690, 692, 707-708, 719, 730, 735, 751-754, 756, 764-774, 912->911, 928, 977, 981-993, 1012, 1022, 1046, 1064, 1066, 1086, 1095, 1114, 1157-1171, 1175-1180, 1188-1199, 1206-1228, 1234-1241
/home/elastalert/elastalert/test_rule.py                         295    295    159      0     0%   3-515
/home/elastalert/elastalert/util.py                              322      6    128      3    98%   297->300, 474, 565->584, 578-582
/home/elastalert/elastalert/yaml.py                                6      0      2      0   100%
----------------------------------------------------------------------------------------------------------
TOTAL                                                           5801   1199   2484    248    77%

========================================================================================================== short test summary info ===========================================================================================================
FAILED alerters/telegram_test.py::test_telegram_markdown - AssertionError: assert {'chat_id': '...barbaz\n ```'} == {'chat_id': '...foobarbaz\n '}
FAILED alerters/telegram_test.py::test_telegram_proxy - AssertionError: assert {'chat_id': '...barbaz\n ```'} == {'chat_id': '...foobarbaz\n '}
FAILED alerters/telegram_test.py::test_telegram_text_maxlength - AssertionError: assert {'chat_id': '...mits!* ⚠ ```'} == {'chat_id': '... limits!* ⚠ '}
FAILED alerters/telegram_test.py::test_telegram_matchs - AssertionError: assert {'chat_id': '...------\n ```'} == {'chat_id': '...---------\n '}
===================================================================================== 4 failed, 911 passed, 4 skipped, 25 warnings in 155.17s (0:02:35) ======================================================================================
py311: exit 1 (157.40 seconds) /home/elastalert/tests> pytest --cov=../elastalert --cov-report=term-missing --cov-branch --strict-markers . -n 4 pid=51
nsano-rururu commented 1 year ago

@villeparamio

Please add that it corresponded to the changelog.

jertel commented 1 year ago

Closing PR due to inactivity. This can be re-opened if work resumes.