Telegram polling discussion group fails

Cenness commented 5 days ago

What went wrong?

What happened:

I've installed OnCall via helm with telegram integration and polling enabled. Target channel has discussion enabled and bot is added to both.

In polling pod I see logs

/usr/local/lib/python3.12/site-packages/telegram/utils/request.py:49: UserWarning: python-telegram-bot is using upstream urllib3. This is allowed but not supported by python-telegram-bot maintainers.
  warnings.warn(
source=engine:app google_trace_id=none logger=engine.management.commands.start_telegram_polling Starting telegram polling...
source=engine:app google_trace_id=none logger=apscheduler.scheduler Scheduler started
source=engine:app google_trace_id=none logger=engine.management.commands.start_telegram_polling Update from Telegram: {'message': {'message_id': 28, 'forward_from_message_id': 2607, 'is_automatic_forward': True, 'supergroup_chat_created': False, 'delete_chat_photo': False, 'caption_entities': [], 'group_chat_created': False, 'forward_date': 1729153738, 'new_chat_members': [], 'forward_from_chat': {'id': -1009999999999, 'type': 'channel', 'title': 'Redacted alerts'}, 'text': 'alertname:  KubePersistentVolumeUsageCritical\n\nStatus: FIRING 🔥\n\nActive Alert List:\n\n================================\nLabel List:\n•     alertgroup = kubernetes-storage\n•     dev_team = AppDevOps\n•     env = dev\n•     instance = 10.10.1.10:10250\n•     metrics_path = /metrics\n•     namespace = monitoring\n•     node = worker-az0-8\n•     persistentvolumeclaim = oncall-test\n•     service = system\n•     severity = critical 🚨\n\nDescription:\n•• message = The persistent volume claimed by oncall-test in namespace monitoring has 4% free.\n•• timestamp = 2024-10-17 08:27:43.659 +0000 UTC\n\n\n Time Start Alert: 2024-10-17T08:28:13.561036624Z \n================================', 'forward_signature': 'Redacted Alert', 'sender_chat': {'id': -1009999999999, 'type': 'channel', 'title': 'Redacted alerts'}, 'new_chat_photo': [], 'channel_chat_created': False, 'photo': [], 'entities': [{'length': 10, 'type': 'bold', 'offset': 0}, {'length': 33, 'type': 'code', 'offset': 12}, {'length': 9, 'type': 'bold', 'offset': 55}, {'length': 18, 'type': 'bold', 'offset': 66}, {'length': 16, 'type': 'url', 'offset': 234}, {'length': 8, 'type': 'bot_command', 'offset': 272}, {'length': 50, 'type': 'italic', 'offset': 596}], 'chat': {'id': -1008888888888, 'type': 'supergroup', 'title': 'Redacted alerts Chat'}, 'date': 1729153741, 'from': {'first_name': 'Telegram', 'id': 777000, 'is_bot': False}}, 'update_id': 629379447}
source=engine:app google_trace_id=none logger=apps.telegram.updates.update_manager Processing update with handler: ChannelToGroupForwardHandler
source=engine:app google_trace_id=none logger=apps.telegram.updates.update_handlers.channel_to_group_forward Tried to send log and action message to comments, but organization deleted the channel connector. Channel chat id: -1009999999999. Channel message id: 2607. Group message id: 28.

ChatOps setting UI shows like there are no TELEGRAM.* env vars set:

but they are:

oncall-engine-7dd5dc9684-htdnc:/etc/app$ env | sort | grep -i teleg | grep -iv sd
FEATURE_TELEGRAM_INTEGRATION_ENABLED=True
FEATURE_TELEGRAM_LONG_POLLING_ENABLED=True
TELEGRAM_TOKEN=REDACTED:REDACTED
TELEGRAM_WEBHOOK_HOST=https://REDACTED

oncall-telegram-polling-7769f6d44d-rkfr5:/etc/app$ env | sort | grep -i teleg | grep -iv sd
FEATURE_TELEGRAM_INTEGRATION_ENABLED=True
FEATURE_TELEGRAM_LONG_POLLING_ENABLED=True
HOSTNAME=oncall-telegram-polling-7769f6d44d-rkfr5
TELEGRAM_TOKEN=REDACTED:REDACTED
TELEGRAM_WEBHOOK_HOST=https://REDACTED

oncall-celery-85695844b7-84p44:/etc/app$ env | sort | grep -i teleg | grep -iv sd
CELERY_WORKER_QUEUE=default,critical,long,slack,telegram,webhook,celery,grafana,retry
FEATURE_TELEGRAM_INTEGRATION_ENABLED=True
FEATURE_TELEGRAM_LONG_POLLING_ENABLED=True
TELEGRAM_TOKEN=REDACTED:REDACTED
TELEGRAM_WEBHOOK_HOST=https://REDACTED

What did you expect to happen:

chatops allowing to add my channels buttons appearing on alert messages

How do we reproduce it?

setup grafana oncall with telegram polling enabled
go to chatops tab - no options to add channel
go to polling logs - error "Tried to send log and action message to comments, but organization deleted the channel connector"

Grafana OnCall Version

v1.11.3

Product Area

Chatops

Grafana OnCall Platform?

Kubernetes

User's Browser?

Firefox 131

Anything else to add?

No response

Cenness commented 4 days ago

After setting FEATURE_LIVE_SETTINGS_ENABLED to False option did appear, but adding new channel can't complete - bot sees the message with uuid and does nothing:

source=engine:app google_trace_id=none logger=engine.management.commands.start_telegram_polling 
  Update from Telegram: {'message': {'text': '412208db-98b0-4c51-8410-ab5ad56ccdfa_5b5bdc56-e629-4ebe-ab56-53ae7a72095a',
     'date': 1729240907, 'photo': [], 'sender_chat': {'id': -1008888888888, 'title': Redacted alerts Chat', 'type': 'supergroup'},
     'message_id': 40, 'group_chat_created': False, 'entities': [], 'delete_chat_photo': False, 'caption_entities': [], 
     'supergroup_chat_created': False, 'chat': {'id': -1008888888888, 'title': 'Redacted alerts Chat', 'type': 'supergroup'}, 
     'new_chat_members': [], 'new_chat_photo': [], 'channel_chat_created': False,
     'from': {'id': 1087968824, 'username': 'GroupAnonymousBot', 'is_bot': True, 'first_name': 'Group'}}, 'update_id': 629379463}
source=engine:app google_trace_id=none logger=apps.telegram.updates.update_manager No update handlers applied for update

Connecting personal account does work:

source=engine:app google_trace_id=none logger=engine.management.commands.start_telegram_polling
  Update from Telegram: {'update_id': 629379487, 'message': {'new_chat_photo': [], 'message_id': 61, 'date': 1729244057,
     'delete_chat_photo': False, 'chat': {Redacted, 'username': 'Cenness'},
     'channel_chat_created': False, 'photo': [], 'group_chat_created': False, 'entities': [], 'supergroup_chat_created': False,
     'new_chat_members': [], 'text': '412208db-98b0-4c51-8410-ab5ad56ccdfa_b9e12fd3-b416-4987-9083-90b71c32c699',
     'caption_entities': [], 'from': {Redacted, 'username': 'Cenness'}}}
source=engine:app google_trace_id=none logger=apps.telegram.updates.update_manager
  Processing update with handler: PersonalVerificationCodeHandler

Cenness commented 3 days ago

Oh. my mistake, instructions mentioning both channel and discussion group made me confused where bot was expecting confirmation code - "Send this verification code to the channel and wait for the confirmation message" Poller clearly seeing that confirmation code in discussion group made my confusion deeper.

Its already parsing both, why wouldn't OnCall accept code from group?

grafana / oncall