gemini-oss / starfleet

Whole-infrastructure automation tooling for AWS
https://gemini-oss.github.io/starfleet/
Apache License 2.0
41 stars 7 forks source link

Slack needs retry #57

Open mikegrima opened 1 year ago

mikegrima commented 1 year ago

Slack needs to have the retry decorator added to it to prevent workers from failing even though they are successful.

mikegrima commented 1 year ago

Sample stacktrace:

Traceback (most recent call last):
  File "/var/task/starfleet/worker_ships/plugins/aws_config/ship.py", line 94, in lambda_handler
    worker.execute(commit=commit)
  File "/var/task/starfleet/worker_ships/plugins/aws_config/ship.py", line 55, in execute
    self.send_alert(
  File "/var/task/starfleet/worker_ships/ship_schematics.py", line 187, in send_alert
    SLACK_CLIENT.post_success(self.alert_channel, title, body_markdown)
  File "/var/task/starfleet/utils/slack.py", line 108, in post_success
    if not self._post_message(channel_id, message):
  File "/var/task/starfleet/utils/slack.py", line 62, in _post_message
    result = self._web_client.chat_postMessage(channel=channel_id, blocks=blocks, text=blocks[0]["text"]["text"])
  File "/var/task/slack_sdk/web/client.py", line 2220, in chat_postMessage
    return self.api_call("chat.postMessage", json=kwargs)
  File "/var/task/slack_sdk/web/base_client.py", line 156, in api_call
    return self._sync_send(api_url=api_url, req_args=req_args)
  File "/var/task/slack_sdk/web/base_client.py", line 187, in _sync_send
    return self._urllib_api_call(
  File "/var/task/slack_sdk/web/base_client.py", line 317, in _urllib_api_call
    ).validate()
  File "/var/task/slack_sdk/web/slack_response.py", line 199, in validate
    raise e.SlackApiError(message=msg, response=self)
slack_sdk.errors.SlackApiError: The request to the Slack API failed. (url: https://www.slack.com/api/chat.postMessage)
The server responded with: {'ok': False, 'error': 'ratelimited'}
mikegrima commented 1 year ago

In addition to this, Slack also needs to have a try...except in the _post_message function. Upon a caught exception, it should return False.

mikegrima commented 1 year ago

Addressed the exception handling in #62. I would like to also detect if a message is bigger than 3001 characters and alert a note that it was too big.