Slack API Rate limiting

[LaurierMantel]

Issue

tl;dr I was trying to figure out why some Slack read query issues were still happening regardless of fixes yesterday. Slack rate limits our API calls. @kushthaker @Callum-Mitchell

First (checking counts of successful Slack Conversation queries on a per user basis)

SELECT COUNT(*) as query_count, slack_conversation_query_run_id, slack_user_id, MAX(query_datetime) FROM slack_conversation_queries GROUP BY slack_conversation_query_run_id, slack_user_id ORDER BY slack_conversation_query_run_id DESC LIMIT 100

Then (looking for when these are < 5, as most users are in far more than 5 Slack channels.

SELECT * FROM (SELECT COUNT(*) as query_count, slack_conversation_query_run_id, slack_user_id, MAX(query_datetime) FROM slack_conversation_queries GROUP BY slack_conversation_query_run_id, slack_user_id ORDER BY slack_conversation_query_run_id DESC LIMIT 1000) src WHERE src.query_count < 5 LIMIT 20;

These show that misses in SlackConversationQuery stuff is kind of random.

I put in a job to record if an error was occurring (the only explanation) And a whole lot of these happened:

[Thu Mar 05 18:00:24.508790 2020] [:error] [pid 11895] 
[Thu Mar 05 18:00:24.587762 2020] [:error] [pid 11895] Excepted SlackApiError:
[Thu Mar 05 18:00:24.587792 2020] [:error] [pid 11895] The request to the Slack API failed.
[Thu Mar 05 18:00:24.587796 2020] [:error] [pid 11895] The server responded with: {'ok': False, 'error': 'ratelimited'}

Gonna read up on here and figure out workaround: https://api.slack.com/docs/rate-limits

[LaurierMantel]

This is super frustrating https://api.slack.com/docs/rate-limits https://api.slack.com/methods/conversations.info Broadly, you'll encounter limits like these, applied on a "per app per workspace" basis.

The Slack Conversation Info (the one we use to get slack_conversation_reads is rate limit Tier 3

What this means This means that at scale, getting conversation reads for (> 5 users per workspace) does not work. 10 users * 20 conversations per user = 100 >> 50

What we can do

• Complain to Slack (I will try but this will almost certainly lead to nothing)

I have written a fix to make our stuff slightly more efficient on API calls but for now it's very frustrating.

[LaurierMantel]

[Thu Mar 05 19:00:34.642538 2020] [:error] [pid 14100] For user w/ ID: 3, 19 new SlackConversationQuery rows were created
[Thu Mar 05 19:00:35.938531 2020] [:error] [pid 14100] For user w/ ID: 6, 20 new SlackConversationQuery rows were created
[Thu Mar 05 19:00:37.042829 2020] [:error] [pid 14100] For user w/ ID: 1, 11 new SlackConversationQuery rows were created

Further proof: they are cutting us off right at 50 (I removed all of the logs about rate limiting).

[LaurierMantel]

I filed a request to Slack, to reduce rate limiting or make the last_read accessible in conversations_list (it is accessible in the API call for private channels).

[LaurierMantel]

Y'all can't access this but my ticket is here for safekeeping. https://doxahq.slack.com/help/requests/2641239

[LaurierMantel]

Have had a lot of updates from Slack about this; basically my boy agrees with me that they API is inconsistent and needs improvement