search

chaoss / grimoirelab-elk

GNU General Public License v3.0
59 stars 120 forks source link

[Slack]: method_deprecated error for slack endpoints #918

Closed linonymous closed 11 months ago

linonymous commented 4 years ago

Grimoire elk run for slack channel errors out with following error stack.

last retry took 1m42.098861066s: 2020-07-20 01:24:14,086 [slack] Incremental from: 2020-07-17 21:14:17.460000+00:00 for https://slack.com/C07JH9WMQ
2020-07-20 01:24:14,087 Fetching messages of 'C07JH9WMQ' channel from 2020-07-17 21:14:17.460000+00:00
2020-07-20 01:24:15,598 Error feeding raw from slack (https://slack.com/C07JH9WMQ): method_deprecated
Traceback (most recent call last):
  File "/repos/grimoirelab-elk/grimoire_elk/elk.py", line 162, in feed_backend
    ocean_backend.feed(**params)
  File "/repos/grimoirelab-elk/grimoire_elk/raw/elastic.py", line 228, in feed
    self.feed_items(items)
  File "/repos/grimoirelab-elk/grimoire_elk/raw/elastic.py", line 244, in feed_items
    for item in items:
  File "/repos/grimoirelab-perceval/perceval/backend.py", line 226, in fetch
    for item in self.fetch_items(category, **kwargs):
  File "/repos/grimoirelab-perceval/perceval/backends/core/slack.py", line 137, in fetch_items
    oldest=oldest, latest=latest)
  File "/repos/grimoirelab-perceval/perceval/backends/core/slack.py", line 388, in history
    response = self._fetch(resource, params)
  File "/repos/grimoirelab-perceval/perceval/backends/core/slack.py", line 442, in _fetch
    raise SlackClientError(error=result['error'])
perceval.backends.core.slack.SlackClientError: method_deprecated
2020-07-20 01:24:15,599 [slack] Done collection for https://slack.com/C07JH9WMQ

Further investigated, certain slack APIs are officially deprecated and can not be used for new slack apps. as mentioned here: https://api.slack.com/changelog 

Beginning June 10th, 2020, all newly created Slack apps will be unable to use the deprecated methods in channels.*, im.*, mpim.*, and groups.*. Existing apps have until February 24th, 2021 to migrate to the Conversations API.

For example: perceval uses https://api.slack.com/methods/channels.history api call to fetch channel data which is now deprecated and is moved to https://api.slack.com/methods/conversations.history

valeriocos commented 4 years ago

Thank you for reporting this issue @linonymous . The current release (0.2.42) uses the new endpoint conversations.history (https://github.com/chaoss/grimoirelab-perceval/pull/657). You should create a slack app and use its user token.

However, the upcoming release 0.2.43 will revert back this change to use the old endpoints. This is due to the fact that the endpoint https://api.slack.com/methods/conversations.history doesn't behave as the old https://api.slack.com/methods/channels.history, in particular all messages of type reply are not returned anymore (check https://github.com/chaoss/grimoirelab-perceval/pull/677 for further details).

We started a conversation with the Slack folks, they acknowledged the limitation with the new endpoint and said that they will evaluate the possibility to implement the old behavior (e.g., an endpoint were all type of messages are returned in a incremental way).

lukaszgryglicki commented 4 years ago

Which slack endpoint is used by the master branch? Because we're using master branches from the following repos:

linonymous commented 4 years ago

@valeriocos Thanks for the quick response! One more question, How to configure specific version of perceval backend with grimoire elk?

valeriocos commented 4 years ago

Which slack endpoint is used by the master branch?

channels.history (ref https://github.com/chaoss/grimoirelab-perceval/commit/fb075a88ce495e65a1e87462f9dd7e662300ddad), @lukaszgryglicki . You shouldn't have any problem since (old) legacy tokens will keep working (https://api.slack.com/legacy/custom-integrations/legacy-tokens). The problem appears when you need to track the data of a new slack instance (as reported by @linonymous).

.., How to configure specific version of perceval backend with grimoire elk?

You're welcome @linonymous! There is no specific support for this and the Perceval pypi package isn't up-to-date. I guess (I have never tried this before) you can:

linonymous commented 4 years ago

Thanks for the approach @valeriocos. Seems like slack has provided a separate API for replies. https://api.slack.com/methods/conversations.replies so are we going to implement this new API or slack is going to return to their old behavior anytime soon?

valeriocos commented 4 years ago

The https://api.slack.com/methods/conversations.replies isn't that good for GrimoireLab since:

A solution could be to iterate over all messages we have in ElasticSearch and constantly get their replies, however this would make the process inefficient.

On the other hand, Slack provides the https://api.slack.com/events-api which allows to get all messages that occurr on a channel in real time. The main limitations here are that:

are we going to implement this new API or slack is going to return to their old behavior anytime soon?

ATM the decision is to use the old endpoints and wait for Slack's feedback. As commented at https://github.com/chaoss/grimoirelab-elk/issues/918#issuecomment-660928175, Slack folks are aware of this limitation, however I ignore how feature requests and corresponding ETAs are handled by them. Said that, it seems that the lack of replies in the conversations.history is a problem reported by others too (https://www.bakejam.com/slacks-conversationshistory-api-ignores-replies/).

If you find it convenient, you can interact with the Slack support and try to gather more information about this limitation and ETA (https://slack.com/intl/en-es/help).

Hope this helps!

jjmerchante commented 11 months ago

This was changed at https://github.com/chaoss/grimoirelab-perceval/commit/fb075a88ce495e65a1e87462f9dd7e662300ddad and reverted at https://github.com/chaoss/grimoirelab-perceval/commit/67ba5042ba1882b28b56588af311cb524e423014. It looks like it is using the right end-point.