Meetup collection broken due to rate limits: sleep-for-rate not working

[canasdiaz]

The Slack collection is not working and prints errors about too many connections. I guess related to the rate limitations. We're already using a dedicated token BTW.

I'm using the version below:

#!/bin/bash
SORTINGHAT='641dabd71d4f4a23547592a10096240343268bd1'
GRIMOIREELK='9a69a31a6499402cddb3c622f12dc5d11e490fed'
PERCEVAL='f1c170ac6692ffe54558d590a96bedf748624651'
KIBITER='f7bf173cb1d79418b9909d877adfb998ce7b52da'
GRIMOIRELAB_GITHUB_IO='2f71c036020f6873efdd079f4ec2b5f26584cb2b'
ARTHUR='3efff3311b52222b7d06bf63ce97f8a42572bc06'
USE_CASES='31ea49ecc28f3e27aa29fddb0e01b7f475069b6a'
PANELS='ffe73323afe84e491d0b1705e86a7db3588bff38'
MORDRED='024019f5ee1df7096f91f7b41286cd58e55e6495'
TRAINING='9df30c3ec955387a423f261bb141f74133bf7ff7'
PERCEVAL_MOZILLA='423615b9745ab0b0e577701504e46e0f32aab6a3'
PERCEVAL_PUPPET='9c0d4c72a0e9b3c7c1e3295f72ebe70c5c874084'
REPORTS='4ad4f7c9d8cd5eb728a95c3a358d8aa339102c75'
GRIMOIRELAB_TOOLKIT='95d271af32744eb38f73df592f2e0dc5ea97683b'
PERCEVAL_OPNFV='0bec845df7e17965892dfb1181da0ab1c77168dc'
GRIMOIRELAB='efd656ff641623e6a639e095f14f38a71cacc094'

This is the Mordred setup being used:

[meetup]
raw_index = meetup_cloudfoundry_170929
enriched_index = meetup_cloudfoundry_170929_enriched_170929
api-token = ****
no-cache = true
sleep-for-rate = true

Below you can see the 'Too Many Requests' error and how it keeps sending queries instead of stopping:

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://api.meetup.com/Berlin-PaaS-Cloud-Foundry-Meetup/events/hrjxjmypsjbdb/comments?page=200&key=6c5d36544b39452392c3746c5f7028&sign=true                       
2017-11-23 15:01:30,815 - grimoire_elk.arthur - ERROR - Error feeding ocean from meetup (https://meetup.com/): 429 Client Error: Too Many Requests for url: https://api.meetup.com/Madison-Cloud-Foundry-Meetup/events?fields=event_hosts,feat
ured,group_topics,plain_text_description,rsvpable,series&status=cancelled,upcoming,past,proposed,suggested,draft&order=updated&page=200&scroll=since%3A2016-05-05T00%3A33%3A18.000Z&key=6c5d36544b39452392c3746c5f7028&sign=true              
Traceback (most recent call last):                                                                                                                                                                                                            
  File "./grimoire_elk/arthur.py", line 130, in feed_backend                                                                                                                                                                                  
    ocean_backend.feed()                                                                                                                                                                                                                      
  File "./grimoire_elk/ocean/elastic.py", line 204, in feed                                                                                                                                                                                   
    for item in items:                                                                                                                                                                                                                        
  File "/usr/local/lib/python3.4/dist-packages/perceval-0.9.5-py3.4.egg/perceval/backend.py", line 360, in decorator                                                                                                                          
    for data in func(self, *args, **kwargs):                                                                                                                                                                                                  
  File "/usr/local/lib/python3.4/dist-packages/perceval-0.9.5-py3.4.egg/perceval/backends/core/meetup.py", line 117, in fetch                                                                                                                 
    for evp in ev_pages:                                                                                                                                                                                                                      
  File "/usr/local/lib/python3.4/dist-packages/perceval-0.9.5-py3.4.egg/perceval/backends/core/meetup.py", line 412, in events                                                                                                                
    for page in self._fetch(resource, params):                                                                                                                                                                                                
  File "/usr/local/lib/python3.4/dist-packages/perceval-0.9.5-py3.4.egg/perceval/backends/core/meetup.py", line 491, in _fetch                                                                                                                
    raise e                                                                                                                                                                                                                                   
  File "/usr/local/lib/python3.4/dist-packages/perceval-0.9.5-py3.4.egg/perceval/backends/core/meetup.py", line 484, in _fetch                                                                                                                
    r.raise_for_status()                                                                                                                                                                                                                      
  File "/usr/local/lib/python3.4/dist-packages/requests/models.py", line 935, in raise_for_status                                                                                                                                             
    raise HTTPError(http_error_msg, response=self)                                                                                                                                                                                            
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://api.meetup.com/Madison-Cloud-Foundry-Meetup/events?fields=event_hosts,featured,group_topics,plain_text_description,rsvpable,series&status=cancelled,upcomi
ng,past,proposed,suggested,draft&order=updated&page=200&scroll=since%3A2016-05-05T00%3A33%3A18.000Z&key=6c5d36544b39452392c3746c5f7028&sign=true                                                                                              
2017-11-23 15:01:35,685 - grimoire_elk.arthur - ERROR - Error feeding ocean from meetup (https://meetup.com/): 403 Client Error: Forbidden for url: https://api.meetup.com/Boulder-Cognitive-Business-Meetup/events/225399399/comments?page=200&key=6c5d36544b39452392c3746c5f7028&sign=true 

[acs]

This is a perceval issue. It must be opened in: https://github.com/grimoirelab/perceval/issues

Could you open it there?

[valeriocos]

The meetup backend implements a mechanism such as if an API call fails with a 429 error message, it tries at most 5 attempts before raising the HTTPError exception. Between every failed call the backend sleeps an amount of time equal to *30 seconds the current attempt* (e.g,, in the third attempt the backend will sleep 30 3 seconds).

You can try to set a greater value than 30 seconds by using the backend parameter sleep_time (e.g., 100).

@sanacl, let me know if this fixes the bug.

[acs]

@sanacl this is a perceval issue. It is better next time that it is opened in perceval repository.

[jgbarah]

Is this fixed already, @sanacl @acs ?

[acs]

Is this fixed already, @sanacl @acs ?

@valeriocos and @sanacl should know it. I don't know.

[sduenas]

We haven't seen this error after it was reported. Closing it. If you find it again, please reopen it.