dockerhub containers not up-to-date with registry

[osallou]

seems details are not fetched for all containers.... (example seaborn...)

logs show:

... INFO:biocontainers.dockerhub.models:tool retrieved from DockerHub -- pandas INFO:biocontainers.dockerhub.models:tool retrieved from DockerHub -- seaborn INFO:biocontainers.dockerhub.models:tool retrieved from DockerHub -- biocontainer-pipelines

then fail on other pages

Traceback (most recent call last): File "pipelines.py", line 268, in main() File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(args, kwargs) File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, *kwargs) File "pipelines.py", line 234, in main import_dockerhub_containers(config, config_profile) File "pipelines.py", line 63, in import_dockerhub_containers dockerhub_containers = reader.get_containers() File "/biocontainers-backend/biocontainers/dockerhub/models.py", line 114, in get_containers response = call_api(url) File "/usr/local/lib/python3.7/site-packages/ratelimit/decorators.py", line 112, in wrapper return func(args, kargs) File "/usr/local/lib/python3.7/site-packages/ratelimit/decorators.py", line 80, in wrapper return func(*args, **kargs) File "/biocontainers-backend/biocontainers/common/utils.py", line 13, in call_api raise Exception('API response: {}'.format(response.status_code)) Exception: API response: 429

[ypriverol]

I will take care of it.

[osallou]

429 Too Many Requests The user has sent too many requests in a given amount of time ("rate limiting").

[osallou]

seems we need to temporize, but we have to see dockerhub api limits

[osallou]

log says that sending report to slack, you should see errors somewhere

[ypriverol]

Yes, we have the issue there.

[osallou]

https://docs.docker.com/docker-hub/api/latest/#tag/resources

If you have hit the limit, you will receive a response status of 429 and the X-Retry-After header in the response.

The X-Retry-After header is a unix timestamp of when you can call the API again.

*Note: These rate limits are separate from anti-abuse and Docker Hub

download, or pull rate limiting.

[osallou]

sample api call via curl shows:

 x-ratelimit-limit: 180
 x-ratelimit-reset: 1683218176
 x-ratelimit-remaining: 180

[mboudet]

@ypriverol Is this issue still active? Newly merged containers do not seems to be in the registry. (Ex: https://github.com/BioContainers/containers/pull/550)