search

PavlidisLab / gemma.R

An R wrapper for the Gemma RESTful API
https://pavlidislab.github.io/gemma.R/
Apache License 2.0
10 stars 1 forks source link

Sporadic test failures #26

Closed oganm closed 2 years ago

oganm commented 2 years ago

New tests occasionally seem to fail. Possibly due to gemma's rate limiting.

oganm commented 2 years ago

getting HTTP code 429

oganm commented 2 years ago

helpful comment by javier

https://github.com/PavlidisLab/gemma.R/blob/master/tests/testthat/testConvenience.R#L8-L9

oganm commented 2 years ago

Dima's description of the server rate limiting

no more than 2 concurrent connections for "/rest/v2/datasets/*/data" no more than 10req/sec (for everything with API) no more than 100 concurrent connections (for everything with API)

Initially suspected the first point could be the issue since Jordan mentioned using asynchronous calls to speed things up but looking at the code this doesn't seem to be a problem. Since the tests do always seem to pass locally will add the skip clause to those tests for now and return after other fixes in case there is a preventable cause of the failure

oganm commented 2 years ago

This is added a crude fix to the issue. Need to see if errors persist

https://github.com/PavlidisLab/gemma.R/blob/ef61dadfe8782ab018d195bb734eb7e28abab46d/R/processors.R#L40-L44

oganm commented 2 years ago

Ammended the solution to check for null responses and retry up to three times in case of failure. So far all looks fine.