search

GoldenCheetah / OpenData

A project to collect, collate and share an open data set with contributions from users of the GoldenCheetah application
38 stars 6 forks source link

OSF rate limiting #4

Closed AartGoossens closed 5 years ago

AartGoossens commented 6 years ago

I'm regularly bumping into 429 RuntimeErrors when downloading data. Apparently access to OSF is rate limited:

Authenticated requests have a rate limit of 10,000/day. Unauthenticated requests have a rate limit of 100/hour.

How I make use of authenticated requests? It appears that having an account at osf.io is not enough. Do I need to fork the project or request access to the project to be able to make use of the higher authenticated rate limit?

liversedge commented 6 years ago

Are you using a personal access token ? https://developer.osf.io/#tag/General-Usage

AartGoossens commented 6 years ago

Are you using a personal access token ?

I didn't try that before (did use username+password) but I still get the 429's with the token now. I suspect I need to have access to the OpenData project on OSF to be able to make use of the authenticated rate limits.

liversedge commented 6 years ago

Its open, so access is public! Do you need to apply for access!?

AartGoossens commented 6 years ago

Yes, access is public of course, but it might be rate limited according to the unauthenticated rate limit. The question is how I can use the authenticated rate limit...

AartGoossens commented 6 years ago

To be clear: it does work from time to time, just not all the time. And tonight apparently some other people are downloading a lot of files too so I keep hitting the rate limit...

AartGoossens commented 5 years ago

Closing this issue as it is resolved by moving (well, actually copying) the data to AWS S3.