Open pnadolny13 opened 1 year ago
@pnadolny13 - We're working with hourly rate limits, correct?
Can we split the streams and queries that we need to run, so that they run in alternating hours?
I think that's the first thing I'd consider and then (if we think this can finish within 2-3 hours' limits) I'd lean towards trying to add sleep/wait like the throttle_requests
approach. The compute cost for 3 hours of execution should be pretty minimal, and is par for other larger datasources at some companies.
Neither is a silver bullet and I think there may be other good options as well. Maybe a good topic for Data Office Hours?
I added PRs and issues streams to the github search EL and were starting to hit the rate limits and the Airflow job wont complete. We do a search for taps and targets so every result gets a bunch of follow on requests for PRs/Issues/Readme content and I think our result set is too big to ever complete within the rate limit ranges. Even with incremental loads we still hit ever repo every time for updates and if we have 5k repos then our 5k request limit is used up quickly.
throttle_requests
to stay within rate limit. If the limit is reached it should sleep vs hard failing.Challenges with each:
@aaronsteers any thoughts on this since you've worked with this tap before and run into similar problems?