search

Benny- / Yahoo-ticker-symbol-downloader

A web scraper for ticker symbols from yahoo finance
https://pypi.python.org/pypi/Yahoo-ticker-downloader/
Other
247 stars 87 forks source link

Rate of download #21

Closed nick-torenvliet closed 7 years ago

nick-torenvliet commented 7 years ago

Benny,

Thanks for your code.

My understanding is that yahoo will only accept 2000 requests on its API per hour from an ip authenticated client.

I would like to get a comprehensive list of available ticker symbols. I feel like slowing down your script might work.

What part of your code would a I have tweak in order to bring down the number of requests your script makes to 2000 per hour?

Am I on the right path here?

Benny- commented 7 years ago

I never heard about this 2000 request limit. Can you provide a source? Also, this piece of software does not use an official API. It uses the html interface for humans, so API restrictions do not apply but some unknown (to me) rate limiting/abuse prevention algorithm might be in play.

At the moment the program waits 5 seconds after every request. So the amount of HTTP requests every hour at the moment is ~720. You can modify this value in YahooTickerDownloader.py.

nick-torenvliet commented 7 years ago

These are the limits on the yql api https://developer.yahoo.com/yql/guide/usage_info_limits.html

I understand that you are scraping the web-site and these limits don't apply.

Do you have any experience with slowing down the query rate? Does it succesfully avoid the abuse prevention/rate limiting issue?

Benny- commented 7 years ago

The last time it ran it ran without any problems. If you have any problems please let me know.

nick-torenvliet commented 7 years ago

Cool - I'll send some total numbers over your way when I run through them all and we can compare.

Thanks man!!

Sent from my iPhone

On Feb 11, 2017, at 12:43 PM, Benny Jacobs notifications@github.com<mailto:notifications@github.com> wrote:

The last time it ran it ran without any problems. If you have any problems please let me know.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Benny-/Yahoo-ticker-symbol-downloader/issues/21#issuecomment-279162647, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AYhIs4Ei2WnIrWqjV-jGdmVrX26yen7eks5rbfMzgaJpZM4L-OBN.

Benny- commented 7 years ago

Sleep time between download is now adjustable using the -s flag on master.

Benny- commented 7 years ago

Please let me know if you ever got a complete set using the most recent version.