search

Benny- / Yahoo-ticker-symbol-downloader

A web scraper for ticker symbols from yahoo finance
https://pypi.python.org/pypi/Yahoo-ticker-downloader/
Other
247 stars 87 forks source link

Searchterms in null #51

Open Benny- opened 5 years ago

Benny- commented 5 years ago

From #50 :

Saving downloader to disk... Downloader successfully saved.

req https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=null?device=console&returnMeta=true Retry attempt: 1 of 5. Sleep period: 5 seconds. req https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=null?device=console&returnMeta=true Retry attempt: 2 of 5. Sleep period: 25 seconds. req https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=null?device=console&returnMeta=true Retry attempt: 3 of 5. Sleep period: 125 seconds. req https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=null?device=console&returnMeta=true Retry attempt: 4 of 5. Sleep period: 625 seconds. req https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=null?device=console&returnMeta=true Retry attempt: 5 of 5. Sleep period: 3125 seconds. req https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=null?device=console&returnMeta=true A exception occurred while downloading. Suspending downloader to disk Successfully saved download state Try removing {type}.pickle file if this error persists Issues can be reported on https://github.com/Benny-/Yahoo-ticker-symbol-downloader/issues

Traceback (most recent call last): File "/home/mountainers/.local/bin/YahooTickerDownloader.py", line 170, in main() File "/home/mountainers/.local/bin/YahooTickerDownloader.py", line 111, in main downloadEverything(downloader, tickerType, args.insecure, args.sleep, args.pandantic) File "/home/mountainers/.local/bin/YahooTickerDownloader.py", line 40, in downloadEverything symbols = downloader.nextRequest(insecure, pandantic) File "/home/mountainers/.local/lib/python3.6/site-packages/ytd/SimpleSymbolDownloader.py", line 95, in nextRequest json = self._fetch(insecure) File "/home/mountainers/.local/lib/python3.6/site-packages/ytd/SimpleSymbolDownloader.py", line 65, in _fetch resp.raise_for_status() File "/home/mountainers/.local/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=null?device=console&returnMeta=true

Benny- commented 5 years ago

I need some information to debug that new problem. What is your python version and how do you invoke the program?

chicoriagithub commented 5 years ago

my version is :+1: Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information.

chicoriagithub commented 5 years ago

i open the terminal and write yahooticketdowloader.py

Benny- commented 5 years ago

I could not reproduce this problem using Python 2.7.15rc1 on Ubuntu 18:10.

Which version of Yahoo-ticker-symbol-downloader do you use?

chicoriagithub commented 5 years ago

3.0

El El dom, 13 ene 2019 a las 19:55, Benny Jacobs notifications@github.com escribió:

I could not reproduce this problem using Python 2.7.15rc1 on Ubuntu 18:10.

Which version of Yahoo-ticker-symbol-downloader do you use?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Benny-/Yahoo-ticker-symbol-downloader/issues/51#issuecomment-453855764, or mute the thread https://github.com/notifications/unsubscribe-auth/AC2zyBO1mxJd1XUlFRODIKaPJfkc0BMfks5vC4EagaJpZM4Z3UgZ .

Eppah-nj commented 5 years ago

Hi, I am using the last version of Yahoo-ticker-symbol-downloader (3.0.1) with Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 12:45:02) [MSC v.1900 32 bit (Intel)] on win32. I just wanted to know if you are working on it or if you have any workaround.

Bruno-Hanzen commented 5 years ago

I have had the same issue with the "null" search term.I think that the issue is on the Yahoo side, not on the python side. If you type the URL (https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=null?device=console&returnMeta=true) in a browser, you'll get some kind of yahoo homepage, not at all the expected result. I have edited the "_fetch" method of SimpleSymbolDownloader" and added a test on "null":

if self.current_q != "null": params = { 'searchTerm': self.current_q, } else: params = { 'searchTerm': "nulll", } Quick and dirty...

NLockwood commented 5 years ago

Thank you Benny for providing this amazing tool. Also, thank you Bruno-Hanzen! Your fix works.

AwesomeCap commented 5 years ago

Hi, I have the same problem.

req https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=null?device=console&returnMeta=true Retry attempt: 1 of 5. Sleep period: 5 seconds. ...

Using Python 3.6 on a Mac.