search

ranaroussi / yfinance

Download market data from Yahoo! Finance's API
https://aroussi.com/post/python-yahoo-finance
Apache License 2.0
13.11k stars 2.33k forks source link

I'd like to use yfinance async but get a invalid crump. #1787

Closed HumanBot000 closed 9 months ago

HumanBot000 commented 9 months ago

Describe bug

info = info['quoteSummary']['result'][0]

KeyError: 'quoteSummary' {'finance': {'result': None, 'error': {'code': 'Unauthorized', 'description': 'Invalid Crumb'}}}

Simple code that reproduces your problem

class YFinance: user_agent_key = "User-Agent" user_agent_value = ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) " "AppleWebKit/537.36 (KHTML, like Gecko) " "Chrome/58.0.3029.110 Safari/537.36")

def __init__(self, ticker):
    self.yahoo_ticker = ticker

def __str__(self):
    return self.yahoo_ticker

async def _get_yahoo_cookie(self):
    cookie = None
    headers = {self.user_agent_key: self.user_agent_value}
    async with aiohttp.ClientSession() as session:
        async with session.get("https://fc.yahoo.com", headers=headers,allow_redirects=True) as response:
            cookie = response.cookies
            aiohttp_cookies = await self.convert_cookie(cookie)
            requests_cookies = requests.cookies.RequestsCookieJar()
            for key, value in aiohttp_cookies.items():
                requests_cookies.set(key, value)
            if not response.cookies:
                raise Exception("Failed to obtain Yahoo auth cookie.")

            cookie = list(requests_cookies)[0]

    return cookie

async def convert_cookie(self,cookie_string):
    # Convert aiohttp cookie string to a dictionary
    cookies = SimpleCookie()
    cookies.load(cookie_string)
    aiohttp_cookies = {cookie.key: cookie.value for cookie in cookies.values()}

    return aiohttp_cookies
async def _get_yahoo_crumb(self, cookie):
    crumb = None
    headers = {self.user_agent_key: self.user_agent_value}
    async with aiohttp.ClientSession() as session:
        async with session.get(
            "https://query1.finance.yahoo.com/v1/test/getcrumb",
            headers=headers,
            cookies={cookie.name: cookie.value},
            allow_redirects=True,
        ) as response:
            crumb = response.text

    if crumb is None:
        raise Exception("Failed to retrieve Yahoo crumb.")

    return str(crumb)

async def get_history(self, period="max"):
    # Obtain cookies and crumb for downloading historical data
    cookie = await self._get_yahoo_cookie()
    crumb = await self._get_yahoo_crumb(cookie)

    # Fetch historical stock data using yfinance
    stock = yfinance.Ticker(self.yahoo_ticker)
    return stock.history(period=period)

@property
async def info(self):
    cookie = await self._get_yahoo_cookie()
    crumb = await self._get_yahoo_crumb(cookie)
    print(crumb)
    info = {}
    ret = {}

    headers = {self.user_agent_key: self.user_agent_value}

    yahoo_modules = ("summaryDetail,"
                     "financialData,"
                     "quoteType,"
                     "assetProfile,"
                     "indexTrend,"
                     "defaultKeyStatistics")

    url = ("https://query1.finance.yahoo.com/v10/finance/"
           f"quoteSummary/{self.yahoo_ticker}"
           f"?modules={urllib.parse.quote_plus(yahoo_modules)}"  # Use urllib.parse.quote_plus
           f"&ssl=true&crumb={urllib.parse.quote_plus(crumb)}")  # Use urllib.parse.quote_plus
    async with aiohttp.ClientSession(cookies={cookie.name: cookie.value}) as session:
        async with session.get(url, headers=headers, allow_redirects=True) as response:
            info = await response.json()
    print(info)
    info = info['quoteSummary']['result'][0]

    for mainKeys in info.keys():
        for key in info[mainKeys].keys():
            if isinstance(info[mainKeys][key], dict):
                try:
                    ret[key] = info[mainKeys][key]['raw']
                except (KeyError, TypeError):
                    pass
            else:
                ret[key] = info[mainKeys][key]

    return ret

  info = YFinance("MSF.DE")
    info = await info.info

Debug log

-

Bad data proof

No response

yfinance version

0.2.26

Python version

3.10

Operating system

Microsoft Windows [Version 10.0.22621.2715]

ValueRaider commented 9 months ago

Are you up-to-date?

HumanBot000 commented 9 months ago

Acctually With the newest version, I get the same error

HumanBot000 commented 9 months ago

@ValueRaider

ValueRaider commented 9 months ago

Can you simplify the example? I only see yfinance used in one function, and that function never called. Don't confuse yfinance with Yahoo.

HumanBot000 commented 9 months ago

I had to use subclassing because of the bug some weeks ago.

Now I tried to use aiohttp for async handling.

class YFinance:
    user_agent_key = "User-Agent"
    user_agent_value = ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) "
                        "AppleWebKit/537.36 (KHTML, like Gecko) "
                        "Chrome/58.0.3029.110 Safari/537.36")

    def __init__(self, ticker):
        self.yahoo_ticker = ticker

    def __str__(self):
        return self.yahoo_ticker

    async def _get_yahoo_cookie(self):
        cookie = None
        headers = {self.user_agent_key: self.user_agent_value}
        async with aiohttp.ClientSession() as session:
            async with session.get("https://fc.yahoo.com", headers=headers,allow_redirects=True) as response:
                cookie = response.cookies
                aiohttp_cookies = await self.convert_cookie(cookie)
                requests_cookies = requests.cookies.RequestsCookieJar()
                for key, value in aiohttp_cookies.items():
                    requests_cookies.set(key, value)
                if not response.cookies:
                    raise Exception("Failed to obtain Yahoo auth cookie.")

                cookie = list(requests_cookies)[0]

        return cookie

    async def convert_cookie(self,cookie_string):
        # Convert aiohttp cookie string to a dictionary
        cookies = SimpleCookie()
        cookies.load(cookie_string)
        aiohttp_cookies = {cookie.key: cookie.value for cookie in cookies.values()}

        return aiohttp_cookies
    async def _get_yahoo_crumb(self, cookie):
        crumb = None
        headers = {self.user_agent_key: self.user_agent_value}
        async with aiohttp.ClientSession() as session:
            async with session.get(
                "https://query1.finance.yahoo.com/v1/test/getcrumb",
                headers=headers,
                cookies={cookie.name: cookie.value},
                allow_redirects=True,
            ) as response:
                print(str(response.text))
                crumb = response.text

        if crumb is None:
            raise Exception("Failed to retrieve Yahoo crumb.")

        return str(crumb)

    async def get_history(self, period="max"):
        # Obtain cookies and crumb for downloading historical data
        cookie = await self._get_yahoo_cookie()
        crumb = await self._get_yahoo_crumb(cookie)

        # Fetch historical stock data using yfinance
        stock = yfinance.Ticker(self.yahoo_ticker)
        return stock.history(period=period)

    @property
    async def info(self):
        cookie = await self._get_yahoo_cookie()
        crumb = await self._get_yahoo_crumb(cookie)
        print(crumb)
        info = {}
        ret = {}

        headers = {self.user_agent_key: self.user_agent_value}

        yahoo_modules = ("summaryDetail,"
                         "financialData,"
                         "quoteType,"
                         "assetProfile,"
                         "indexTrend,"
                         "defaultKeyStatistics")

        url = ("https://query1.finance.yahoo.com/v10/finance/"
               f"quoteSummary/{self.yahoo_ticker}"
               f"?modules={urllib.parse.quote_plus(yahoo_modules)}"  # Use urllib.parse.quote_plus
               f"&ssl=true&crumb={urllib.parse.quote_plus(crumb)}")  # Use urllib.parse.quote_plus
        async with aiohttp.ClientSession(cookies={cookie.name: cookie.value}) as session:
            async with session.get(url, headers=headers, allow_redirects=True) as response:
                info = await response.json()
        print(info)
        info = info['quoteSummary']['result'][0]

        for mainKeys in info.keys():
            for key in info[mainKeys].keys():
                if isinstance(info[mainKeys][key], dict):
                    try:
                        ret[key] = info[mainKeys][key]['raw']
                    except (KeyError, TypeError):
                        pass
                else:
                    ret[key] = info[mainKeys][key]

        return ret

info = YFinance("MSF.DE")
 info = await info.info
HumanBot000 commented 9 months ago

<bound method ClientResponse.text of <ClientResponse(https://query1.finance.yahoo.com/v1/test/getcrumb) [200 OK]> <CIMultiDictProxy('Content-Type': 'text/plain;charset=utf-8', 'Cache-Control': 'private', 'x-frame-options': 'SAMEORIGIN', 'x-envoy-upstream-service-time': '1', 'Date': 'Fri, 15 Dec 2023 20:06:05 GMT', 'Server': 'ATS', 'x-envoy-decorator-operation': 'finance-yql--mtls-default-production-ir2.finance-k8s.svc.yahoo.local:4080/*', 'Age': '0', 'Strict-Transport-Security': 'max-age=31536000', 'Referrer-Policy': 'no-referrer-when-downgrade', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Expect-CT': 'max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff')>

My crumb:

HumanBot000 commented 9 months ago

and?

ValueRaider commented 9 months ago
#### \*\*\* IMPORTANT LEGAL DISCLAIMER \*\*\* --- **Yahoo!, Y!Finance, and Yahoo! finance are registered trademarks of Yahoo, Inc.** yfinance is **not** affiliated, endorsed, or vetted by Yahoo, Inc. It's an open-source tool that uses Yahoo's publicly available APIs, and is intended for research and educational purposes. **You should refer to Yahoo!'s terms of use** ([here](https://policies.yahoo.com/us/en/yahoo/terms/product-atos/apiforydn/index.htm), [here](https://legal.yahoo.com/us/en/yahoo/terms/otos/index.html), and [here](https://policies.yahoo.com/us/en/yahoo/terms/index.htm)) **for details on your rights to use the actual data downloaded. Remember - the Yahoo! finance API is intended for personal use only.**
HumanBot000 commented 9 months ago

Is there a async implementation in the package?

HumanBot000 commented 9 months ago

?

@ValueRaider

ValueRaider commented 9 months ago

Have a look: https://github.com/ranaroussi/yfinance/tree/main