search

iprak / yahoofinance

Home Assistant component which allows you to get stock updates from Yahoo finance.
MIT License
81 stars 11 forks source link

Handle cookies consent page #97

Closed swiergot closed 1 year ago

swiergot commented 1 year ago

Fixes #96.

iprak commented 1 year ago

I am seeing https://consent.yahoo.com/ being accessible, maybe it is region specific. image

I have been playing with a different approach of first navigating to yahoo.com and grabbing the cookies from there. And then navigating to https://query1.finance.yahoo.com/v1/test/getcrumb. The latter appears when accessing https://finance.yahoo.com/quote/NQ%3DF.

swiergot commented 1 year ago

When I simply go to consent.yahoo.com, I get the error page too. But when I open an incognito window and go to https://finance.yahoo.com/quote/NQ%3DF, I get redirected to consent.yahoo.com with some parameters. I think it just requires a valid session or something which is created only on regular pages.

It is the same every time I start with a fresh incognito window. The same happens to my HA when your component tries to open https://finance.yahoo.com/quote/NQ%3DF for the first time. After the consent form is submitted (with my changes), it redirects to https://finance.yahoo.com/quote/NQ%3DF and everything is as it used to be.

If you don't live in the EU, it is entirely possible that you will never be shown the cookies consent screen as it is required by the EU law. I suppose some websites take the easy way and show the consent form to everyone without exception, but there are also websites that show it or not based on geolocation.

iprak commented 1 year ago

That makes sense, I have never seen that consent page. I did VPN to Germany but still was not able to access that consent page. In US, the request just errors out with 401.

swiergot commented 1 year ago

I assume you tried incognito when on VPN?

swiergot commented 1 year ago

Just to be clear, API URLs do not redirect to consent.yahoo.com, only yahoo.com and finance.yahoo.com do.

I have also tried https://query1.finance.yahoo.com/v1/test/getcrumb but it works only after I visit yahoo.com and submit the consent form. So no real gain.

swiergot commented 1 year ago

One additional note - this PR also fixes a bug in handling of crumbs containing a slash character encoded as a Unicode escape sequence.

Not related to the consent page. I suspect it is the reason why sometimes sensors become unavailable and come back after a while.

iprak commented 1 year ago

Got it. It works well from US region. Could we add something to the logging to differentiate it from the same logging within CONSENT_HOST check?

   _LOGGER.debug("Response status: %d", response.status)
   _LOGGER.debug("Response URL: %s", response.url)

Never mind, I have an idea to log the content as well. So I will adjust the logging. Thanks for looking into this.