Cookie/crumb issues after about 10-20 minutes

[gadicc]

Originally posted by @samducker in https://github.com/gadicc/node-yahoo-finance2/issues/736#issuecomment-1959844563

Hi its working on restart of my server, but then seems to stop working within 10-20mins.

My server is based in US EAST. How can I force the cookie to renew on each request?

Is there a way to force cookie refresh if I get an error like this?

I am running a honojs server on railway with no serverless (its constantly running). Naturally storing the cookie in a volume, doesn't seem to make any sense if I'm needing to refresh the cookie.

with followup:

No worries thank you for this library and your amazing work @gadicc I felt bad to tag :D, but I have a client on my back.

In the meantime I introduced a caching solution on my end which has limited the impact of my problem. So no worries, enjoy the weekend.

Happy to follow up with my sentry logs if helpful.

p.s. I am happy to submit a PR, if you can point me in the right direction also <3!

[gadicc]

Thanks @samducker for your reply. Glad you were able to mitigate the impact somewhat, but yes, let's try solve this properly.

Yes, the full logs would be great... maybe it will give me an indication of where things are getting stuck. In the meantime, going to share a bit about the insides in case you want to take a look too.

The relevant code is in src/lib/getCrumb.ts, and actually on L36-40 we have:

  if (crumb && !noCache) {
    // If we still have a valid (non-expired) cookie, return the existing crumb.
    const existingCookies = await cookieJar.getCookies(url, { expire: true });
    if (existingCookies.length) return crumb;
  }

Unfortunately noCache option is not exposed in the API and is only used for testing, so no easy workaround there. The issue is, it really should be noticing that the cookie is expired here, and fetch a new one automatically. So we'll have to check that too. I'd like to understand if it's trying to re-use an expired cookie or if something else is going wrong. I also need to look at the code more carefully and double check what happens to the existing crumb when the cookie is refreshed.

I guess I really need to set up a long running process to track this properly, and will do that when I get the chance. However, I'd love to give you more options in the meantime. You could also try manually clearing the cookieJar, someting like:

const yf2 = require('yahoo-finance2').default;

setInterval(() => {
    yf2._opts.cookieJar.removeAllCookiesSync();
}, 60_000 * 10);

and see how that goes, too. Let's keep in touch.

[nocodehummel]

Not sure if below error is related to the above. Requesting summary data on a fresh started server results in an invalid crumb. I did not yet try the suggestion to empty the cookieJar.

const res = await yahooFinance.quoteSummary(symbol, {
    modules: ['summaryProfile', 'quoteType', 'financialData'],
});

Results in below error.

Fetching crumb and cookies from https://finance.yahoo.com/quote/AAPL...
fetch https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=HaNZK6c&done=https%3A%2F%2Ffinance.yahoo.com%2Fquote%2FAAPL
fetch https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_9b772abd-9720-4015-8db3-f287cffe7253
fetch https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_9b772abd-9720-4015-8db3-f287cffe7253
fetch https://guce.yahoo.com/copyConsent?sessionId=3_cc-session_9b772abd-9720-4015-8db3-f287cffe7253&lang=sv-SE
Fetching crumb and cookies from https://finance.yahoo.com/quote/AAPL?guccounter=1...
Success. Cookie expires on Infinity
New crumb: yRG6Yj2wkbt
Error: Invalid Crumb

[NarendranKT]

Not sure if below error is related to the above. Requesting summary data on a fresh started server results in an invalid crumb. I did not yet try the suggestion to empty the cookieJar.

const res = await yahooFinance.quoteSummary(symbol, {
  modules: ['summaryProfile', 'quoteType', 'financialData'],
});

Results in below error.

Fetching crumb and cookies from https://finance.yahoo.com/quote/AAPL...
fetch https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=HaNZK6c&done=https%3A%2F%2Ffinance.yahoo.com%2Fquote%2FAAPL
fetch https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_9b772abd-9720-4015-8db3-f287cffe7253
fetch https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_9b772abd-9720-4015-8db3-f287cffe7253
fetch https://guce.yahoo.com/copyConsent?sessionId=3_cc-session_9b772abd-9720-4015-8db3-f287cffe7253&lang=sv-SE
Fetching crumb and cookies from https://finance.yahoo.com/quote/AAPL?guccounter=1...
Success. Cookie expires on Infinity
New crumb: yRG6Yj2wkbt
Error: Invalid Crumb

Yeah for me too this issue has been live today

[samducker]

I mitigated the issue by moving to serverless. Meaning essentially a new server / cookie would spawn and also adding a cache infront of my api meaning I made less requests and if a request failed I would still be able to return the previous fetched result.

[gadicc]

Hi, all. Bad news, I'm afraid. Please see https://github.com/gadicc/node-yahoo-finance2/issues/764 for more info and for our official stance.

[watadarkstar]

@gadicc I believe this issue can be closed now given this fix got released. No?

https://github.com/gadicc/node-yahoo-finance2/issues/764#issuecomment-2061075355

[gadicc]

Thanks, @watadarkstar! As much as I'd love to close this, there are two different issues here. The original issue is an issue with cookies on long running processes. The OP moved over to serverless so there hasn't been much further movement on this... I'd love to leave the issue open in case anyone notices this and can provide more input (or if I eventually ever have time to do long running debug sessions :sweat_smile:).

Then there was the more recent crumb issue of last week which through the usual magic of open source collaboration, we managed to solve :) But that issue is newer than the original issue here, even though it was related.

Nevertheless, thanks for chiming in. I admit the multitude of issues are a bit overwhelming for me so I do appreciate any help in getting some closed! :pray: :grin:

[Theo92100]

@gadicc i get this issue after several requests ( sometimes just 1 or 2 ) and with several delay intervals in between if anyone can help ? Error: No set-cookie header on collectConsentSubmitResponse, please report. at _getCrumb (webpack-internal:///(rsc)/./node_modules/.pnpm/yahoo-finance2@2.11.2/node_modules/yahoo-finance2/dist/esm/src/lib/getCrumb.js:106:27) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.yahooFinanceFetch [as _fetch] (webpack-internal:///(rsc)/./node_modules/.pnpm/yahoo-finance2@2.11.2/node_modules/yahoo-finance2/dist/esm/src/lib/yahooFinanceFetch.js:62:23) at async Object.moduleExec [as _moduleExec] (webpack-internal:///(rsc)/./node_modules/.pnpm/yahoo-finance2@2.11.2/node_modules/yahoo-finance2/dist/esm/src/lib/moduleExec.js:58:18) at async Object.quote (webpack-internal:///(rsc)/./node_modules/.pnpm/yahoo-finance2@2.11.2/node_modules/yahoo-finance2/dist/esm/src/modules/quote.js:9:21)