Error fetching market data

[educanorama]

I'm afraid that the problem cited in:

has returned. I saw bakerbrandond's message at the bottom of this thread, and confirm that mop is no longer showing market data. The familiar message:

json: cannot unmarshal object into Go value of type []map[string]interface {}

[educanorama]

The link didn't take in the previous message. Sorry:

https://github.com/mop-tracker/mop/issues/106

[danteodor]

Confirmed on 25-May-2023, Yahoo interface broken ./mop... cannot unmarshal object into Go value of type []map[string]interface {}

curl https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL,GOOG,F 2>/dev/null ... {"finance":{"result":null,"error":{"code":"Unauthorized","description":"Invalid Cookie"}}}

Possible the Yahoo endpoint services are down? This has only been an issue 24 hours (or less) at this point.

[sethmiller]

Apparently they shutdown anonymous access to the API. The response now provides a link to a Google form stating:

We’re sorry for the inconvenience, but API-level access to Yahoo Finance quotes data has been disabled.

Yahoo Finance licenses data from 3rd-party providers that do not currently authorize us to redistribute these data in API form. Licenses that authorize redistribution come with a greater cost that varies depending on a number of factors, including whether the data is for personal or commercial use, the type of data, the volume of queries, and additional features which may be available.

We would appreciate your feedback to ensure that we can continue to serve your needs. By understanding your intended use of these API data, we will be better able to acquire the appropriate licenses. We appreciate your feedback, and we read every response.

They have added a query parameter call crumb that appears to act as a sort of API key.

[spudone]

I started preliminary work on grabbing quotes by simulating a web browser call. Not sure I will have time to polish this up but the initial work is here:

https://github.com/mop-tracker/mop/compare/master...spudone:mop:master

These values were retrieved by directing a web browser to:

https://finance.yahoo.com/quote/AAPL/

and using the debug console to watch what's going on. It looks like any top level yahoo page sets the required cookies. https://login.yahoo.com loads quickly without a bunch of articles.

There's an endpoint to get the crumb; you have to supply the existing cookie. They have to match when calling the quotes endpoint.

This works in my fork, but I still need to extract the cookie / crumb helpers to another file and update yahoo_market.go to use them, in addition to yahoo_quotes.go

[gsspdev]

I'm afraid that the problem cited in:

has returned. I saw bakerbrandond's message at the bottom of this thread, and confirm that mop is no longer showing market data. The familiar message:

json: cannot unmarshal object into Go value of type []map[string]interface {}

I'm having this issue as well. How could you retrieve the values from the command line?

[danteodor]

It looks like the problem is that Yahoo Finance now requires that a session cookie be provided. When I hit the URL with curl, I get 'invalid cookie'. Is it possible that Yahoo Finance has switched to requiring you first log in to establish a cookie?

[spudone]

I'll try to finish this fix - see my changes above. Cookie and crumb are required, but you don't have to log in. I haven't done enough testing to know if they rate limit these requests.

Edit: PR is up.

[brandleesee]

Merged: https://github.com/mop-tracker/mop/commit/9ba04b7c56b1d3efa3ae5c42ee9d3d0f075a5a6c Thank you, @spudone!

Can you, please, verify that mop is now functioning correctly?

Another comment/suggestion I have is that due to these frequent downtimes because of the current API used, maybe it is time to switch to something else? As maintainer, I am open to developers proposing patches with alternative, free services. I am sure this community - me included! - would be grateful.

[spudone]

I saw in some old changelists that mop used to have CNN instead of Yahoo, but the CNN handler was deleted. We should keep the separate handlers alive in code and have the main function just select a primary with fallbacks.

I know Google Finance also provides similar data. I haven't looked into retrieving it from them yet, though.

[brandleesee]

Indeed! It used to have the topbar data extracted from CNN finance website then everything was consolidated to Yahoo!

Without imposing, if you wish to try Google or any other free service, please do :)

[spudone]

Also a small thing: the copyright notices go through 2019 and should be updated. I noticed it because I added a completely new file and did a cut&paste of the notice. I figured it should be separate from this fix though.

[brandleesee]

All files are now updated with correct year! Thank you :D

[spudone]

I did a clean build off this repo and as an interactive tool it appears to be working - at least until the next time Yahoo obfuscates something.

The README has a section about fetching Yahoo data into a CSV using curl. This will no longer easily work - although if determined, you could make curl versions of the cookie and crumb calls and wrap the whole thing into a shell script.

[brandleesee]

The CSV feature is not a core functionality so I am sure we can make do without it. Could you pull a request to remove CSV functionality, please?

If in time, the feature is requested by many, then we will proceed accordingly.

[spudone]

It looks like it's just the README - it has instructions on using curl. Nothing specific to mop that I can see. I lean towards leaving it alone. No need to document every Yahoo workaround as we play whack-a-mole.

[brandleesee]

I agree. I will remove the section, however, as I foresee future problems for those who try it.

[spudone]

One more suggestion:

In cmd/mop/main.go, the refresh times are hardcoded:

timestampQueue := time.NewTicker(1 * time.Second)
quotesQueue := time.NewTicker(5 * time.Second)
marketQueue := time.NewTicker(12 * time.Second)`

I know some people want near realtime updates, but many are ok with a periodic glance. We could default to something slower, say 5 minutes, and have a command line or interactive option to change it. I think less frequent requests might draw less attention from the site supplying quotes.

[brandleesee]

To be honest, I would rather it be realtime updates. So much so, that in my personal fork (https://github.com/brandleesee/TerminalStocks/blob/master/cmd/TerminalStocks/main.go) I wrote the smallest number possible (1).

My rationale is that at any time someone glances at mop/terminalstocks one sees the latest update possible.

BUT your rationale is valid, all those requests per second may be drawing attention so I would default to 5 minutes as per your suggestion with the ability for the user to change a setting to set it to the least minimum possible (per 1 second). Regarding the setting there is the moprc/TSrc in the main folder that lets users change the stock tickers, there could be placed an option to change the update frequency.

If this is possible, I'd be game :) thank you

[spudone]

Also my fix was quick to get things going. An improvement would be to cache the cookie and crumb values and reuse them. Right now it grabs them per-quotes or market fetch. So there's 2 extra requests going out each time.

[brandleesee]

Yes, I understand it was a quick fix to get mop to run so I am very appreciative of your contribution. Please take your time to do all the necessary changes in all that we have discussed. I am grateful.

[brandleesee]

The original issue regarding the error when fetching market data was solved two weeks ago, hence closing this. PRs and new issues are welcome.