search

gadicc / node-yahoo-finance2

Unofficial API for Yahoo Finance
https://www.npmjs.com/package/yahoo-finance2
MIT License
385 stars 63 forks source link

Other modules: tracking progress #8

Open gadicc opened 3 years ago

gadicc commented 3 years ago

Thanks to the awesome @roblav96 for all this beautiful work back over at https://github.com/pilwon/node-yahoo-finance/issues/43 (in 2017 :sweat_smile:) Have added status [X]'s on each point to track progress, followed by any additional comments below.

Hey man, love what you're doing here. I've been reverse engineering Yahoo's API recently and thought I'd provide some of my findings.

[n/a] Crum Manager

sstrickx/yahoofinance-api @sstrickx wrote a crum manager in Java.

As reported by @smeijer in https://github.com/pilwon/node-yahoo-finance/issues/73 (thanks again!) and confirmed by me (and @MichaelDeBoey?), this is no longer required.

[X] Recommended Symbols

GET https://query1.finance.yahoo.com/v6/finance/recommendationsbysymbol/PLUG

Thanks, @pudgereyem, who added on 2021-02-09 in #28.

[ ] Daily Gainers

GET https://query2.finance.yahoo.com/v1/finance/screener/predefined/saved?formatted=false&lang=en-US&region=US&scrIds=day_gainers&count=5&corsDomain=finance.yahoo.com

[X] Search Autocomplete

GET https://autoc.finance.yahoo.com/autoc?query=PLUG&region=1&lang=en

[ ] Symbol Conversations

GET https://finance.yahoo.com/_finance_doubledown/api/resource/canvass.getMessageList;count=20;oauthConsumerKey=finance.oauth.client.canvass.prod.consumerKey;oauthConsumerSecret=finance.oauth.client.canvass.prod.consumerSecret;query=namespace = "yahoo_finance" and (tag = "PLUG");region=US;sortBy=createdAt;userActivity=true

[ ] Symbol News

GET http://feeds.finance.yahoo.com/rss/2.0/headline?s=PLUG&region=US&lang=en-US Returns in RSS format, couldn't figure out how to get JSON.

[X] Multiple Symbol Quote

GET https://query2.finance.yahoo.com/v7/finance/quote?symbols=NVDA,PLUG,AMD

[ ] Multiple Symbol Historical Charts

GET https://query1.finance.yahoo.com/v7/finance/spark?symbols=PLUG,AMD&range=1d&formated=true Only returns close data though =/

[X] Trending Symbols

GET https://query1.finance.yahoo.com/v1/finance/trending/US?lang=en-US&region=US&count=5&corsDomain=finance.yahoo.com

Thanks, @PythonCreator27, who added this on 2021-02-18 in #66.

[ ] Symbol Options

GET https://query1.finance.yahoo.com/v7/finance/options/[symbol]

Found by @PythonCreator27, more details below in https://github.com/gadicc/node-yahoo-finance2/issues/8#issuecomment-802463467.

Live Streamer

import * as Promise from 'bluebird'
import * as Nstream from 'stream'
import * as Nhttp from 'http'
import * as Nnet from 'net'
import * as Axios from 'axios'

const YH_IDK_KEY = {
  a00: 'Ask',
  a50: 'AskSize',
  b00: 'Bid',
  b60: 'BidSize',
  c10: 'Change',
  c63: 'ChangeRealtime',
  c64: 'DisputedChangeRealtimeAfterHours',
  c85: 'ChangeRealtimeAfterHours',
  c86: 'PercentChangeRealtimeAfterHours',
  g53: 'DayLow',
  h53: 'DayHigh',
  j10: 'MarketCap',
  l84: 'PriceRealtime',
  l86: 'PriceRealtimeAfterHours',
  p20: 'PercentChange',
  p43: 'PercentChangeRealtime',
  p44: 'PercentChangeRealtimeAfterHours',
  t53: 'DisputedTimestampForCommodities',
  t54: 'DisputedTimestampForStocks',
  v53: 'Volume',
  v00: 'Volume2',
  l10: 'LastSalePrice',
  t10: 'LastSaleTime',
  l90: 'EcnQuoteLastValue',
}
const YH_KEY_IDK = {
  Ask: 'a00',
  AskSize: 'a50',
  Bid: 'b00',
  BidSize: 'b60',
  Change: 'c10',
  ChangeRealtime: 'c63',
  DisputedChangeRealtimeAfterHours: 'c64',
  ChangeRealtimeAfterHours: 'c85',
  PercentChangeRealtimeAfterHours: 'p44',
  DayLow: 'g53',
  DayHigh: 'h53',
  MarketCap: 'j10',
  PriceRealtime: 'l84',
  PriceRealtimeAfterHours: 'l86',
  PercentChange: 'p20',
  PercentChangeRealtime: 'p43',
  DisputedTimestampForCommodities: 't53',
  DisputedTimestampForStocks: 't54',
  Volume: 'v53',
  Volume2: 'v00',
  LastSalePrice: 'l10',
  LastSaleTime: 't10',
  EcnQuoteLastValue: 'l90',
}

let total = 0
let stream = null as Nhttp.IncomingMessage
function startYahooStream() {
  return Promise.resolve().then(function() {
      let keys = [
          YH_KEY_IDK.Ask,
          YH_KEY_IDK.AskSize,
          YH_KEY_IDK.Bid,
          YH_KEY_IDK.BidSize,
          YH_KEY_IDK.Volume,
          YH_KEY_IDK.Volume2,
          YH_KEY_IDK.PriceRealtime,
          YH_KEY_IDK.PriceRealtimeAfterHours,
          YH_KEY_IDK.LastSalePrice,
          YH_KEY_IDK.EcnQuoteLastValue,
          YH_KEY_IDK.LastSaleTime,
          YH_KEY_IDK.DisputedTimestampForStocks,
      ]
      return Axios.default.get('https://streamerapi.finance.yahoo.com/streamer/1.0', {
          params: {
              s: 'NVDA,PLUG,AMD',
              k: keys.join(','),
              r: 0,
              callback: 'parent.yfs_u1f',
              mktmcb: 'parent.yfs_mktmcb',
              gencallback: 'parent.yfs_gencb',
              mu: 1,
              lang: 'en-US',
              region: 'US',
              localize: 0,
          },
          responseType: 'stream',
      }).then<any>(function({ data }) {
          return new Promise<string>(function(resolve, reject) {
              stream = data as Nhttp.IncomingMessage

              stream.on('data', function(chunk) {
                  try {
                      let input = chunk.toString()
                      let jsons = [] as Array<{ [key: string]: any }>

                      let split = input.split('yfs_u1f(')
                      split.shift()
                      total = total + split.length
                      split.forEach(function(v) {
                          let i = v.indexOf(');}')
                          let parsed = v.substring(0, i).replace(/(['"])?([a-zA-Z0-9_]+)(['"])?:/g, '"$2": ')
                          if (!parsed) return;
                          jsons.push(JSON.parse(parsed))
                      })

                      let sendi = {}
                      jsons.forEach(function(s) {
                          let symbol = Object.keys(s)[0]
                          if (!sendi[symbol]) sendi[symbol] = {};
                          let v = s[symbol]
                          Object.keys(v).forEach(function(k) {
                              let n = parseFloat(v[k])
                              if (n == 0) return;
                              let key = YH_IDK_KEY[k] as string
                              sendi[symbol][key] = n
                          })
                      })
                      console.log('sendi >', JSON.stringify(sendi, null, 4))

                  } catch (error) {
                      console.error('quotes.startYahooStream > catch > error', error)
                  }
              })

              stream.on('end', resolve)
              stream.on('error', reject)
          })
      })

  }).catch(function(error) {
      console.error('quotes.startYahooStream > error', error)
      return Promise.resolve()

  }).finally(function() {
      console.warn('quotes.stream > FINALLY')
      if (stream) {
          stream.removeAllListeners()
          stream.destroy()
          stream = null
      }
      return startYahooStream()
  })
}
startYahooStream()

console.log('sendi >', JSON.stringify(sendi, null, 4))

sendi >
{
    NVDA: {
        LastSalePrice: 149.6,
        Volume2: 92,
        Ask: 149.5,
        AskSize: 1800,
        Bid: 148.2,
        BidSize: 100,
        PriceRealtime: 149.6
    }
}

Hope you find this helpful!

gadicc commented 3 years ago

Added quote module in https://github.com/gadicc/node-yahoo-finance2/compare/e0f8d35f857826d394c6e3e811fc054e53540533...1cbc17a1741392cd29e82f2a9763bbf28007840c, released in v1.6.0.

pudgereyem commented 3 years ago

Created a PR that adds a new module for recommendationsBySymbol here; https://github.com/gadicc/node-yahoo-finance2/pull/28

pudgereyem commented 3 years ago

New module recommendationsBySymbol was added in with https://github.com/gadicc/node-yahoo-finance2/pull/28.

@gadicc, would you mind updating the Issue to reflect this since I can't do it myself? 🙏

gadicc commented 3 years ago

Thanks, @pudgereyem! Have updated the checklist. Thanks again for all your awesome work on this.

advaiyalad commented 3 years ago

@gadicc Found another endpoint: https://query1.finance.yahoo.com/v7/finance/options/[symbol]. Not sure if this was found already.

Query Params

Param Name Param Type Required
formatted Boolean false
crumb String false
lang String false
region String false

corsDomain omitted as it does nothing.

Another finding: query1, query2, and iquery are the same. We can fetch from query1.finance.yahoo.com OR query2.finance.yahoo.com OR iquery.finance.yahoo.com for the same data. Probably want to unify what we use in our code, or allow users to choose from each domain.

Hope this helps!

gadicc commented 3 years ago

Hey, that's awesome, thanks @PythonCreator27! Have added it to the list in the first post.

And for noticing about the alternative host names, we should indeed accommodate that.

roblav96 commented 3 years ago

@gadicc Appreciate the creds friend =]

Take a peek at https://github.com/roblav96/robinhood.tools it has some additional sources of financial data n more.

roblav96 commented 3 years ago

More specifically:

For example:

curl 'https://quoteapi.webull.com/api/quote/tickerRealTimes/full?tickerIds=925334567,925353501,913255891'
gadicc commented 3 years ago

Hey @roblav96, great to see you're still involved with all this stuff, and to see you on this new repo!

Thanks for sharing the above. I recall on the original repo too people were looking for additional sources, and I guess we should add this to the README or an FAQ or something, so thank you!

What is your rough breakdown of consumption from Yahoo vs Webull now? Just out of curiosity.

roblav96 commented 3 years ago

@gadicc Webull provides the best scrapable market data, but they make you work for it. lol Start by exploring the network traffic of https://app.webull.com/ using your web browser's network debug tools.

advaiyalad commented 3 years ago

Another note: recommendationsBySymbol is using an older version of the endpoint. The newer version is: https://query1.finance.yahoo.com/v7/finance/recommendationsbysymbol/PLUG. Yahoo's website does seem to use the older version, though. Changing the endpoint, though, would be a definite breaking change. @gadicc Do you think that it is worth it, or should we just create a brand new module for the newer version of the endpoint? The newer endpoint seems to just return a Quote[] instead of its own response format.

gadicc commented 3 years ago

Well found, @PythonCreator27. I would just leave it for now, from what you've said (and thanks for checking this all out!), there doesn't seem to be a compelling reason to upgrade. If something happens to the v6 one, maybe we could switch to v7 but return the results in the same format if that's the only difference (and our validation checks will let us know if we missed anything).

It is something to think about though for all modules. Maybe in future we should name the modules with Yahoo's API version, e.g. recommendationsBySymbolV7(). That would make it easier to cover these kinds of cases without needing to publish a new major version of the library just for one end point. Is a bit verbose though.

In short, I suggest to leave for now and see what other cases come up in the future and then make a call. Thanks for bringing attention to this.

gadicc commented 3 years ago

@gadicc Webull provides the best scrapable market data, but they make you work for it. lol Start by exploring the network traffic of https://app.webull.com/ using your web browser's network debug tools.

Thanks, @roblav96! I've made a start of referencing this (a long with some of the older suggestsions from node-yahoo-finance (v1) issues) on the Wiki: https://github.com/gadicc/node-yahoo-finance2/wiki/Alternative-Sources

I invite anyone to help improve this page with anything useful.

advaiyalad commented 3 years ago

Another finding (endpoint): https://query2.finance.yahoo.com/ws/insights/v2/finance/insights?lang=en-US&region=US&symbol=AMD&getAllResearchReports=true&reportsCount=2

Insights - seems interesting. I was looking for answers to #138, and found this along the way... Will send a PR soon to add this.