Recent helpful findings: crum manager, additional routes, etc.

roblav96 commented 7 years ago

Hey man, love what you're doing here. I've been reverse engineering Yahoo's API recently and thought I'd provide some of my findings.

Crum Manager

sstrickx/yahoofinance-api @sstrickx wrote a crum manager in Java.

Recommended Symbols

GET https://query1.finance.yahoo.com/v6/finance/recommendationsbysymbol/PLUG

Daily Gainers

GET https://query2.finance.yahoo.com/v1/finance/screener/predefined/saved?formatted=false&lang=en-US&region=US&scrIds=day_gainers&count=5&corsDomain=finance.yahoo.com

Search Autocomplete

GET https://autoc.finance.yahoo.com/autoc?query=PLUG&region=1&lang=en

Symbol Conversations

GET https://finance.yahoo.com/_finance_doubledown/api/resource/canvass.getMessageList;count=20;oauthConsumerKey=finance.oauth.client.canvass.prod.consumerKey;oauthConsumerSecret=finance.oauth.client.canvass.prod.consumerSecret;query=namespace = "yahoo_finance" and (tag = "PLUG");region=US;sortBy=createdAt;userActivity=true

Symbol News

GET http://feeds.finance.yahoo.com/rss/2.0/headline?s=PLUG&region=US&lang=en-US Returns in RSS format, couldn't figure out how to get JSON.

Multiple Symbol Quote

GET https://query2.finance.yahoo.com/v7/finance/quote?symbols=NVDA,PLUG,AMD

Multiple Symbol Historical Charts

GET https://query1.finance.yahoo.com/v7/finance/spark?symbols=PLUG,AMD&range=1d&formated=true Only returns close data though =/

Trending Symbols

GET https://query1.finance.yahoo.com/v1/finance/trending/US?lang=en-US&region=US&count=5&corsDomain=finance.yahoo.com

Live Streamer

import * as Promise from 'bluebird'
import * as Nstream from 'stream'
import * as Nhttp from 'http'
import * as Nnet from 'net'
import * as Axios from 'axios'

const YH_IDK_KEY = {
    a00: 'Ask',
    a50: 'AskSize',
    b00: 'Bid',
    b60: 'BidSize',
    c10: 'Change',
    c63: 'ChangeRealtime',
    c64: 'DisputedChangeRealtimeAfterHours',
    c85: 'ChangeRealtimeAfterHours',
    c86: 'PercentChangeRealtimeAfterHours',
    g53: 'DayLow',
    h53: 'DayHigh',
    j10: 'MarketCap',
    l84: 'PriceRealtime',
    l86: 'PriceRealtimeAfterHours',
    p20: 'PercentChange',
    p43: 'PercentChangeRealtime',
    p44: 'PercentChangeRealtimeAfterHours',
    t53: 'DisputedTimestampForCommodities',
    t54: 'DisputedTimestampForStocks',
    v53: 'Volume',
    v00: 'Volume2',
    l10: 'LastSalePrice',
    t10: 'LastSaleTime',
    l90: 'EcnQuoteLastValue',
}
const YH_KEY_IDK = {
    Ask: 'a00',
    AskSize: 'a50',
    Bid: 'b00',
    BidSize: 'b60',
    Change: 'c10',
    ChangeRealtime: 'c63',
    DisputedChangeRealtimeAfterHours: 'c64',
    ChangeRealtimeAfterHours: 'c85',
    PercentChangeRealtimeAfterHours: 'p44',
    DayLow: 'g53',
    DayHigh: 'h53',
    MarketCap: 'j10',
    PriceRealtime: 'l84',
    PriceRealtimeAfterHours: 'l86',
    PercentChange: 'p20',
    PercentChangeRealtime: 'p43',
    DisputedTimestampForCommodities: 't53',
    DisputedTimestampForStocks: 't54',
    Volume: 'v53',
    Volume2: 'v00',
    LastSalePrice: 'l10',
    LastSaleTime: 't10',
    EcnQuoteLastValue: 'l90',
}

let total = 0
let stream = null as Nhttp.IncomingMessage
function startYahooStream() {
    return Promise.resolve().then(function() {
        let keys = [
            YH_KEY_IDK.Ask,
            YH_KEY_IDK.AskSize,
            YH_KEY_IDK.Bid,
            YH_KEY_IDK.BidSize,
            YH_KEY_IDK.Volume,
            YH_KEY_IDK.Volume2,
            YH_KEY_IDK.PriceRealtime,
            YH_KEY_IDK.PriceRealtimeAfterHours,
            YH_KEY_IDK.LastSalePrice,
            YH_KEY_IDK.EcnQuoteLastValue,
            YH_KEY_IDK.LastSaleTime,
            YH_KEY_IDK.DisputedTimestampForStocks,
        ]
        return Axios.default.get('https://streamerapi.finance.yahoo.com/streamer/1.0', {
            params: {
                s: 'NVDA,PLUG,AMD',
                k: keys.join(','),
                r: 0,
                callback: 'parent.yfs_u1f',
                mktmcb: 'parent.yfs_mktmcb',
                gencallback: 'parent.yfs_gencb',
                mu: 1,
                lang: 'en-US',
                region: 'US',
                localize: 0,
            },
            responseType: 'stream',
        }).then<any>(function({ data }) {
            return new Promise<string>(function(resolve, reject) {
                stream = data as Nhttp.IncomingMessage

                stream.on('data', function(chunk) {
                    try {
                        let input = chunk.toString()
                        let jsons = [] as Array<{ [key: string]: any }>

                        let split = input.split('yfs_u1f(')
                        split.shift()
                        total = total + split.length
                        split.forEach(function(v) {
                            let i = v.indexOf(');}')
                            let parsed = v.substring(0, i).replace(/(['"])?([a-zA-Z0-9_]+)(['"])?:/g, '"$2": ')
                            if (!parsed) return;
                            jsons.push(JSON.parse(parsed))
                        })

                        let sendi = {}
                        jsons.forEach(function(s) {
                            let symbol = Object.keys(s)[0]
                            if (!sendi[symbol]) sendi[symbol] = {};
                            let v = s[symbol]
                            Object.keys(v).forEach(function(k) {
                                let n = parseFloat(v[k])
                                if (n == 0) return;
                                let key = YH_IDK_KEY[k] as string
                                sendi[symbol][key] = n
                            })
                        })
                        console.log('sendi >', JSON.stringify(sendi, null, 4))

                    } catch (error) {
                        console.error('quotes.startYahooStream > catch > error', error)
                    }
                })

                stream.on('end', resolve)
                stream.on('error', reject)
            })
        })

    }).catch(function(error) {
        console.error('quotes.startYahooStream > error', error)
        return Promise.resolve()

    }).finally(function() {
        console.warn('quotes.stream > FINALLY')
        if (stream) {
            stream.removeAllListeners()
            stream.destroy()
            stream = null
        }
        return startYahooStream()
    })
}
startYahooStream()

console.log('sendi >', JSON.stringify(sendi, null, 4))

sendi >
{
    NVDA: {
        LastSalePrice: 149.6,
        Volume2: 92,
        Ask: 149.5,
        AskSize: 1800,
        Bid: 148.2,
        BidSize: 100,
        PriceRealtime: 149.6
    }
}

Hope you find this helpful!

pilwon commented 7 years ago

@roblav96 Thanks for the excellent info. It might help @gadicc with his PR #42!

36bits commented 6 years ago

@roblav96 Just wanted to second the thanks for this info. The Yahoo stock quote CSV API at http://download.finance.yahoo.com/d/quotes.csv?s=FOO,BAR has been giving a '999 Request Denied' error since yesterday. The 'Multiple Symbol Quote' URL above was invaluable in getting my quotes going again.