Closed shayandavoodii closed 1 year ago
Hey @shayandavoodii,
Thanks for noticing, suggesting a way of implementing it, and implementing it all together! The initial idea was to see if I can not implement a search function that accesses Yahoo where one can run a generic search for company names, and the API then returns a ticker and some general information. What you have provided is super helpful, though, and I will merge it with the master branch. I might change a few things before releasing a new YFinance version. It might be good to return the company names the API provides on top of the ticker. The only thing I am concerned about is that Yahoo Finance's internal tickers sometimes are a bit different – this is mostly true for stocks outside of North America. So I think what you have implemented should work great for the currently supported markets.
Thanks again for all your help/work. Very much appreciated!
@eohne Thank you. I'm glad that you've found it helpful. I would be pleased to help further.
The only thing I am concerned about is that Yahoo Finance's internal tickers sometimes are a bit different – this is mostly true for stocks outside of North America.
Yes, I thought about it. Unfortunately, I don't know whether the API's maintainer keeps the content updated and dynamic. It would be a relief if we could figure it out. Anyway, for now it's not bad, and we can rely on it. But it's better to keep an eye on the API to check whether it changes the content.
It might be good to return the company names the API provides on top of the ticker.
Through this API, we can fetch the company name by getting the value of the name
key. However, a suitable data structure should be chosen since a good tradeoff between readability and performance is promising. For example, one idea can be returning a DataFrame
with two columns: 1. Symbol 2. Company. But, it comes with an additional dependency. Another idea can be returning a Vector{Tuples{String, String}}
that holds all pairs in a Vector
; in this case, using a NamedTuple
might be a better choice since it can come with better functionality due to the ability to get a specific element by calling a key:
julia> Symbols = (;AA="Alcoa Corporation")
(AA = "Alcoa Corporation",)
julia> Symbols.AA
"Alcoa Corporation"
But if we have a large number of pairs, creating a NamedTuple
(or even a Tuple
) may not be reasonable considering the performance. However, we should consider that get_symbols
is not part of an iterative procedure! So, the performance won't be a considerable challenge here.
@shayandavoodii Agreed that we should keep an eye on the API.
Don't really want to depend on DataFrames
if we dont have to.
I think me might have to go with a struct that allows users to just retrieve some sort of vector of tickers in the end so people can just broadcast get_prices
over the symbols.
I dont know whehter it is overkill to go with a custom struct where we can define simple accessor functions...
I don't think we have to worry about performance too much on this one. I think people might just call this once and that is it.
I think me might have to go with a struct that allows users to just retrieve some sort of vector of tickers in the end so people can just broadcast get_prices over the symbols.
Yes, why not. Currently, the following combo is available:
julia> tickers = get_symbols("Nyse")[1:3]
3-element Vector{String}:
"A"
"AA"
"AAC"
julia> using DataFrames
julia> data = get_prices.(tickers, range="1d", interval="90m");
julia> vcat([DataFrame(i) for i in data]...)
17×7 DataFrame
Row │ ticker timestamp open high low close vol
│ String DateTime Float64 Float64 Float64 Float64 Float64
─────┼─────────────────────────────────────────────────────────────────────────────
1 │ A 2023-03-27T13:30:00 133.58 134.62 132.88 133.11 334697.0
2 │ A 2023-03-27T15:00:00 133.115 133.41 132.73 133.23 105426.0
3 │ A 2023-03-27T16:30:00 133.215 133.69 133.11 133.43 108107.0
4 │ A 2023-03-27T18:00:00 133.43 133.97 133.15 133.58 161012.0
5 │ A 2023-03-27T19:30:00 133.575 133.69 132.9 133.11 240372.0
6 │ A 2023-03-27T20:00:00 133.07 133.07 133.07 133.07 0.0
7 │ AA 2023-03-27T13:30:00 40.51 40.77 39.775 39.85 789134.0
8 │ AA 2023-03-27T15:00:00 39.865 40.325 39.5734 40.26 703436.0
9 │ AA 2023-03-27T16:30:00 40.27 40.29 39.875 40.1 635980.0
10 │ AA 2023-03-27T18:00:00 40.0968 40.59 40.04 40.32 636301.0
11 │ AA 2023-03-27T19:30:00 40.305 40.33 40.02 40.14 586977.0
12 │ AA 2023-03-27T20:00:00 40.12 40.12 40.12 40.12 0.0
13 │ AAC 2023-03-27T13:30:00 10.33 10.33 10.3213 10.325 5873.0
14 │ AAC 2023-03-27T15:00:00 10.325 10.325 10.325 10.325 402.0
15 │ AAC 2023-03-27T16:30:00 10.325 10.34 10.325 10.335 13172.0
16 │ AAC 2023-03-27T18:00:00 10.335 10.34 10.32 10.32 12784.0
17 │ AAC 2023-03-27T19:30:00 10.33 10.33 10.325 10.325 2438.0
However, another approach could be encapsulating it in a struct
. It's up to your taste and the framework that you have in your mind.
Yes now it is. Although if we also return the company names it may not - depending on the way we go about returning the name + symbol combo.
I dont know whehter it is overkill to go with a custom struct where we can define simple accessor functions...
Before I made my PR, I checked your API by probing the /src
contents; I didn't confront any structs
within your framework. It's okay, but if you want to bring struct
s into the game, it might be better to use them comprehensively rather than using them just for a specific function. Then, you can use them to adjust the created function from scratch to adapt them with defined struct
s! Henceforth, we can define further functions for specific composite types.
After all, it depends on your personal taste and desired API. Utilizing struct
s won't be overkill at all.
@shayandavoodii Yes I don't use any struct
s so far because I didnt really see a good reason to use them.
When I ment overkill I was mostly refering to using a struct
only for the get_symbol output which would have been weird because the package does not use them elsewhere - as you pointed out yourself.
Hi @shayandavoodii I did some digging, and it turns out I may have found an add-on provided by Yahoo Finance directly.
Your function is super useful if someone wants to get all tickers for a certain market.
What I had initially envisioned for that empty file was a search function where one can look up tickers for companies (one may not know the ticker for the company they want information for). This is why I thought it would be useful to return company names as well.
I have written a quick sample function where this can be achieved below.
Thus I think we can leave the get_symbols
function as is and add a separate search function as defined below:
function get_search(search_term; search_country = "United States", n_results = 10)
yfinance_search_link = "https://query2.finance.yahoo.com/v1/finance/search"
query = Dict("q" => search_term, "quote_count" => "$n_results", "country" => search_country)
response = HTTP.get(yfinance_search_link,query = query)
repsonse_parsed = JSON3.read(response.body)
quotes = repsonse_parsed.quotes
# Also provides news under response_parsed.news
return quotes
end
Example output showing the available info below. Although will need to make the output nicer...
julia> get_search("advanced micro", n_results = 5)
7-element JSON3.Array{JSON3.Object, Vector{UInt8}, SubArray{UInt64, 1, Vector{UInt64}, Tuple{UnitRange{Int64}}, true}}:
{
"exchange": "NMS",
"shortname": "Advanced Micro Devices, Inc.",
"quoteType": "EQUITY",
"symbol": "AMD",
"index": "quotes",
"score": 195897,
"typeDisp": "Equity",
"longname": "Advanced Micro Devices, Inc.",
"exchDisp": "NASDAQ",
"sector": "Technology",
"industry": "Semiconductors",
"dispSecIndFlag": true,
"isYahooFinance": true
}
{
"exchange": "SHH",
"shortname": "ADVANCED MICRO-FABRICATION EQUI",
"quoteType": "EQUITY",
"symbol": "688012.SS",
"index": "quotes",
"score": 20054,
"typeDisp": "Equity",
"longname": "Advanced Micro-Fabrication Equipment Inc. China",
"exchDisp": "Shanghai",
"sector": "Technology",
"industry": "Semiconductor Equipment & Materials",
"isYahooFinance": true
}
{
"exchange": "GER",
"shortname": "ADVANCED MICRO DEVICES INC",
"quoteType": "EQUITY",
"symbol": "AMD.DE",
"index": "quotes",
"score": 20044,
"typeDisp": "Equity",
"longname": "Advanced Micro Devices, Inc.",
"exchDisp": "XETRA",
"sector": "Technology",
"industry": "Semiconductors",
"isYahooFinance": true
}
{
"exchange": "NEO",
"shortname": "ADVANCED MICRO DEVICES CDR (CAD",
"quoteType": "EQUITY",
"symbol": "AMD.NE",
"index": "quotes",
"score": 20029,
"typeDisp": "Equity",
"longname": "Advanced Micro Devices, Inc.",
"exchDisp": "NEO",
"sector": "Technology",
"industry": "Semiconductors",
"isYahooFinance": true
}
{
"exchange": "SAO",
"shortname": "ADVANCED MICDRN",
"quoteType": "EQUITY",
"symbol": "A1MD34.SA",
"index": "quotes",
"score": 20015,
"typeDisp": "Equity",
"longname": "Advanced Micro Devices, Inc.",
"exchDisp": "São Paulo",
"sector": "Technology",
"industry": "Semiconductors",
"isYahooFinance": true
}
{
"exchange": "MEX",
"shortname": "ADVANCED MICRO DEVICES INC",
"quoteType": "EQUITY",
"symbol": "AMD.MX",
"index": "quotes",
"score": 20009,
"typeDisp": "Equity",
"longname": "Advanced Micro Devices, Inc.",
"exchDisp": "Mexico",
"sector": "Technology",
"industry": "Semiconductors",
"isYahooFinance": true
}
{
"exchange": "LSE",
"shortname": "LEVERAGE SHARES PUBLIC LIMITED ",
"quoteType": "EQUITY",
"symbol": "AMD2.L",
"index": "quotes",
"score": 20005,
"typeDisp": "Equity",
"longname": "Leverage Shares 2x Advanced Micro Devices ETC",
"exchDisp": "London",
"isYahooFinance": true
}
Hi @eohne !
I did some digging, and it turns out I may have found an add-on provided by Yahoo Finance directly.
You know what? It's great! Because the data is provided by Yahoo Finance itself, which we fetch price data from.
If we find a better API (especially one that originated from Yahoo Finance), it would be better to replace it with the one in the get_symbols
.
Your function is super useful if someone wants to get all tickers for a certain market.
I'm glad to hear this! I hope I can help you further!
Although will need to make the output nicer...
In this case, if we've used struct
, we would be able to define a specific Base.show()
function for the specifically defined type. Hence, every time that a user runs the function, Julia calls the most appropriate Base.show()
function (definitely the one that matches the defined type) automatically.
After all, personally, I like the output. It's super clear. But, maybe we can design something like a table for it without using Tables.jl
! It's a good challenge and I would like to try it.
Hi @shayandavoodii
I have now implemented the search functionality as is with the JSON output.
I renamed your function get_all_symbols
and called the ticker search function get_symbols
.
I have also added a section in the docs for these functions.
I plan to implement a function that returns news articles soon and then release v0.1.4. As the current master branch is quite a bit more advanced compared to the current release (for a summary)
Hi!
Nice changes!
I'll keep in mind to look for a way to fetch the symbols from yahoo finance itself rather than a foreign API. I hope we can improve get_all_symbols
function.
@shayandavoodii Hi, I am rethinking the use of structs here. I have locally implemented a news search function and wasn't particularly happy with how it was printed by default. Default:
julia> search_news_old("AAPL")
8-element JSON3.Array{JSON3.Object, Vector{UInt8}, SubArray{UInt64, 1, Vector{UInt64}, Tuple{UnitRange{Int64}}, true}}:
{
"uuid": "6ab8b7e8-a53b-3813-8009-c06c4b83801d",
"title": "How the great home working experiment fell apart",
"publisher": "The Telegraph",
"link": "https://finance.yahoo.com/news/great-home-working-experiment-fell-080000785.html",
"providerPublishTime": 1681200000,
"type": "STORY",
"thumbnail": {
"resolutions": [
{
"url": "https://s.yimg.com/uu/api/res/1.2/h_21DDKm0lkX_yIOLnOOvw--~B/aD01MzY7dz04NTg7YXBwaWQ9eXRhY2h5b24-/https://media.zenfs.com/en/the_telegraph_258/93bd07ed17be85acb0830599366272c6",
"width": 858,
"height": 536,
"tag": "original"
},
{
"url": "https://s.yimg.com/uu/api/res/1.2/NhQLTJTgbJbBw9Lvzslu5Q--~B/Zmk9ZmlsbDtoPTE0MDtweW9mZj0wO3c9MTQwO2FwcGlkPXl0YWNoeW9u/https://media.zenfs.com/en/the_telegraph_258/93bd07ed17be85acb0830599366272c6",
"width": 140,
"height": 140,
"tag": "140x140"
}
]
},
"relatedTickers": [
"META",
"AAPL"
]
}
{
"uuid": "8c88807a-d56c-3b99-b42a-aa0db899af32",
"title": "Cook to Open First Apple Stores in India in Pivot Beyond China",
"publisher": "Bloomberg",
"link": "https://finance.yahoo.com/news/cook-open-first-apple-stores-052324685.html",
"providerPublishTime": 1681190604,
"type": "STORY",
"thumbnail": {
"resolutions": [
{
"url": "https://s.yimg.com/uu/api/res/1.2/nG.861jB.k90Bw410zjWZA--~B/aD0xMzMzO3c9MjAwMDthcHBpZD15dGFjaHlvbg--/https://media.zenfs.com/en/bloomberg_technology_68/a834419f9170db643a7fbbab95b73453",
"width": 2000,
"height": 1333,
"tag": "original"
},
{
"url": "https://s.yimg.com/uu/api/res/1.2/_ERnoCTZ6H3oyHnLZHWSGg--~B/Zmk9ZmlsbDtoPTE0MDtweW9mZj0wO3c9MTQwO2FwcGlkPXl0YWNoeW9u/https://media.zenfs.com/en/bloomberg_technology_68/a834419f9170db643a7fbbab95b73453",
"width": 140,
"height": 140,
"tag": "140x140"
}
]
},
"relatedTickers": [
"AAPL"
]
}
{
"uuid": "b7b2c280-59fc-31dd-9c14-41964fa1b947",
"title": "UPDATE 2-Apple says issues with Music app resolved, other services back up",
"publisher": "Reuters",
"link": "https://finance.yahoo.com/news/1-apple-music-down-thousands-023205664.html",
"providerPublishTime": 1681180325,
"type": "STORY",
"relatedTickers": [
"AAPL"
]
}
{
"uuid": "a0f4fb4d-6010-32ea-a2dd-720f90f60df6",
"title": "Apple says issues with Music app resolved, other services back up",
"publisher": "Reuters",
"link": "https://finance.yahoo.com/news/several-apple-services-down-users-012917927.html",
"providerPublishTime": 1681176557,
"type": "STORY",
"thumbnail": {
"resolutions": [
{
"url": "https://s.yimg.com/uu/api/res/1.2/Bz6_jk6USJe_XvFiKXWAsQ--~B/aD01MzQ7dz04MDA7YXBwaWQ9eXRhY2h5b24-/https://media.zenfs.com/en/reuters-finance.com/b6be1a488d116fbf651d3055e4448ebd",
"width": 800,
"height": 534,
"tag": "original"
},
{
"url": "https://s.yimg.com/uu/api/res/1.2/duSNz9lKRVSkQj68MvGMSQ--~B/Zmk9ZmlsbDtoPTE0MDtweW9mZj0wO3c9MTQwO2FwcGlkPXl0YWNoeW9u/https://media.zenfs.com/en/reuters-finance.com/b6be1a488d116fbf651d3055e4448ebd",
"width": 140,
"height": 140,
"tag": "140x140"
}
]
},
"relatedTickers": [
"AAPL"
]
}
{
"uuid": "4eb8b993-5b74-3095-ad39-3432ac9bee4b",
"title": "How to delete TikTok. YF explains",
"publisher": "Yahoo Finance Video",
"link": "https://finance.yahoo.com/video/delete-tiktok-yf-explains-005943947.html",
"providerPublishTime": 1681174783,
"type": "VIDEO",
"thumbnail": {
"resolutions": [
{
"url": "https://s.yimg.com/uu/api/res/1.2/fYFdea6VYB9vsZ.xOC2Ubw--~B/aD0xMDgwO3c9MTkyMDthcHBpZD15dGFjaHlvbg--/https://s.yimg.com/os/creatr-uploaded-images/2023-04/14e0ff10-d801-11ed-bffd-d95e97321761",
"width": 1920,
"height": 1080,
"tag": "original"
},
{
"url": "https://s.yimg.com/uu/api/res/1.2/oKxYuzspm9VKH3qvNA9xPA--~B/Zmk9ZmlsbDtoPTE0MDtweW9mZj0wO3c9MTQwO2FwcGlkPXl0YWNoeW9u/https://s.yimg.com/os/creatr-uploaded-images/2023-04/14e0ff10-d801-11ed-bffd-d95e97321761",
"width": 140,
"height": 140,
"tag": "140x140"
}
]
},
"relatedTickers": [
"AAPL",
"SNAP",
"META",
"GOOG",
"GOOGL"
]
}
{
"uuid": "6123bfb6-3092-33d2-8162-77e48ab19d17",
"title": "Katie Cotton, Who Led Apple’s Media Strategy for 18 Years, Dies",
"publisher": "Bloomberg",
"link": "https://finance.yahoo.com/news/katie-cotton-led-apple-media-001601799.html",
"providerPublishTime": 1681172161,
"type": "STORY",
"thumbnail": {
"resolutions": [
{
"url": "https://s.yimg.com/uu/api/res/1.2/sma1WEaKiWFKZaWnKgd5jg--~B/aD0yMDMxO3c9MzAwMDthcHBpZD15dGFjaHlvbg--/https://media.zenfs.com/en/bloomberg_markets_842/94e70353ece3cc05076b5f8f665ea159",
"width": 3000,
"height": 2031,
"tag": "original"
},
{
"url": "https://s.yimg.com/uu/api/res/1.2/ZXDrpIcvJhk14Q2IrNB66g--~B/Zmk9ZmlsbDtoPTE0MDtweW9mZj0wO3c9MTQwO2FwcGlkPXl0YWNoeW9u/https://media.zenfs.com/en/bloomberg_markets_842/94e70353ece3cc05076b5f8f665ea159",
"width": 140,
"height": 140,
"tag": "140x140"
}
]
},
"relatedTickers": [
"AAPL"
]
}
{
"uuid": "f827b6a2-950e-3338-839c-ce29d7cf5f35",
"title": "Apple computer shipments fell 40% from a year ago, report finds",
"publisher": "Fox Business",
"link": "https://finance.yahoo.com/news/apple-computer-shipments-fell-40-000018509.html",
"providerPublishTime": 1681171218,
"type": "STORY",
"thumbnail": {
"resolutions": [
{
"url": "https://s.yimg.com/uu/api/res/1.2/y.NXR_DJWSHa1PCEB0eqKw--~B/aD03MjA7dz0xMjgwO2FwcGlkPXl0YWNoeW9u/https://media.zenfs.com/en/fox_business_text_367/167952adbe743af6610184aadf41e0a3",
"width": 1280,
"height": 720,
"tag": "original"
},
{
"url": "https://s.yimg.com/uu/api/res/1.2/oQH2lKW2vqD12FwXy160nw--~B/Zmk9ZmlsbDtoPTE0MDtweW9mZj0wO3c9MTQwO2FwcGlkPXl0YWNoeW9u/https://media.zenfs.com/en/fox_business_text_367/167952adbe743af6610184aadf41e0a3",
"width": 140,
"height": 140,
"tag": "140x140"
}
]
},
"relatedTickers": [
"AAPL"
]
}
{
"uuid": "c8beee81-eed5-3585-bee2-d3eb8d898f31",
"title": "Japanese stocks surge on report of Warren Buffett interest",
"publisher": "Investing.com",
"link": "https://finance.yahoo.com/news/japanese-stocks-surge-report-warren-225806673.html",
"providerPublishTime": 1681167486,
"type": "STORY",
"relatedTickers": [
"AAPL"
]
}
Custom struct printing:
julia> search_news("AAPL")
8-element YahooNews{NewsItem, 1}:
Title: How the great home working experiment fell apart
Timestamp: Apr 11 08:00 AM
Publisher: The Telegraph
Link: https://finance.yahoo.com/news/great-home-working-experiment-fell-080000785.html
Symbols: META, AAPL
Title: Cook to Open First Apple Stores in India in Pivot Beyond China
Timestamp: Apr 11 05:23 AM
Publisher: Bloomberg
Link: https://finance.yahoo.com/news/cook-open-first-apple-stores-052324685.html
Symbols: AAPL
Title: UPDATE 2-Apple says issues with Music app resolved, other services back up
Timestamp: Apr 11 02:32 AM
Publisher: Reuters
Link: https://finance.yahoo.com/news/1-apple-music-down-thousands-023205664.html
Symbols: AAPL
Title: Apple says issues with Music app resolved, other services back up
Timestamp: Apr 11 01:29 AM
Publisher: Reuters
Link: https://finance.yahoo.com/news/several-apple-services-down-users-012917927.html
Symbols: AAPL
Title: How to delete TikTok. YF explains
Timestamp: Apr 11 00:59 AM
Publisher: Yahoo Finance Video
Link: https://finance.yahoo.com/video/delete-tiktok-yf-explains-005943947.html
Symbols: AAPL, SNAP, META, GOOG, GOOGL
Title: Katie Cotton, Who Led Apple’s Media Strategy for 18 Years, Dies
Timestamp: Apr 11 00:16 AM
Publisher: Bloomberg
Link: https://finance.yahoo.com/news/katie-cotton-led-apple-media-001601799.html
Symbols: AAPL
Title: Apple computer shipments fell 40% from a year ago, report finds
Timestamp: Apr 11 00:00 AM
Publisher: Fox Business
Link: https://finance.yahoo.com/news/apple-computer-shipments-fell-40-000018509.html
Symbols: AAPL
Title: Japanese stocks surge on report of Warren Buffett interest
Timestamp: Apr 10 22:58 PM
Publisher: Investing.com
Link: https://finance.yahoo.com/news/japanese-stocks-surge-report-warren-225806673.html
Symbols: AAPL
Now that I am thinking of implementing a custom struct for this part of the package anyway, I also thought about implementing a struct for the ticker search. The new printing would look like this:
julia> get_symbols("Micro")
7-element YahooSearch{YahooSearchItem, 1}:
Symbol: MGC=F
Name: Micro Gold Futures,Jun-2023
Type: FUTURE
Exch.: New York Commodity Exchange (CMX)
Symbol: MU
Name: Micron Technology, Inc.
Type: EQUITY
Exch.: NASDAQ (NMS)
Sec.: Technology
Ind.: Semiconductors
Symbol: MSFT
Name: Microsoft Corporation
Type: EQUITY
Exch.: NASDAQ (NMS)
Sec.: Technology
Ind.: Software—Infrastructure
Symbol: AMD
Name: Advanced Micro Devices, Inc.
Type: EQUITY
Exch.: NASDAQ (NMS)
Sec.: Technology
Ind.: Semiconductors
Symbol: MSTR
Name: MicroStrategy Incorporated
Type: EQUITY
Exch.: NASDAQ (NMS)
Sec.: Technology
Ind.: Software—Application
Symbol: MES=F
Name: MICRO E-MINI S&P 500 INDEX FUTU
Type: FUTURE
Exch.: Chicago Mercantile Exchange (CME)
Symbol: MNQ=F
Name: Micro E-mini Nasdaq-100 Index F
Type: FUTURE
Exch.: Chicago Mercantile Exchange (CME)
Which one do you prefer?
Hey! First, I see the outputs pretty neat on the GitHub Android client.
Custom struct printing:
The custom print is pretty neat and promising! Good job! I was thinking about embedding them in a customized table, but what you've proposed is already readable, neat, and informative. I don't know how it can be any better 🤙🏻
Thanks for asking! This means the world to me. 🔥
Registering v0.1.4 with this implemented.
Happy for the contribution! 🥳
Hey, I see you've provided a
Search_Symbol.jl
file for fetching/reading the symbols (tickers). I know it's empty, but I think it's possible to fill this file, and provide a function to catch the Symbols of several markets. But first, I want to know what your desired workflow is for it. Do you want to fetch the data from an API, or load an existing.csv
file that contains the symbol names?I guess you want to keep dependencies at the lowest level possible -> Reading from a CSV file isn't a good option. But, I would be happy if you explicitly share your desired workflow.