search

joshuaulrich / TTR

Technical analysis and other functions to construct technical trading rules with R
GNU General Public License v2.0
330 stars 103 forks source link

stockSymbols function is no longer working #104

Closed ghost closed 4 years ago

ghost commented 4 years ago

the function

stockSymbols()

is no longer working. Here is the minimal example:

stockSymbols("NYSE")

Then I get this error:

Fetching NYSE symbols...
Error in file(file, "rt") : 
  cannot open the connection to 'https://www.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=NYSE&render=download'
In addition: Warning message:
In file(file, "rt") :
  cannot open URL 'https://www.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=NYSE&render=download': HTTP status was '403 Forbidden'
joshuaulrich commented 4 years ago

Thanks for using TTR and taking time to make a report. I'm closing this because it is a duplicate of #97 and #98. It has been fixed and is on CRAN. Please upgrade to the latest release.

Also, make sure to update to the latest development version before creating an issue. That will help you determine if it's already been fixed.

d03723003 commented 4 years ago

Howdy

I would like to politely ask whether it is possible to retrieve a comprehensive list of Top 6,000+ U.S. stocks on NYSE, NASDAQ, and AMEX in the rank order of market capitalization.

When I run the function stockSymbols(), I get about 9,000+ stock symbols with no market cap information. It is important for me to sort the stock symbols by market cap. Could you please help?

temp= stockSymbols() Fetching NASDAQ symbols... Fetching non-NASDAQ symbols... dim(temp) [1] 9051 17

Many thanks.

Kind regards.

Andy

PS. I upgraded to the latest version of R 4.0.2 earlier today. I used to get about 6,000 stock symbols with market cap and IPO information etc.

joshuaulrich commented 4 years ago

I used to get about 6,000 stock symbols with market cap and IPO information etc.

That functionality no longer works. Here's what happens with the prior version:

R$ s <- TTR::stockSymbols()
Fetching AMEX symbols...
Error in file(file, "rt") : 
  cannot open the connection to 'https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=AMEX&render=download'
In addition: Warning message:
In file(file, "rt") :
  cannot open URL 'https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=AMEX&render=download': HTTP status was '403 Forbidden'

You can go to that site and manually download the CSVs, but it doesn't work automatically any more.

Yogicsv commented 4 years ago

stockSymbols is still on CRAN even though it does not work. This is confusing for users of the TTR package and should either be fixed or removed.

joshuaulrich commented 4 years ago

stockSymbols is still on CRAN even though it does not work. This is confusing for users of the TTR package and should either be fixed or removed.

stockSymbols() works for me using TTR on CRAN (0.24.2). Please provide some evidence that it doesn't work for you. And I would appreciate it if you could be more certain of your claim and more polite before making strong suggestions like removing something from CRAN.


R$ s <- TTR::stockSymbols()
Fetching NASDAQ symbols...
Fetching non-NASDAQ symbols...
R$ packageVersion("TTR")
[1] '0.24.2'
R$ str(s)
'data.frame':   9148 obs. of  17 variables:
 $ Symbol          : chr  "AAMC" "AAU" "ACU" "ACY" ...
 $ Name            : chr  "Altisource Asset Management Corp Com" "Almaden Minerals, Ltd. Common Shares" "Acme United Corporation. Common Stock" "AeroCentury Corp. Common Stock" ...
 $ LastSale        : logi  NA NA NA NA NA NA ...
 $ MarketCap       : logi  NA NA NA NA NA NA ...
 $ IPOyear         : logi  NA NA NA NA NA NA ...
 $ Sector          : logi  NA NA NA NA NA NA ...
 $ Industry        : logi  NA NA NA NA NA NA ...
 $ Exchange        : chr  "AMEX" "AMEX" "AMEX" "AMEX" ...
 $ Test.Issue      : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
 $ Round.Lot.Size  : int  100 100 100 100 100 100 100 100 100 100 ...
 $ ETF             : logi  FALSE FALSE FALSE FALSE FALSE FALSE ...
 $ Market.Category : chr  NA NA NA NA ...
 $ Financial.Status: chr  NA NA NA NA ...
 $ Next.Shares     : logi  NA NA NA NA NA NA ...
 $ ACT.Symbol      : chr  "AAMC" "AAU" "ACU" "ACY" ...
 $ CQS.Symbol      : chr  "AAMC" "AAU" "ACU" "ACY" ...
 $ NASDAQ.Symbol   : chr  "AAMC" "AAU" "ACU" "ACY" ...
d03723003 commented 4 years ago

Hey Joshua

We appreciate your great work on TTR and stockSymbols(). I think the market cap data can be very valuable. Would it be possible to parse each of the 3 main sources (NYSE, NASDAQ, and AMEX), for instance, with the pseudo code snippets below:

nyse= as.data.frame(read.csv(NYSE-URL))
nasdaq= as.data.frame(read.csv(NASDAQ-URL))
amex= as.data.frame(read.csv(AMEX-URL))

Then we would be able to merge these 3 data frames to sort the data by market cap. It would be great if you could help explain whether this alternative workaround be feasible.

Many thanks.

Kind regards.

Andy

joshuaulrich commented 4 years ago

I think the market cap data can be very valuable.

That data isn't available in the files on their FTP site. It was only available on the stock screener download, and that no longer works. Their server rejects the request, which is what broke the function.

Would it be possible to parse each of the 3 main sources (NYSE, NASDAQ, and AMEX)

There are only 2 sources: NASDAQ and non-NASDAQ. The sources you refer to are no longer available.

d03723003 commented 4 years ago

Hey Joshua

Thanks for your hard work on TTR maintenance.

The former function stockSymbols() had been a useful and convenient tool for many years.

We can still manually download the list of stock ticker symbols from NYSE, NASDAQ, and AMEX:

NYSE https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=NYSE&render=download

NASDAQ https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=NASDAQ&render=download

AMEX https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=AMEX&render=download

However, as you suggest in the previous posts, we can no longer download each list of stock ticker symbols with market cap and IPO information etc automatically in one go.

Here I share the code snippets with different R packages for me to square the circle, but none of these strategies work now.

Maybe one day someone can come up with a better alternative way to download each list automatically in one go.

Many thanks.

Kind regards.

Andy Yeh

library(data.table)
amex= as.data.frame(fread("https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=AMEX&render=download"))

library(readr)
amex= as.data.frame(read_csv("https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=AMEX&render=download"))

library(rio)
amex= import("https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=AMEX&render=download",
             format="csv")

library(RCurl)
amex= download.file("https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=AMEX&render=download",
                    destfile= list.files(pattern="*.csv$"),
                    method= "auto")
joshuaulrich commented 4 years ago

Here I share the code snippets with different R packages for me to square the circle, but none of these strategies work now.

I wouldn't expect any of these to work. The problem is not with the R code. The issue is that the Nasdaq server refuses the requests.

Maybe one day someone can come up with a better alternative way to download each list automatically in one go.

For this to work, the data would need to come from another source.

d03723003 commented 3 years ago

Hello Joshua

I notice that there has been some changes made to the 3 hyperlinks to Nasdaq.com above (for NYSE, Nasdaq, and AMEX).

Could we please sort the data by market cap for all 3 stock exchanges in one go?

Many thanks.

Kind regards.

Andy

joshuaulrich commented 3 years ago

Hi Andy. The data at those URLs is no longer available (see #98). The new URLs do not have market capitalization data. So there's no way to sort the data by market capitalization.

d03723003 commented 3 years ago

Hello Joshua

Thanks for your feedback.

I tried to download the 3 separate csv files for NYSE, NASDAQ, and AMEX. The new csv datasets contains numeric market cap information plus some additional columns. Would it be possible for us to load the new ticker symbol data etc automatically into R via stockSymbols()? In the meantime, I can only manually compile these 3 csv datasets into one (see also my zip attachment and csv file therein).

Many thanks.

Kind regards.

Andy american-stock-checklist20210214.zip

joshuaulrich commented 3 years ago

Andy, I don't believe anything has changed since this past September, when I said

That functionality no longer works. Here's what happens with the prior version:

R$ s <- TTR::stockSymbols()
Fetching AMEX symbols...
Error in file(file, "rt") : 
  cannot open the connection to 'https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=AMEX&render=download'
In addition: Warning message:
In file(file, "rt") :
  cannot open URL 'https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=AMEX&render=download': HTTP status was '403 Forbidden'

You can go to that site and manually download the CSVs, but it doesn't work automatically any more.

I just tried that again with TTR_0.23-6 and I get the same error. So the functionality you're asking for is not possible with the prior code. I might consider a patch if you come up with one that's robust. I'm hesitant to add anything that uses the stock screener site though, because they started blocking it.

d03723003 commented 3 years ago

Howdy Joshua

I can manually download the 3 separate csv files from the URLs below by clicking on the Download CSV blue button each time (see also the Winzip attachment and the csv list of stock symbols available on NYSE, NASDAQ, and AMEX in my previous comment):

https://www.nasdaq.com/market-activity/stocks/screener?exchange=NYSE https://www.nasdaq.com/market-activity/stocks/screener?exchange=NASDAQ https://www.nasdaq.com/market-activity/stocks/screener?exchange=AMEX

However, I cannot figure out how we can automatically download the csv files into R. The blue button design seems to prevent the end user from downloading the csv files via simple hyperlinks.

Kind regards.

Andy

joshuaulrich commented 3 years ago

However, I cannot figure out how we can automatically download the csv files into R.

Agreed. That's why I've said a few times now, "You can go to that site and manually download the CSVs, but it doesn't work automatically any more."

What you're asking for simply cannot be done with the prior code. I'm not going to spend time trying to figure out how to get it to work with the changes they've made. I'll consider a patch if you figure something out, but I can't promise it will be added.

I don't think it's a good use of time for us to keep going over the same points multiple times.

ethanbsmith commented 3 years ago

This is probably a reasonable place to start, if someone wanted to pursue this

r <- lapply(c("NYSE", "AMEX", "NASDAQ"), function(exch) {
      URL <- sprintf("https://api.nasdaq.com/api/screener/stocks?tableonly=true&limit=25&offset=0&exchange=%s&download=true", exch)
      d <- jsonlite::fromJSON(URL)$data$rows
      d$Exchange <- exch
      return(d)
    })

rbind(r[[1]], r[[2]], r[[3]])
mgarv commented 3 years ago

Hey Joshua, Thanks for all your hard work on this. I'm trying to get industry information from stockSymbols() but when I run the function it seems to give me NA for that column. Any chance you know a work around or how to attain all the industry and sector information for each Ticker?

symbols2 <- stockSymbols(exchange = "NASDAQ")

image

Thanks, M

d03723003 commented 3 years ago

Hey Joshua

Many thanks for your hard work and support on TTR and other R packages. We are grateful for your help and guidance.

I tried the code snippets from Ethan, and R returns 7,029 stock symbols from NYSE, NASDAQ, and AMEX with 13 columns.

We can remove the stock symbols with zero market cap values.

Thanks for your advice, Ethan.

I am curious to know how you got the specific URL for JSON download?

Kind regards.

Andy