[Bug] ETF Screener is not responding to settings in the configuration file.

[deeleeramone]

/etf/scr/screen

This function is unresponsive to presets. It returns an alphabetized list, and all sort functions are based on the limit number of results returned.

Separately, there are many screener filters available from StockAnalysis that are not included in the etf_config.ini file. Configuring for all available filters should be possible like it is for the stock screener.

[DidierRLopes]

I don't understand what's the issue, can you be more clear?

E.g. I was able to play with the etf_config and the screen was reactive:

PS: Open a different issue for the filters that are missing and I will add them

[jmaslek]

This is actually an issue with stockanalysis changing their scraping again.

If you look at my repo that gets this data, I can only pull a few tickers before getting errors in the requests (notice how all your tickers start with A).

Also we don't take the ETF screener from their website, I make a custom screener from the overview that I grab so there is in fact nothing more to add.

This feature may need to get taken out with how unreliable the scraping is.

[DidierRLopes]

Can we fix that scraping by any chance?

@minhhoang1023 how much is this feature currently used?

[deeleeramone]

This feature can't be used because it doesn't function, so metrics will not provide any insight here. If you look at your example results, they are all exactly the same tickers. Your filters for sorting are being only applied to those 3 tickers and nothing else.

[DidierRLopes]

This feature can't be used because it doesn't function, so metrics will not provide any insight here. If you look at your example results, they are all exactly the same tickers. Your filters for sorting are being only applied to those 3 tickers and nothing else.

Those 3 tickers because of the arguments I selected. The functionality works well, the issue as @jmaslek pointed out is on his database being limited to the initial tickers of the database starting with A

[deeleeramone]

A screener for ETFs that only start with the letter A is akin to broken, and will only result in more issues being generated by other uses. Therefore, I suggest that this screener be deprecated for now.

[DidierRLopes]

Depends on how easy it is to fix or replace the database being used. Because the filtering middleman works. Do you know any datasource we can use to replace this? Would be good to get @jmaslek estimate of effort to fix this as well.

[deeleeramone]

https://portfolio.nasdaq.com/smart-portfolio/screener/etfs https://www.nasdaq.com/market-activity/etf/screener https://finance.yahoo.com/screener/etf/new/ https://www.justetf.com/en/find-etf.html https://www.etf.com/etfanalytics/etf-finder https://www.etfchannel.com https://seekingalpha.com/etfs-and-funds/etf-screener https://etfdb.com

Canadian ETFs: https://etfmarket.neo.inc/en/etf-screener https://www.fundlibrary.com/Etfs/Search

[jmaslek]

Depends on how easy it is to fix or replace the database being used. Because the filtering middleman works. Do you know any datasource we can use to replace this? Would be good to get @jmaslek estimate of effort to fix this as well.

If we replace this solution I would want to use an API instead of scraping to avoid having to change this code every couple months.

I'm gonna try playing around today to see if I can get around the errors, if not I'll pr and unfortunately remove the feature for now.

[jmaslek]

Update: I can scrape 500 etfs in.....5 hours (added a retry). I am setting up runners throughout the day to hopefully get close to the full 2800 they offer.

There should be more than just tickers starting with A now, but I think I only got through "D"

[minhhoang1023]

late to the party, but here's the usage. It got used 144 times. Not amazing, but not the worst. For some context, out of almost 700 commands we have, this command ranks at 300 appx.

[jmaslek]

I ended up getting rid of this for now since I cant scrape stockanalysis.

Investpy is a potential replacement.