search

pranjal-joshi / Screeni-py

A Python-based stock screener to find stocks with potential breakout probability from NSE India.
MIT License
583 stars 205 forks source link

Enhancement: Caching fetched stock data into shared dictionary for reuse. #50

Closed swarpatel23 closed 2 years ago

swarpatel23 commented 3 years ago

In the current version, every time stock data fetched which causes increased data usage. We can store fetched stock data into a shared dictionary while running the first time and on the next option selection we can use cached data.

I tried to use multiprocessing.Manager().dict() to cache stock data. for peried period = 365d it takes 350mb of memory for 1678 stock codes . So considering high memory usage I created cacheEnabled option in configManager.py using which we can enable/disable caching.

Can I make PR ?

raviyellani commented 3 years ago

Hi, I have been following this repo, as in nsetools the data is fetched in the form of excel and the stocks are segregated, in my view in 1678 stocks lot of are penny stocks, so instead of fetching from nse tools excel, if we can provide a good nifty 500 stocks as input in the form of excel, so that we can analyze and go through only few quality stocks charts based on this logics output

This the link from where nse tools is downloading the stocks data

http://www1.nseindia.com/content/equities/EQUITY_L.csv

On Sat, May 22, 2021 at 6:55 PM Swar Patel @.***> wrote:

In the current version, every time stock data fetched which causes increased data usage. We can store fetched stock data into a shared dictionary while running the first time and on the next option selection we can use cached data.

I tried to use multiprocessing.Manager().dict() to cache stock data. for peried period = 365d it takes 350mb of memory for 1678 stock codes . So considering high memory usage I created cacheEnabled option in configManager.py using which we can enable/disable caching.

Can I make PR ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pranjal-joshi/Screeni-py/issues/50, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABF2G3F5EXS722UR2ZYA4TLTO6WDDANCNFSM45KUTTDA .

-- Thanking You & Best Regards

pranjal-joshi commented 3 years ago

@swarpatel23 Sure! Let's start working on it. Please sync your existing main with the upstream to avoid conflicts.

@raviyellani We are using yfinance which shares stockdata in form of DataFrame If you want Nifty500 Feature, You can contribute by creating a list which fetch Nifty500 stocks using some mechanism. We can use this list for further input to screener.

raviyellani commented 3 years ago

Just tried giving input using nifty500 excel file instead of nsetools stockcodes, looks like worked out, a short video attached kindly go through and confirm, is it right way Screen Recording (22-05-2021 21-27-11).wmv https://drive.google.com/file/d/1MdAnqwxKcQAEARTO2er9Y8Gu8F91e3RE/view?usp=drive_web ..thank you

On Sat, May 22, 2021 at 8:30 PM Pranjal Joshi @.***> wrote:

@swarpatel23 https://github.com/swarpatel23 Sure! Let's start working on it. Please sync your existing main with the upstream to avoid conflicts.

@raviyellani https://github.com/raviyellani We are using yfinance which shares stockdata in form of DataFrame If you want Nifty500 Feature, You can contribute by creating a list which fetch Nifty500 stocks using some mechanism. We can use this list for further input to screener.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pranjal-joshi/Screeni-py/issues/50#issuecomment-846420278, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABF2G3E4P44U5TMX352O42DTO7BKNANCNFSM45KUTTDA .

-- Thanking You & Best Regards

pranjal-joshi commented 3 years ago

@raviyellani I understood the video. Please create another issue so we can independently develope your feature. This feature can be used to get detailed information from the stocks name that are on your watchlist or got from some other online screeners. So user just have to save the watchlist names in the sheet and with new feature, wr can do it's analysis. Looking forward for a Pull Request from you for further development of this feature. Please Go through Contributing Guidelines before creating a PR.

raviyellani commented 3 years ago

okay..thank you

On Sat, May 22, 2021 at 9:47 PM Pranjal Joshi @.***> wrote:

Just tried giving input using nifty500 excel file instead of nsetools stockcodes, looks like worked out, a short video attached kindly go through and confirm, is it right way Screen Recording (22-05-2021 21-27-11).wmv

https://drive.google.com/file/d/1MdAnqwxKcQAEARTO2er9Y8Gu8F91e3RE/view?usp=drive_web ..thank you

On Sat, May 22, 2021 at 8:30 PM Pranjal Joshi @.***> wrote:

@swarpatel23 https://github.com/swarpatel23 https://github.com/swarpatel23 Sure! Let's start working on it. Please sync your existing main with the upstream to avoid conflicts.

@raviyellani https://github.com/raviyellani https://github.com/raviyellani We are using yfinance which shares stockdata in form of DataFrame If you want Nifty500 Feature, You can contribute by creating a list which fetch Nifty500 stocks using some mechanism. We can use this list for further input to screener.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub

50 (comment)

https://github.com/pranjal-joshi/Screeni-py/issues/50#issuecomment-846420278 , or unsubscribe

https://github.com/notifications/unsubscribe-auth/ABF2G3E4P44U5TMX352O42DTO7BKNANCNFSM45KUTTDA .

-- Thanking You & Best Regards

-

       Ravi Yellani*

I understood the video. Please create another issue so we can independently develope your feature. This feature can be used to get detailed information from the stocks name that are on your watchlist or got from some other online screeners. So user just have to save the watchlist names in the sheet and with new feature, wr can do it's analysis. Looking forward for a Pull Request from you for further development of this feature. Please Go through Contributing Guidelines before creating a PR.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pranjal-joshi/Screeni-py/issues/50#issuecomment-846430516, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABF2G3F5WK234TYIAHSTMODTO7KJ7ANCNFSM45KUTTDA .

-- Thanking You & Best Regards

pranjal-joshi commented 3 years ago

@swarpatel23 How about storing processed data-frame with all the parameters as a cache rather than storing raw stock data? It will reduce screening time significantly for subsequent screens at cost of more disk space which is usually no concern.

swarpatel23 commented 3 years ago

@pranjal-joshi good idea!!! that will definitely reduce CPU usage and reduce screening time.

pranjal-joshi commented 3 years ago

@raviyellani

  1. The Screenshot has not been attached.
  2. The Screenshot has not been attached.
pranjal-joshi commented 3 years ago

@raviyellani Requested Feature has been added in 370f00f. You can test it now.

raviyellani commented 3 years ago

@pranjal-joshi https://github.com/pranjal-joshi Ya sure brother, will test...appreciation for great work

On Fri, May 28, 2021 at 1:55 PM Pranjal Joshi @.***> wrote:

@raviyellani https://github.com/raviyellani Requested Feature has been added in 370f00f https://github.com/pranjal-joshi/Screeni-py/commit/370f00f5395d0baf1594223a8ead4ae54a26daef. You can test it now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pranjal-joshi/Screeni-py/issues/50#issuecomment-850245629, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABF2G3FPASFAAMDDEGT6SF3TP5HONANCNFSM45KUTTDA .

-- Thanking You & Best Regards

pranjal-joshi commented 2 years ago

Hello @swarpatel23

I've discovered a minor bug with the cache loading method while doing early morning analysis. I had stock_data_280122.pkl file for an earlier trading session, as I'm running screening today before 9.15 AM, the system should consider the old cache as the data source. But currently, it is forcing to download a new .pkl file as it is looking for a file named stock_data_310122.pkl which is not actually possible to generate before 3.30 PM today.

Can you help resolving this? Please keep your fork up to date before getting started :)