CarlosOliveira1988 / Investment_Portfolio_Analysis

This repository is useful to perform some financial analysis in Investment Portfolio
4 stars 0 forks source link

Apply multiprocessing in the tasks related to web-scraping (main tabs of the application) #78

Closed CarlosOliveira1988 closed 2 years ago

CarlosOliveira1988 commented 2 years ago

Is your feature request related to a problem? Please describe.

The task of getting information from websites (through web-scrapping or YFinance API) takes too much time to be performed. This is really noticed when the application needs to work with 15 or more stocks.

Describe the solution you'd like

Apply the multiprocessing in the 'Renda Variável', 'Renda Fixa' and 'Tesouro Direto' tabs

Describe alternatives you've considered

Use the 'multiprocessing.Pool' library to perform the described activity. This link has useful information: https://www.analyticsvidhya.com/blog/2021/04/a-beginners-guide-to-multi-processing-in-python/

Additional context

Create a test spreadsheet with 20 opened positions per tab ('Renda Variável', 'Renda Fixa' and 'Tesouro Direto' tabs). Then, try to measure the current time and the improved time, in order to check the difference of processing time.

CarlosOliveira1988 commented 2 years ago

Based on the tests using the initial 'web-scraping-multiprocessing' branch, using the attached PORTFOLIO_TEMPLATE_PERFORMANCE.xlsx spreadsheet, we have the following results:

CarlosOliveira1988 commented 2 years ago

I've worked in a new approach in order to reduce the time to open new spreadsheets and also to launch the application. It seems the slowest tasks are related to web-scraping using 'BeautifulSoup' and 'requests' modules. I could not reduce their times for now. Maybe in the future I try other things.

Anyway, running the same mentioned spreadsheet with new code, I could reduce the time from 46 to 35 seconds, that means 24% less time.