search

alvarobartt / investpy

Financial Data Extraction from Investing.com with Python
https://investpy.readthedocs.io/
MIT License
1.62k stars 376 forks source link

Improve update frequency of the static files or use another method #256

Open arainboldt opened 3 years ago

arainboldt commented 3 years ago

Hi,

Great package! I really appreciate its simple and useful api. It would be better, however, to have stock/etf/...etc data updated, or update-able, instead of being stored permanently in the package's resources directory. IPOs, new fund offerings and the like are frequent in the US markets and consequently the data needs to be updated frequently in order to stay relevant and be useful.

You might consider adding some methods to explicitly update the local database if you want to keep query times down using local data for the rest of the api.

Happy new year!

Best,

Andrew

alvarobartt commented 3 years ago

Hi @arainboldt! That sounds great, anyway I'm still not sure how to proceed with this issue, some solutions were proposed in #279 but as said, not sure yet...

Anyway, feel free to propose any solution! 👍🏻 Another option for you to get the most updated data is to use the investpy.search_quotes() function to use Investing.com search engine instead of the static files. More information regarding this functionality at https://github.com/alvarobartt/investpy/wiki/investpy.search_quotes()

alvarobartt commented 3 years ago

So I'll be listening to your proposal/s 👍🏻 Let me know if you have any feasible solution in mind!

arainboldt commented 3 years ago

hey @alvarobartt thanks for following up and for the invitation for proposals. I'll read through the linked issue and the code again and get back to you with my thoughts.

all the best and stay safe!

datatalking commented 2 years ago

@arainboldt and @alvarobartt regarding the data, if it is being lost perhaps persistent data might be a solution? I'm expanding my algorithmic trading stack this summer and your repo is high on my list to integrate. Would parquet files or a sqlite mysql database work?

alvarobartt commented 2 years ago

Hi @datatalking thanks for proposing those file formats, anyway I still need to think about how am I going to refactor it, according to my development plan I should start in a couple of weeks and the refactor will thoroughly remove the static files if possible, otherwise I may use a cloud storage account from AWS or GCP so as to store those files there with a cloud function that runs every week to collect the data from Investing.com.

But the ideal scenario should be removing the static files and just using the Investing.com API, but there are some data that need to be manually scraped... During the development phase of investpy v2.0.0 I'll post all the updates in a Discussion, so that any person can comment there and help me out with proposals :hugs: