🤏🏻 Use `investiny` while we fix `investpy`!

alvarobartt commented 1 year ago

Hi everyone, I'm Alvaro Bartolome, ML Engineer, open source developer, and the creator of investpy. As many of you may already know, Investing.com which is the data source that investpy uses has recently changed its APIs as well as the protection of those with Cloudflare, which means that the data is not accessible the same way as before. I and other community members are working actively on solving this issue and looking for alternatives to pull the data from Investing.com.

Most of the solutions proposed by the community worked at some point, but are now discontinued or no longer working, but one solution proposed by @ramakrishnamekala129 at https://github.com/alvarobartt/investpy/issues/610 seems to be working fine and consistently.

So on, I've decided to develop investiny in the meantime, so that everyone can pull the data from Investing.com but in a simpler and faster way. Also as Investing.com is changing so many things lately, I decided to create a new repository named investiny so as to be easily adaptative to changes. Note that investiny is not a replacement for investpy, just an intermediate solution in the meantime while we solve the current issues with investpy.

I'd like all of you to star investiny and contribute to it, as the code is much simpler and easy to understand, but it obviously has less functionality, but we'll get there at some point. Also showing me support by starring, liking, retweeting, etc. is much appreciated.

Thanks, everyone! 🤗

alvarobartt commented 1 year ago

❓ How to use it?

from investiny import historical)data

data = historical_data(investing_id=6408, from_date="09/01/2022", to_date="10/01/2022")
print(data)

🤔 Q&A

Where does the investing_id parameter come from? -> It's the Investing.com unique identifier for each asset that they have listed in their platform, you can look for it either by inspecting the code of Investing.com, or just checking the ID field in the investpy/resources which are a collection of CSV files scraped from Investing.com and contain some of that inner information.

For example, the value investing_id=6408 used in the example above, is retrieved from united states,Apple,Apple Inc,apple-computer-inc,US0378331005,6408,USD,AAPL in https://raw.githubusercontent.com/alvarobartt/investpy/master/investpy/resources/stocks.csv from the column id.

Why investiny instead of publishing a patch to investpy? -> Since Investing.com is constantly changing things, and currently the investpy code is outdated and not well formatted, I've decided to create investiny to easily develop new features there and let everyone contribute as the code is much simpler, while also improving the size of the package, the requests' latency, etc.

investiny available at https://github.com/alvarobartt/investiny

andresrogers commented 1 year ago

Can it be used to get currency crosses ?

alvarobartt commented 1 year ago

Hi, @andresrogers yes, it can be used for that, if you want to know how, please open a new Q&A discussion at https://github.com/alvarobartt/investiny/discussions/categories/q-a Thanks :hugs:

yanghoonkim commented 1 year ago

I found investiny doesn't work too

alvarobartt commented 1 year ago

Hi @yanghoonkim it seems that the Investing.com APIs that investiny was using are blocked by Cloudflare too... Let's see if I get a response from Investing.com as I contacted them to get to an agreement / collaboration.

webdevzilla commented 1 year ago

@alvarobartt Not quite sure why there's hesitation to use my solution for now.

Here's the API I built specifically for this community: http://api.scraperlink.com/investpy/

Using it in your scripts is just a matter of prefixing your current fetch requests with this: http://api.scraperlink.com/investpy/?email=your@email.com&url=

Example:

http://api.scraperlink.com/investpy/?email=your@email.com&url=https%3A%2F%2Fapi.investing.com%2Fapi%2Ffinancialdata%2Fhistorical%2F7%3Fstart-date%3D2022-09-29%26end-date%3D2022-09-29%26time-frame%3DDaily%26add-missing-rows%3Dfalse

I have also added another option that supports historical data for stocks, cryptos, currencies and bonds:

Basics:
* email= So we can send important updates
* type= historical_data
* products=cryptos,currency_crosses,stocks,bonds
* from_date=m/d/Y
* to_date=m/d/Y
- time_frame=Daily,Monthly,Weekly

stocks:
- country= (if not using 'id')
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

cryptos:
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=cryptos&symbol=BTC&from_date=09/27/2022&to_date=09/28/2022

currency_crosses:
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=currency_crosses&name=USD/CAD&from_date=09/27/2022&to_date=09/28/2022

bonds:
- country= (if not using 'id')
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=bonds&from_date=01/01/2022&to_date=09/28/2022&time_frame=Monthly&country=india&name=India%2010Y

* - Required

If anyone needs anything else added, it won't take me more than a few minutes to do it.

It works perfectly fine and is better than nothing.

varshasahasrabuddhe commented 1 year ago

Hi, we are using investpy.economic_calendar and that is breaking too

anarchy89 commented 1 year ago

@webdevzilla could you add all the categories under the markets button?

Indices Stocks Commodities Cryptocurrency Currencies ETFs Funds Bonds Certificates

patrikot commented 1 year ago

@webdevzilla yes, please open up for indices, currencies and commodities, as well.

superjunk1 commented 1 year ago

This works fine for me with currencies and stocks. But I can't figure out how to make it work for funds. Is this category available too?

alvarobartt commented 1 year ago

@webdevzilla can you migrate all those discussions/issues related to your project to a repository under your GitHub? Thanks

webdevzilla commented 1 year ago

@webdevzilla could you add all the categories under the markets button?

Indices Stocks Commodities Cryptocurrency Currencies ETFs Funds Bonds Certificates

@webdevzilla yes, please open up for indices, currencies and commodities, as well.

This works fine for me with currencies and stocks. But I can't figure out how to make it work for funds. Is this category available too?

Yep, it's all added: http://api.scraperlink.com/investpy/

Here's the sample APIs:

stocks:
- country= (if not using 'id')
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

cryptos:
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=cryptos&symbol=BTC&from_date=09/27/2022&to_date=09/28/2022

currency_crosses:
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=currency_crosses&name=USD/CAD&from_date=09/27/2022&to_date=09/28/2022

bonds:
- country= (if not using 'id')
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=bonds&from_date=01/01/2022&to_date=09/28/2022&time_frame=Monthly&country=india&name=India%2010Y

commodities:
- country= (if not using 'id')
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=commodities&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&name=gold

etfs:
- country=
* id=, or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=etfs&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&country=united%20states&symbol=VTI

funds:
- country=
* id=, symbol=, or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=funds&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&symbol=VTSAX

indices:
- country=
* id=, symbol=, or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=indices&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&symbol=SPX

certificates:
- country=
* id=, symbol=, or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=certificates&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&symbol=DEAA01WG

* - Required

@webdevzilla can you migrate all those discussions/issues related to your project to a repository under your GitHub? Thanks

This was only meant to serve as a temporary solution. It wasn't my intention to create a competing repository.

If you feel that's appropriate, I can put one together.

elricStormBringer commented 1 year ago

@alvarobartt Not quite sure why there's hesitation to use my solution for now.

Here's the API I built specifically for this community: http://api.scraperlink.com/investpy/

Using it in your scripts is just a matter of prefixing your current fetch requests with this: http://api.scraperlink.com/investpy/?email=your@email.com&url=

Example:

http://api.scraperlink.com/investpy/?email=your@email.com&url=https%3A%2F%2Fapi.investing.com%2Fapi%2Ffinancialdata%2Fhistorical%2F7%3Fstart-date%3D2022-09-29%26end-date%3D2022-09-29%26time-frame%3DDaily%26add-missing-rows%3Dfalse

I have also added another option that supports historical data for stocks, cryptos, currencies and bonds:
Basics:
* email= So we can send important updates
* type= historical_data
* products=cryptos,currency_crosses,stocks,bonds
* from_date=m/d/Y
* to_date=m/d/Y
- time_frame=Daily,Monthly,Weekly

stocks:
- country= (if not using 'id')
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

cryptos:
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=cryptos&symbol=BTC&from_date=09/27/2022&to_date=09/28/2022

currency_crosses:
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=currency_crosses&name=USD/CAD&from_date=09/27/2022&to_date=09/28/2022

bonds:
- country= (if not using 'id')
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=bonds&from_date=01/01/2022&to_date=09/28/2022&time_frame=Monthly&country=india&name=India%2010Y

* - Required 
If anyone needs anything else added, it won't take me more than a few minutes to do it.

It works perfectly fine and is better than nothing.

hi @alvarobartt / @webdevzilla , I came to know of this great python module 'investiny' and 'investpy' today, from StackOverflow.com.

I tried using it, and keeps getting the dreaded '403' error even though I am a new user. Is this APIscraperLink the only current fix?

I have been using python for more than 4 years, but my programming is more in modelling, rather than IT infrastructure coding...

mscharstrom commented 1 year ago

@alvarobartt any updates from investing.com? I'm mostly interested in the economic calendar. Does your workaround fix that @webdevzilla ?

Thanks in advance.

vincentli2023 commented 1 year ago

@webdevzilla I appreciate it! Do you think your solution will still function for another few months in the future? Cause I am worried that it will be blocked again by investing.com and I have to change my code infrastructure once again....

webdevzilla commented 1 year ago

@alvarobartt any updates from investing.com? I'm mostly interested in the economic calendar. Does your workaround fix that @webdevzilla ?

Thanks in advance.

Sure, I'll add it. Please email me so that I can update you once added.

@webdevzilla I appreciate it! Do you think your solution will still function for another few months in the future? Cause I am worried that it will be blocked again by investing.com and I have to change my code infrastructure once again....

Shouldn't have any problems.

I am processing millions of queries without issues.

vincentli2023 commented 1 year ago

@webdevzilla I think there might be an issue with bond API endpoints, as the response can't be converted to json. The response text of bond API is in html format instead of dictionary/json format

webdevzilla commented 1 year ago

Yes, ScraperLink only captures the raw response. It doesn't parse the data. This repository probably has the code for parsing the HTML response into JSON. I can also add parsing to JSON directly, but I'm super bogged down right now. I might need a few days to get to it. If you email me, I can update you once it's added.

anarchy89 commented 1 year ago

Yes, ScraperLink only captures the raw response. It doesn't parse the data. This repository probably has the code for parsing the HTML response into JSON. I can also add parsing to JSON directly, but I'm super bogged down right now. I might need a few days to get to it. If you email me, I can update you once it's added.

Hi @webdevzilla could you explain how your solution works and share the code for your website and how you implemented it im really curious as to how you did it. It's a really good solution.

webdevzilla commented 1 year ago

It requires a massive pool of IP addresses. Not sure if I should say more. 🤐

vincentli2023 commented 1 year ago

It requires a massive pool of IP addresses. Not sure if I should say more. 🤐

@webdevzilla thank you for providing this alternative solution! Really appreciate it!!

anarchy89 commented 1 year ago

It requires a massive pool of IP addresses. Not sure if I should say more. 🤐

I understand but how do you get around cloudflare haha

webdevzilla commented 1 year ago

I don't get Cloudflare blocks when using my proxies.

ramakrishnamekala129 commented 1 year ago

if u send us your proxies solution may be we can investiny or investpy

On Thu, Dec 8, 2022 at 2:23 AM webdevzilla @.***> wrote:

I don't get Cloudflare blocks when using my proxies.

— Reply to this email directly, view it on GitHub https://github.com/alvarobartt/investpy/issues/611#issuecomment-1341582865, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHEE5RBA35QIWAWC4JAHU3LWMD2OHANCNFSM6AAAAAAQZ4NRQY . You are receiving this because you were mentioned.Message ID: @.***>

webdevzilla commented 1 year ago

There wouldn't be a way for me to control where they're used. My proxy pool would get abused to oblivion.

The solution I've provided is 100% free and I can presumably make it work forever. In fact, I will probably add backups to it as well, so that even if Investing.com no longer exists, the API will continue working seamlessly.

I need this for my own purposes so there's no way I would let it die.

anarchy89 commented 1 year ago

@webdevzilla so you funnel the requests through your proxies and that's why cloudflare allows the requests? I don't want access to your pools, just wanna understand how it's technically done.

webdevzilla commented 1 year ago

@webdevzilla so you funnel the requests through your proxies and that's why cloudflare allows the requests? I don't want access to your pools, just wanna understand how it's technically done.

Yes.

anarchy89 commented 1 year ago

@webdevzilla so you funnel the requests through your proxies and that's why cloudflare allows the requests? I don't want access to your pools, just wanna understand how it's technically done.

Yes.

Is it something like this? https://www.bestproxyreviews.com/how-to-bypass-cloudflare/

webdevzilla commented 1 year ago

I have my own pool of proxies. If you want access it, we can work something out.

vincentli2023 commented 1 year ago

@webdevzilla I can't run bonds API endpoint this morning, I also tried the example you gave but no luck, the error message is below:

Access denied | www.investing.com used Cloudflare to restrict access Githubissues.

Githubissues is a development platform for aggregating issues.

alvarobartt / investpy

🤏🏻 Use `investiny` while we fix `investpy`! #611

❓ How to use it?

🤔 Q&A