ConnectionError: ERR#0015: error 403, try again later.

[divyankm]

Code-

import investpy

df = investpy.get_stock_historical_data(stock='AAPL',
                                        country='United States',
                                        from_date='01/01/2010',
                                        to_date='01/01/2020')
print(df.head())

Error-

ConnectionError                           Traceback (most recent call last)
[<ipython-input-4-f6f4235b7e47>](https://localhost:8080/#) in <module>
      4                                         country='United States',
      5                                         from_date='01/01/2010',
----> 6                                         to_date='01/01/2020')
      7 print(df.head())

[/usr/local/lib/python3.7/dist-packages/investpy/stocks.py](https://localhost:8080/#) in get_stock_historical_data(stock, country, from_date, to_date, as_json, order, interval)
    663         if req.status_code != 200:
    664             raise ConnectionError(
--> 665                 "ERR#0015: error " + str(req.status_code) + ", try again later."
    666             )
    667 

ConnectionError: ERR#0015: error 403, try again later.

[Exganza]

Thanks, @samjmck this is really useful, there's a limitation on the daily period it's retrieved 365 days maximum, but still very useful

[alvarobartt]

So I've already contacted Investing.com and started a conversation there, I'll let you all know the outcome of it. Thanks for your patience! :hugs:

[nicklatin]

thanks Alvaro. I guess a relevant question here is whether there's a violation of their (Investing.com) terms and services?

[arexms]

@alvarobartt , all

I have a workaround / temporary solution which migh be applicable for some of you.

Below python script works on windows with edge web browser but might work also with other environments (after some modifications). Script uses selenium to get content page and then XPATH to find required value.

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

# Initialize edge driver
options = webdriver.EdgeOptions()
options.add_argument("headless")
driver = webdriver.Edge(options=options)

# download page
driver.get('https://www.investing.com/currencies/eur-usd')

# wait few seconds
time.sleep(5)

# we need to accept cookies, etc
element = driver.find_element(By.ID, 'onetrust-accept-btn-handler')
driver.execute_script("arguments[0].click();", element)

# get value using XPATH
number = driver.find_element(By.XPATH,'//*[@id="__next"]/div/div/div/div[2]/main/div/div[1]/div[2]/div[1]/span')

# done :)
print("##################")
print(number.text)
print("##################")

# some cleanup
time.sleep(60)

As a next step, all we need to do is to create a job with such script in Jenkins or in other scheduler. Let me know what you think.

[ymyke]

BTW & FWIW: I updated the tessa package (https://github.com/ymyke/tessa) to use yfinance instead of investpy. (There's all kinds of other improvements as well.) yfinance is of course not the optimal replacement for investpy but better than nothing for many use cases. – Still hoping for a resolution here and will re-add investpy once it becomes available again.

[alvarobartt]

thanks Alvaro. I guess a relevant question here is whether there's a violation of their (Investing.com) terms and services?

Hi @nicklatin it's nice that you ask this, I contacted them more than 2 years ago before developing investpy to ask the same thing and they told me that it was OK to develop and open-source investpy as long as the data source was being mentioned, so Investing.com is mentioned so it's part of the agreement! So everything's fine regarding this matter!

[alvarobartt]

BTW & FWIW: I updated the tessa package (https://github.com/ymyke/tessa) to use yfinance instead of investpy. (There's all kinds of other improvements as well.) yfinance is of course not the optimal replacement for investpy but better than nothing for many use cases. – Still hoping for a resolution here and will re-add investpy once it becomes available again.

So to everyone looking for an alternative as @ymyke is mentioning you can use finance from @ranaroussi, or even his own custom package tessa which is currently using yfinance in the background.

[maread99]

For anyone looking to temporarily switch to yahoo for prices, market-prices offers an interface with enhanced querying and additional functionality.

I was hoping to add investpy as a data source for market-prices although was waiting for intraday data to be added (I know there had been intermittent chatter about this). If investpy can get back up and running (🤞🤞) then I'll look to add it as a option for the source data.

Good luck @alvarobartt!

EDIT: for anyone looking at market-prices, the pandas dependency is temporarily <= 1.4.4 (working on a fix for the recent 1.5 release - https://github.com/maread99/market_prices/issues/89).

[alvarobartt]

Thanks also for your proposal @maread99! Let's see how the conversation with Investing.com turns out and hopefully I can get back to work on investpy and all my other personal projects that have been left aside due to the lack of time during the last year! 💪🏻

[alvarobartt]

Hi everyone! Good news 🎉

I've just released investiny which is a tiny investpy with a single function to download the historical data from Investing.com, check it out at https://github.com/alvarobartt/investiny (support is welcomed so if it's useful to you in the meantime while investpy is fixed please star it and share it so that anyone can use it while we fix investpy)

Thanks to everyone contributing and helping out! 🤗

[wajihwajeh]

Hi everyone! Good news 🎉

I've just released investiny which is a tiny investpy with a single function to download the historical data from Investing.com, check it out at https://github.com/alvarobartt/investiny (support is welcomed so if it's useful to you in the meantime while investpy is fixed please star it and share it so that anyone can use it while we fix investpy)

Thanks to everyone contributing and helping out! 🤗

How do we get the investing_id? running search quotes on investor gives the same 403 error

[webdevzilla]

Hello everyone!

I was using a custom script for investing.com, which led me to investpy when things broke.

I ended up building a solution, but feel guilty not sharing it with everyone stuck on this issue. So, here is the API to bypass Cloudflare: http://api.scraperlink.com/investpy/

Using it in your scripts is just a matter of prefixing your current fetch requests with this: http://api.scraperlink.com/investpy/?email=your@email.com&url=

Example:

http://api.scraperlink.com/investpy/?email=your@email.com&url=https%3A%2F%2Fapi.investing.com%2Fapi%2Ffinancialdata%2Fhistorical%2F7%3Fstart-date%3D2022-09-29%26end-date%3D2022-09-29%26time-frame%3DDaily%26add-missing-rows%3Dfalse

I have also added another option that supports historical data for stocks, cryptos and currencies for now:

Basics:
* email= So we can send important updates
* type= historical_data
* products=cryptos,currency_crosses,stocks
* from_date=Y-m-d
* to_date= Y-m-d
- time_frame=Daily,Monthly,Weekly

stocks:
* country=
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

cryptos:
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=cryptos&symbol=BTC&from_date=09/27/2022&to_date=09/28/2022

currency_crosses:
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=currency_crosses&name=USD/CAD&from_date=09/27/2022&to_date=09/28/2022

If there's interest from the community, I am happy to add more features and data points.

Just let me know what's important to you.

[nicklatin]

Alvaro, that's great! Does this mean interest rate and commodities data is retrievable?

Btw, for those who need access to a range of cryptoasset data, feel free to check out my project CryptoDataPy (which is integrating InvestPy): https://github.com/systamental/cryptodatapy

[fredericojordan]

Hi everyone! Good news 🎉 I've just released investiny which is a tiny investpy with a single function to download the historical data from Investing.com, check it out at https://github.com/alvarobartt/investiny (support is welcomed so if it's useful to you in the meantime while investpy is fixed please star it and share it so that anyone can use it while we fix investpy) Thanks to everyone contributing and helping out! 🤗

How do we get the investing_id? running search quotes on investor gives the same 403 error

To get the investing_id:

1. Navigate to https://tvc4.investing.com/
2. Open your browser console, go to network tab
3. Use the investing.com search bar to find your stock and click it
4. Search the network tab for the /history?symbol= and the symbol query param is your investing_id

[alvarobartt]

Hi everyone! Good news 🎉 I've just released investiny which is a tiny investpy with a single function to download the historical data from Investing.com, check it out at https://github.com/alvarobartt/investiny (support is welcomed so if it's useful to you in the meantime while investpy is fixed please star it and share it so that anyone can use it while we fix investpy) Thanks to everyone contributing and helping out! 🤗

How do we get the investing_id? running search quotes on investor gives the same 403 error

Hi @wajihwajeh, so you can either retrieve those manually from the resources/stocks.csv or any other CSV file under resources, taking the id of the column that you want, anyway I'll implement something in investiny to easily look for those!

Please star investiny and create all the related issues there, not here, as it's easier for me to track, thanks! 🤗

[alvarobartt]

Hi everyone! Good news 🎉 I've just released investiny which is a tiny investpy with a single function to download the historical data from Investing.com, check it out at https://github.com/alvarobartt/investiny (support is welcomed so if it's useful to you in the meantime while investpy is fixed please star it and share it so that anyone can use it while we fix investpy) Thanks to everyone contributing and helping out! 🤗

How do we get the investing_id? running search quotes on investor gives the same 403 error

To get the investing_id:

1. Navigate to https://tvc4.investing.com/

2. Open your browser console, go to network tab

3. Use the investing.com search bar to find your stock and click it

4. Search the network tab for the `/history?symbol=` and the `symbol` query param is your `investing_id`

Hi @fredericojordan, so this works if doing it manually, but I'm planning to actually develop a way to automatically retrieve those in order to avoid either manual search or relying on static CSV files!

Please star investiny and create all the related issues there, not here, as it's easier for me to track, thanks! 🤗

[alvarobartt]

Alvaro, that's great! Does this mean interest rate and commodities data is retrievable?

Btw, for those who need access to a range of cryptoasset data, feel free to check out my project CryptoDataPy (which is integrating InvestPy): https://github.com/systamental/cryptodatapy

Hi @nicklatin, yes, all the Investing.com data is retrievable from Investing.com using the historical_data function from investiny, but you need to know the Investing.com ID of your asset in advance! You can either look for it in the id column of the CSV files under investpy/resources, or just inspect the Investing.com source code. Anyway, I'm developing a way to automatically do this so as to stop relying on static files.

Please star investiny and create all the related issues there, not here, as it's easier for me to track, thanks! 🤗

[alvarobartt]

Hi, @webdevzilla I'll test your solution, but I'm trying to contact Investing.com or look for alternatives, not to actually bypass their protection as there may be legal terms there (so check those before actually sharing your solution), as I'm not responsible for that.

I'll let you all know when I get a response from Investing.com so as to see whether we can reach an agreement to continue investpy, but in the meantime, you can use investiny which is a project that I've created so as to develop there temporarily solutions to investpy in a fast and tiny way!

Please star investiny and create all the related issues there, not here, as it's easier for me to track, thanks! 🤗

[alvarobartt]

FYI I've created here an issue at https://github.com/alvarobartt/investpy/issues/611, and a Discussing at https://github.com/alvarobartt/investpy/discussions/612, both are pinned and have some details on how to use investiny which is a tiny version of investpy that I've just created so as to develop there some temporary solutions to keep on pulling data from Investing.com, without having to bypass their protection or violate their terms of use.

[ymyke]

Are you sure that the workaround via https://tvc4.investing.com/ doesn't violate their TOS, @alvarobartt ?

[alvarobartt]

Are you sure that the workaround via https://tvc4.investing.com/ doesn't violate their TOS, @alvarobartt ?

Hi @ymyke so we are mentioning the source the data comes from and it's a publicly available API. What may violate those is to actually bypass their protection protocols, but not sure, this is why I'm telling people developing their own solutions to contact Investing.com just like I did some time ago, to get their approval.

[OGsiji]

One simple question that has not been answered is, How do I install investiny?

[alvarobartt]

Hi @OGsiji thanks for asking! To install it just pip install investiny and that will do the work, anyway make sure to read and go through investiny's README that contains more details and information!

FYI I'm also working on the documentation so hopefully, I can release it soon too to answer all the questions and provide code examples!

[OGsiji]

ERROR: Could not find a version that satisfies the requirement investiny (from versions: none)
ERROR: No matching distribution found for investiny

I got this error when using this command, Can you investigate this too. Thanks for the help, you are doing a great work.

[alvarobartt]

Hi @OGsiji can you report this issue at https://github.com/alvarobartt/investiny/issues?

[AyseTheArk]

I wanted to use the historical data for the trendet library, like in the code here https://github.com/alvarobartt/trendet

I had the same error, but with investiny I was able to create the df finally, however the column names seem to be different now. In the code you have 'Up Trend' , 'Down Trend', 'Date'. But in the investiny I only saw 'open', 'high', 'low'. Am I using the wrong data? I got the one from investiny as follows:

from investiny import historical_data, search_assets

search_results = search_assets(query="AAPL", limit=1, type="Stock", exchange="NASDAQ")
investing_id = int(search_results[0]["ticker"]) # Assuming the first entry is the desired one (top result in Investing.com)

data = historical_data(investing_id=investing_id, from_date="09/01/2022", to_date="10/01/2022")

[alvarobartt]

Hi, @AyseTheArk do you mind opening that as a separate issue at https://github.com/alvarobartt/investiny/issues? Otherwise is just a mess for me to go through all the issues here, thanks!

[KostyaCholak]

Hi, @alvarobartt ! I've encountered the 403 Error problem today and found that using curl seem to be working fine, no 403 error. And the only difference I can see is the headers ordering - requests shuffles headers, while curl preserves them as provided. So I tried using urllib.request and it worked.

I'm using Python 3.10.5

Maybe this can solve all 403 errors in the project?

minimal working example:

import urllib.request

# take them from your browser, no cookies required
headers = {}

req = urllib.request.Request(f'https://sbcharts.investing.com/events_charts/us/222.json', b"", headers)
with urllib.request.urlopen(req) as response:
    response = response.read().decode()

[alvarobartt]

Hi, @alvarobartt !

I've encountered the 403 Error problem today and found that using curl seem to be working fine, no 403 error.

And the only difference I can see is the headers ordering - requests shuffles headers, while curl preserves them as provided.

So I tried using urllib.request and it worked.

I'm using Python 3.10.5

Maybe this can solve all 403 errors in the project?

minimal working example:

import urllib.request

# take them from your browser, no cookies required

headers = {}

req = urllib.request.Request(f'https://sbcharts.investing.com/events_charts/us/222.json', b"", headers)

with urllib.request.urlopen(req) as response:

    response = response.read().decode()

Thanks @KostyaCholak can you please open a separate issue at https://github.com/alvarobartt/investiny? I'll try to test that approach later today, thanks!

[AndresFe1996]

hello everyone, my name is andres, I am a python programming student and investPY made me very happy, I would like to know if maybe investPY could find a solution by the end of the year? I don't know how serious the error is but from what I've been investigating it seems to be an update issue of investing.com itself. I thank all developers for their time and stay tuned.

[kth2]

thanks Alvaro. I guess a relevant question here is whether there's a violation of their (Investing.com) terms and services?

Hi @nicklatin it's nice that you ask this, I contacted them more than 2 years ago before developing investpy to ask the same thing and they told me that it was OK to develop and open-source investpy as long as the data source was being mentioned, so Investing.com is mentioned so it's part of the agreement! So everything's fine regarding this matter!

Hi @alvarobartt ,wondering if you get any update or feedback from Investing.com regarding this restriction? If they are open-source, it should allow everyone to do web-scraping. Thanks!

[Merijn68]

Hi, @alvarobartt,

I am a student of data science from the Netherlands and I am looking for some open source bond data. I came across InvestPy and this looks like a very useful package to get some history of corporate bonds. Unfortunately I encountered this 403 error when I try to run the examples. Is there a fix now? Or a fix expected shortly?

Also - when I install the package it gives an error on the idna package:

Installing collected packages: idna
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ray 2.0.1 requires click<=8.0.4,>=7.0, which is not installed.
google-api-core 2.10.2 requires google-auth<3.0dev,>=1.25.0, which is not installed.
ccxt 1.66.32 requires cryptography>=2.6.1, which is not installed.
autorom 0.4.2 requires click, which is not installed.
autorom-accept-rom-license 0.4.2 requires click, which is not installed.

Then it says it does install succesfully:

successfully installed idna-3.4

So I was wondering if this is a problem as well.

Thanks in advance, Kind regards, Merijn

[KTHor2]

Hi, @alvarobartt,

I am a student of data science from the Netherlands and I am looking for some open source bond data. I came across InvestPy and this looks like a very useful package to get some history of corporate bonds. Unfortunately I encountered this 403 error when I try to run the examples. Is there a fix now? Or a fix expected shortly?

Also - when I install the package it gives an error on the idna package:

Installing collected packages: idna
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ray 2.0.1 requires click<=8.0.4,>=7.0, which is not installed.
google-api-core 2.10.2 requires google-auth<3.0dev,>=1.25.0, which is not installed.
ccxt 1.66.32 requires cryptography>=2.6.1, which is not installed.
autorom 0.4.2 requires click, which is not installed.
autorom-accept-rom-license 0.4.2 requires click, which is not installed.

Then it says it does install succesfully:

successfully installed idna-3.4

So I was wondering if this is a problem as well.

Thanks in advance, Kind regards, Merijn

Hi @Merijn68 you can refer to "Issues" query here, currently there are known error 403 on investpy script running. This is due to the webscrapping to investing.com got blocked, not due to any sort of new Investing.com protection against investpy, but actually due to Cloudflare, as they included protection in the latest release of Investing.com, so the previous API was deprecated and the current one is protected.

[Merijn68]

Hi KTHor2,

Thanks for your reply. So Investpy is currently not working due to this error, and there is no solution for this? What do you mean when you say the "current API is protected"?

[alvarobartt]

Hi KTHor2,

Thanks for your reply. So Investpy is currently not working due to this error, and there is no solution for this? What do you mean when you say the "current API is protected"?

You can use investiny in the meantime, even though at some point you'll face the same issue with Investing.com temporarily blocking you if you send too many requests... I've already contacted Investing.com in order to look for a solution to this... I'll keep you all posted regarding this matter!

[gfx-prog]

You can use [investiny] I do have the same error message using it. In fine, anyone has a reliable alternative as it can be serious to build a download script given this restriction. Anyone got answer from investing; com ?

[romanminkoff]

You can use [investiny] I do have the same error message using it. In fine, anyone has a reliable alternative as it can be serious to build a download script given this restriction. Anyone got answer from investing; com ?

There is still yfinance around. Before investpy I was developing X-way scrapping/getting data from different sources (using timeouts to avoid potential bans for API overuse, and DB for caching results locally), and cross checking that data to make sure it's correct... need to resume that project.

[gfx-prog]

haven't tried scrapping yet. May be the inly safe way ...

[gfx-prog]

anyone having a hint and/or a function that can download using url such as ... 'https://www.investing.com/equities/ubisoft-historical-data' it seems that we can't pass dates in the url

[webdevzilla]

You can use [investiny] I do have the same error message using it. In fine, anyone has a reliable alternative as it can be serious to build a download script given this restriction. Anyone got answer from investing; com ?

You can use the workaround API that I built for this community: http://api.scraperlink.com/investpy/

For example, here's the sample API for stocks:

http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

[stagnoman]

You can use [investiny] I do have the same error message using it. In fine, anyone has a reliable alternative as it can be serious to build a download script given this restriction. Anyone got answer from investing; com ?

You can use the workaround API that I built for this community: http://api.scraperlink.com/investpy/

For example, here's the sample API for stocks:

http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

webdevzilla, well done!

[claudio2323]

hi, still having issues.

import investiny from investiny import historical_data data = historical_data(investing_id=6408, from_date="09/01/2022", to_date="10/01/2022")

i get "ConnectionError: Request to Investing.com API failed with error code: 403."**

[gfx-prog]

You can use [investiny] I do have the same error message using it. In fine, anyone has a reliable alternative as it can be serious to build a download script given this restriction. Anyone got answer from investing; com ?

You can use the workaround API that I built for this community: http://api.scraperlink.com/investpy/

For example, here's the sample API for stocks:

http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

Thks ! Great. how do you handle it back to python ?

url = "http://api.scraperlink.com/investpy/?email=myemail@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=11/14/2022&to_date=11/15/2022"
response = requests.get(url).content

can't manage to access to individual fields.

{"data":[{"direction_color":"greenFont","rowDate":"Nov 15, 2022","rowDateRaw":1668470400,"rowDateTimestamp":"2022-11-15T00:00:00Z","last_close":"194.42","last_open":"195.88","last_max":"200.82","last_min":"192.06","volume":"90.86M","volumeRaw":90859000,"change_precent":"1.82","last_closeRaw":"194.41999816894531","last_openRaw":"195.88000488281250","last_maxRaw":"200.82369995117188","last_minRaw":"192.05999755859375","change_precentRaw":1.8172303095892082},{"direction_color":"redFont","rowDate":"Nov 14, 2022","rowDateRaw":1668384000,"rowDateTimestamp":"2022-11-14T00:00:00Z","last_close":"190.95","last_open":"192.77","last_max":"195.73","last_min":"186.34","volume":"91.85M","volumeRaw":91846000,"change_precent":"-2.56","last_closeRaw":"190.94999694824219","last_openRaw":"192.77000427246094","last_maxRaw":"195.72999572753906","last_minRaw":"186.33999633789062","change_precentRaw":-2.561618738169708}],"summary":{"last_highest":"200.82","last_lowest":"186.34","last_difference":"14.48","last_avarage_total":"192.68","last_change_percent":"-0.79"}}

[stagnoman]

use json to parse and deserialize the response into an object. import json ...

https://json2csharp.com/code-converters/json-to-python

[gfx-prog]

use json to parse and deserialize the response into an object. import json ...

https://json2csharp.com/code-converters/json-to-python

Thks. For those who may be interested.

response = requests.get(url) resp_dict = response.json() print(type(resp_dict)) df = pd.DataFrame(resp_dict.get('data')) print(df.columns.values) print(df['last_close'], df['rowDate'])

[arrowstem]

You can use [investiny] I do have the same error message using it. In fine, anyone has a reliable alternative as it can be serious to build a download script given this restriction. Anyone got answer from investing; com ?

You can use the workaround API that I built for this community: http://api.scraperlink.com/investpy/

For example, here's the sample API for stocks:

'http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

Thanks, but can I use your tool to get the economic calendar? I tried but didn't succeed:

url = 'http://api.scraperlink.com/investpy/?email=my@email.com&type=economic_calendar&country=united%20states&from_date=21/11/2022&to_date=25/11/2022'

response:

b'Please specify &id=, &symbol= or &name='

I'm not sure if this response is from your tool (looks like it) or from investing.com. Now I'm lost because investiny isn't capable of getting the economic calendar.

[PolBarreiro]

Max retries exceeded with url investing.com with scraperlink Does anyone know how much time do we have to wait go call again?

[GregorioRozario]

Max retries exceeded with url investing.com with scraperlink Does anyone know how much time do we have to wait go call again?

I use 2,5 seconds. 2 seconds between each request may work: https://github.com/alvarobartt/investpy/issues/467#issuecomment-962017694

[webdevzilla]

Max retries exceeded with url investing.com with scraperlink Does anyone know how much time do we have to wait go call again?

It shouldn't be happening. Can you contact me the next time it happens?

I am guessing the issue was with Investing.com downtime.

[OldNewb7]

Hi everyone. I face the same issues. Investiny does not work either. I would like to pull a list of the biggest ETFs with additional information like (ISIN, AUM, Underlying Index). Does anyone can recommend a solution?