alvarobartt / investpy

Financial Data Extraction from Investing.com with Python
https://investpy.readthedocs.io/
MIT License
1.59k stars 374 forks source link

I got a way to access investing without cloudflare u can try it with requests library #610

Closed ramakrishnamekala129 closed 1 year ago

ramakrishnamekala129 commented 1 year ago

import requests from requests.structures import CaseInsensitiveDict

url = "https://tvc4.investing.com/b12dcd570c91a748a3b7dd4a7ee79167/1664515691/56/56/23/history?symbol=101810&resolution=D&from=1633411692&to=1664515752"

headers = CaseInsensitiveDict() headers["authority"] = "tvc4.investing.com" headers["accept"] = "/" headers["accept-language"] = "en-US,en;q=0.9" headers["content-type"] = "text/plain" headers["origin"] = "https://tvc-invdn-com.investing.com" headers["referer"] = "https://tvc-invdn-com.investing.com/" headers["sec-fetch-dest"] = "empty" headers["sec-fetch-mode"] = "cors" headers["sec-fetch-site"] = "same-site" headers["user-agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36"

resp = requests.get(url, headers=headers)

print(resp.status_code) resp.json()

ramakrishnamekala129 commented 1 year ago

in this b12dcd570c91a748a3b7dd4a7ee79167 is just like apitoken once u get it then u access all historical data

ramakrishnamekala129 commented 1 year ago

why requests library doesnt work because of http2or3 version in investing protocol thats why we can use CaseInsensitiveDict

ramakrishnamekala129 commented 1 year ago

api key generated from md5 format

alvarobartt commented 1 year ago

Hi, @ramakrishnamekala129 can you elaborate a little bit more on what those parameters in the URL mean? Thanks :hugs:

ramakrishnamekala129 commented 1 year ago

Hi, @ramakrishnamekala129 can you elaborate a little bit more on what those parameters in the URL mean? Thanks :hugs:

  • https://tvc4.investing.com -> API Host
  • b12dcd570c91a748a3b7dd4a7ee79167 -> API Token? Where did you get this from?
  • 1664515691 -> current time
  • 56 -> Fized parameter for all
  • 56 -> Fixed parameter for all
  • 23 -> Fixed parameter for all
  • history -> HTTP GET Endpoint
  • symbol=101810 -> Investing.com symbol parameter
  • resolution=D -> Investing.com interval parameter (I assume this can be D, W, or M)
  • from=1633411692 -> Investing.com from_date as timestamp parameter
  • to=1664515752 -> Investing.com to_date as timestamp parameter

b12dcd570c91a748a3b7dd4a7ee79167 is md5 encode it will work for all

ramakrishnamekala129 commented 1 year ago

U can generate any random md5 code for api

alvarobartt commented 1 year ago

OK thanks @ramakrishnamekala129 I'll test your proposal and if it works I'll push some changes to the patch branch 😄

alvarobartt commented 1 year ago

I've just tested it and it doesn't work consistently, it works for some requests, but then it stops working... Same as with api.investing.com, it works the first times then you get instantly blocked :cry:

ramakrishnamekala129 commented 1 year ago

I've just tested it and it doesn't work consistently, it works for some requests, but then it stops working... Same as with api.investing.com, it works the first times then you get instantly blocked 😢

can u share blocking msg

ramakrishnamekala129 commented 1 year ago

it is not blocking me while True: url = "https://tvc4.investing.com/0e818888003657176127862245791911/1664515691/56/56/23/history?symbol=101810&resolution=D&from=1633411692&to=1664515752"

headers = CaseInsensitiveDict()
headers["authority"] = "tvc4.investing.com"
headers["accept"] = "*/*"
headers["accept-language"] = "en-US,en;q=0.9"
headers["content-type"] = "text/plain"
headers["origin"] = "https://tvc-invdn-com.investing.com"
headers["referer"] = "https://tvc-invdn-com.investing.com/"
headers["sec-fetch-dest"] = "empty"
headers["sec-fetch-mode"] = "cors"
headers["sec-fetch-site"] = "same-site"
headers["user-agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36"

resp = requests.get(url, headers=headers)

print(resp.status_code)
time.sleep(1)
if resp.status_code==200:
    print(resp.json())
ramakrishnamekala129 commented 1 year ago

https://tvc4.investing.com/4c89d7a6b632c7b26e8cfcb918bd00e8/1664516207/56/56/23/search?limit=30&query=USD&type=&exchange=

Search function

alvarobartt commented 1 year ago

Ok, you're right! The error just appears when I try to replace the current time value you mentioned with the current timestamp value, otherwise, it works! I'll probably release something with this patch in the meantime, thanks :hugs:

alvarobartt commented 1 year ago

I've just released investiny with a function to retrieve historical_data with your proposal @ramakrishnamekala129, check it at https://github.com/alvarobartt/investiny :hugs:

elricStormBringer commented 1 year ago

it is not blocking me while True: url = "https://tvc4.investing.com/0e818888003657176127862245791911/1664515691/56/56/23/history?symbol=101810&resolution=D&from=1633411692&to=1664515752"

headers = CaseInsensitiveDict()
headers["authority"] = "tvc4.investing.com"
headers["accept"] = "*/*"
headers["accept-language"] = "en-US,en;q=0.9"
headers["content-type"] = "text/plain"
headers["origin"] = "https://tvc-invdn-com.investing.com"
headers["referer"] = "https://tvc-invdn-com.investing.com/"
headers["sec-fetch-dest"] = "empty"
headers["sec-fetch-mode"] = "cors"
headers["sec-fetch-site"] = "same-site"
headers["user-agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36"

resp = requests.get(url, headers=headers)

print(resp.status_code)
time.sleep(1)
if resp.status_code==200:
    print(resp.json())

@ramakrishnamekala129 , I used this snippet, and unfortunately, it just loos with a 403 error code. This is as of 17/11/2022, United Kingdom location.

ramakrishnamekala129 commented 1 year ago

Because we used this api that why investing.com close this backend api

KapteinBabbalas commented 1 year ago

Because we used this api that why investing.com close this backend api

I dont think that is the case. If you open the url in any browser it works perfectly. If you just do a get in postman for that link you see that it return html with a form and a script with a function in there that does some magic in the window. I'm not that clued up with front end but maybe have a look to see if you can take that raw response and do something with it.