search

jugaad-py / jugaad-data

Download live and historical data for Indian stock market
https://marketsetup.in/documentation/jugaad-data/
369 stars 147 forks source link

Historical, Live Data related API's not working #84

Open divyankm opened 4 days ago

divyankm commented 4 days ago

Issue description

The Historical, Live Stock and F&O Api's not working post 11 AM Yesterday. Response is coming empty.

Stock Quote

# Download stock data to pandas dataframe
from jugaad_data.nse import stock_df
df = stock_df(symbol="SBIN", from_date=date(2020,1,1),
            to_date=date(2020,1,30), series="EQ")
df

Error

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
    973         try:
--> 974             return complexjson.loads(self.text, **kwargs)
    975         except JSONDecodeError as e:

13 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
    976             # Catch JSON-related errors and raise as requests.JSONDecodeError
    977             # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
--> 978             raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
    979 
    980     @property

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Live Data, market Quote

from jugaad_data.nse import NSELive

n = NSELive()
status = n.market_status()

# Print the raw response text
print("Raw response:", status.text)  # Check if status has a .text attribute

Error:

--------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
    973         try:
--> 974             return complexjson.loads(self.text, **kwargs)
    975         except JSONDecodeError as e:

7 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
    976             # Catch JSON-related errors and raise as requests.JSONDecodeError
    977             # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
--> 978             raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
    979 
    980     @property

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Code

from jugaad_data.nse import NSELive
from pandas import json_normalize
n = NSELive()
quotes = n.stock_quote_fno("TCS")
quotes

ERROR:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/requests/models.py](https://localhost:8080/#) in json(self, **kwargs)
    973         try:
--> 974             return complexjson.loads(self.text, **kwargs)
    975         except JSONDecodeError as e:

7 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/requests/models.py](https://localhost:8080/#) in json(self, **kwargs)
    976             # Catch JSON-related errors and raise as requests.JSONDecodeError
    977             # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
--> 978             raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
    979 
    980     @property

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
bharathg017 commented 4 days ago

I am also facing the same issue. Request for guidance from experienced persons in this

divyankm commented 4 days ago

Seems like data is coming encoded format from api's from live.py resulting in error.

Encoded Api response:

Code

from datetime import datetime
from requests import Session

class NSELive:
    time_out = 5
    base_url = "https://www.nseindia.com/api"
    page_url = "https://www.nseindia.com/get-quotes/equity?symbol=LT"
    _routes = {
        "market_status": "/marketStatus",
        # Add other routes as necessary
    }

    def __init__(self):
        self.s = Session()
        h = {
            "Host": "www.nseindia.com",
            "Referer": "https://www.nseindia.com/get-quotes/equity?symbol=SBIN",
            "X-Requested-With": "XMLHttpRequest",
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate, br",
            "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
        }
        self.s.headers.update(h)
        self.s.get(self.page_url)  # Initial request to set cookies

    def get(self, route, payload={}):
        url = self.base_url + self._routes[route]
        r = self.s.get(url, params=payload)
        print("URL:", url)
        print("Response Status Code:", r.status_code)
        print("Raw Response:", r.text)

        try:
            return r.json()
        except Exception as e:
            print("Failed to decode JSON:", e)
            return None

    def market_status(self):
        return self.get("market_status", {})

# Test the NSELive class
if __name__ == "__main__":
    n = NSELive()
    market_status = n.market_status()
    print("Market Status:", market_status)

Output:

URL: https://www.nseindia.com/api/marketStatus
Response Status Code: 200
�ikh��O-l�Q��{�ڡΧ)��R��Հ�| �ీX>��W߶}R��g���(��M�S{�I�R������ބk�@�?h�e�C������1�7]��ѷ��Z��QZ��0��V_�&�2A`.�����r��OZ��,(�t��x�7��f�u��/_=>·4�A���B8�p!<���'�i`I���zAJ�����$E��i$����X�r�ve������9һTO:y� N�d@��V�b�z��˺��)O��4����e�<K�۳�
� +;�����t�iH��h���hƑ��뻧��
Failed to decode JSON: Expecting value: line 1 column 2 (char 1)
Market Status: None

Normal Request using requests showing output:

Code

import requests
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0'}
# url = 'https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY'
url = 'https://www.nseindia.com/api/marketStatus'

with requests.session() as s:
    data = s.get(url, headers=headers).json()
    print(type(data))
    print(data)

Output:

<class 'dict'>
{'marketState': [{'market': 'Capital Market', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024 11:04', 'index': 'NIFTY 50', 'last': 25068.55, 'variation': -59.400000000001455, 'percentChange': -0.24, 'marketStatusMessage': 'Normal Market is Open'}, {'market': 'Currency', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open'}, {'market': 'Commodity', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open'}, {'market': 'Debt', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open'}, {'market': 'currencyfuture', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '84.1100', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open', 'expiryDate': '29-Oct-2024', 'underlying': 'USDINR', 'updated_time': '15-Oct-2024 11:02', 'tradeDateFormatted': '15-Oct-2024 11:02', 'slickclass': 'slick-item'}], 'marketcap': {'timeStamp': '14-Oct-2024', 'marketCapinTRDollars': 5.47, 'marketCapinLACCRRupees': 459.9931470995962, 'marketCapinCRRupees': 45999314.71, 'marketCapinCRRupeesFormatted': '45,999,314.71', 'marketCapinLACCRRupeesFormatted': '459.99', 'underlying': 'Market Cap'}, 'indicativenifty50': {'dateTime': '14-Oct-2024 15:30', 'indicativeTime': None, 'indexName': 'NIFTY 50', 'indexLast': None, 'indexPercChange': None, 'indexTimeVal': None, 'closingValue': 25127.95, 'finalClosingValue': 25127.95, 'change': 163.71, 'perChange': 0.66, 'status': 'CLOSE'}}
aman15012 commented 4 days ago

Just try replacing the header dictionary in jugaad_data/nse/history.py with the header you have mentioned. It works for me.

divyankm commented 4 days ago

Earlier till Yesterday, with existing headers, data was coming in original format for all cloud providers wherever the api is running. Now data is coming in encoded format for original header , due to which api's are failing.

Existing Header:

        h = {
            "Host": "www.nseindia.com",
            "Referer": "https://www.nseindia.com/get-quotes/equity?symbol=SBIN",
            "X-Requested-With": "XMLHttpRequest",
            "pragma": "no-cache",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate, br",
            "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            }

Below Header: Only working in Local Machine:

The issue with this header: headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0'} is API request are getting blocked by NSE as its giving IP of Cloud Providers, NSE Blocks all IP's of Cloud Providers (AWS, GCP, Azure).

gopikrishnan17 commented 4 days ago

Facing the same issue, please guide in resolving the same

k-prabin commented 3 days ago

How to solve this. Please guide

amritjain123 commented 3 days ago

Hi All, I resolved this by navigating to live.py file in installation directory and locating live.py file.(use pip show jugaad-data) In that file locate header dictoniary by 'h' name. and replace "Accept-Encoding": "gzip, deflate, br" with this one "Accept-Encoding": "gzip, deflate", It starts running for me.

I hope this helps temporarily until fix is released.

sahvinay commented 3 days ago

Hi amritjain123, Great!!! Thanks for temp solution. Could you please elaborate how removing "br" for Accept-Encoding solve this issue? I'm still in learning phase, just curious.

To all, Do you think this api is loophole for NSE, will the NSE api will get remove sooner or later? as we know the api is not free by NSE, also NSE endpoint contain api in it, making me doubt.

Do we need to limit API access per second to avoid getting blocked?

rmantena commented 3 days ago

Thanks for the workaround @amritjain123! This worked for me.

gopikrishnan17 commented 2 days ago

Thanks @amritjain123 , the fix works!!