search

BennyThadikaran / eod2

A fully automated script to download and update NSE EOD Historical stock, index and delivery data with added features
GNU General Public License v3.0
57 stars 18 forks source link

Data Update Error #59

Closed rohit1409 closed 10 months ago

rohit1409 commented 10 months ago

Hi Benny,

Firstly, this is amazing !, I have never seen something prepared so neatly. Kudos !

I tried this few days back and it ran well, however today I tried updating the data and encountered below error, couldn't figure out what's missing as internet connection is running well. I can download other stuff manually. Can you please suggest.

D:\eod2-main\src>init.py Downloading Files Download error. Try again later: ChunkedEncodingError(ProtocolError('Connection broken: IncompleteRead(271 bytes read, 138 more expected)', IncompleteRead(271 bytes read, 138 more expected)))

BennyThadikaran commented 10 months ago

@rohit1409 This error is because NSE has not updated a report - Full Bhavcopy and Security Deliverable data (sec_bhavdata_full_26102023.csv). See NSE daily reports

NSE sometimes delays its reports by a few hours. EOD2 will work just fine once the report becomes available. I encountered this error yesterday. At the time of this reply the report is still not updated. I would just retry on Monday after 7pm.

rohit1409 commented 10 months ago

Thanks Benny, appreciate your prompt reply, happy coding :) !

BennyThadikaran commented 10 months ago

The error itself is quite ugly. :smiley_cat: Does not leave a good impression for new users.

When a report is unavailable, NSE server sends a HTML/text response and ends the connection. Since we are downloading in parts/chunks, after the first chunk is received, the connection is closed resulting in this error.

Thanks for reporting this. I'll try and catch this error early, so next time a more friendly message is displayed.

rohit1409 commented 10 months ago

Most welcome... and I agree.. especially if someone not accustomed to NSE ways of working :)

BennyThadikaran commented 10 months ago

I don't think the delivery report for friday is coming. Still I will wait till evening. If today's report is available, friday delivery values will have to be set to 0 to continue.

I will put a mechanism is place so no manual intervention is required in future.

BennyThadikaran commented 10 months ago

@rohit1409

NSE has updated its Friday report. You can go ahead and sync EOD2.

The error has been corrected by checking the content-type of the response. If its a text/html we raise a runtime error with a friendly message. error

I made this change in NseIndiaApi with this commit. EOD2 will start using this package for all NSE related functions.

As for the mechanism to prevent this breakdown entirely:

  1. If delivery report if not available, create a entry for the day with NaN values. (Using 0 will cause averages to be skewed)
  2. Create a log file with the date for which delivery file was not available.
  3. On every sync, check for this log file. If it exists, retry downloading the delivery file for that date.
  4. Once download was successful, update the delivery info and delete the log file.
  5. If file not available after 2 days on sync delete the log file.

This is just my initial thought. Since there is no urgency anymore, i will take a few days to roll this update. You can leave this issue open for the time being.

rohit1409 commented 10 months ago

Hi Benny, Steps sounds fine, just a recommendation on your fifth point to have 5 working days instead of 2, why because normally eod/positional trading is analyzed weekly, and like in current example, by next friday if records are retrieved then analysis remain strong, else can be left blank.

rohit1409 commented 10 months ago

And yes, please take your time, will look forward to the updates in coming days.

BennyThadikaran commented 10 months ago

Thanks for that tip. Will incorporate it