search

BennyThadikaran / eod2

A fully automated script to download and update NSE EOD Historical stock, index and delivery data with added features
GNU General Public License v3.0
58 stars 18 forks source link

Today Error Updating #184

Open cotton1234 opened 1 day ago

cotton1234 commented 1 day ago

Hello Benny,

Today getting error while updating.

(.venv) saurabhgarg@MacBook-Pro src % python3 init.py sh: color: command not found INFO: 20-09-2024 19:07 - main - Downloading Files INFO: 20-09-2024 19:07 - defs.defs - Starting Data Sync INFO: 20-09-2024 19:07 - defs.defs - EOD sync complete INFO: 20-09-2024 19:07 - defs.defs - ### Nifty PE at 23.98 ### INFO: 20-09-2024 19:07 - defs.defs - Index sync complete. INFO: 20-09-2024 19:07 - defs.defs - Making adjustments for splits and bonus CRITICAL: 20-09-2024 19:07 - root - Adjustment Error - Context PHOENIXLTD - Bonus - 20-Sep-2024 ERROR: 20-09-2024 19:07 - main - Error while making adjustments. All adjustments have been discarded. Traceback (most recent call last): File "/Users/saurabhgarg/shivshakti/eod2/src/init.py", line 125, in defs.adjustNseStocks() File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 833, in adjustNseStocks raise e File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 813, in adjustNseStocks commit = makeAdjustment( ^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 656, in makeAdjustment df = pd.read_csv(file, index_col="Date", parse_dates=["Date"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv return _read(filepath_or_buffer, kwds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 626, in _read return parser.read(nrows) ^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1923, in read ) = self._engine.read( # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 234, in read chunks = self._reader.read_low_memory(nrows) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "parsers.pyx", line 838, in pandas._libs.parsers.TextReader.read_low_memory File "parsers.pyx", line 905, in pandas._libs.parsers.TextReader._read_rows File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status File "parsers.pyx", line 2061, in pandas._libs.parsers.raise_parser_error pandas.errors.ParserError: Error tokenizing data. C error: Expected 9 fields in line 4312, saw 17

INFO: 20-09-2024 19:07 - defs.defs - Rolling back changes from 2024-09-20: /Users/saurabhgarg/shivshakti/eod2/src/eod2_data/daily INFO: 20-09-2024 19:07 - defs.defs - Rollback successful

Thanks Saurabh

BennyThadikaran commented 1 day ago

I had a slightly different error, a keyerror. Can you send me Phoenixltd.csv file.

I will have a look and let you know

Regards, Benny Thadikaran

rohit1409 commented 1 day ago

metoo same error :|

cotton1234 commented 1 day ago

I had a slightly different error, a keyerror. Can you send me Phoenixltd.csv file. I will have a look and let you know On 20 September 2024 13:41:49 UTC, Saurabh @.> wrote: Hello Benny, Today getting error while updating. (.venv) @. src % python3 init.py sh: color: command not found INFO: 20-09-2024 19:07 - main - Downloading Files INFO: 20-09-2024 19:07 - defs.defs - Starting Data Sync INFO: 20-09-2024 19:07 - defs.defs - EOD sync complete INFO: 20-09-2024 19:07 - defs.defs - ### Nifty PE at 23.98 ### INFO: 20-09-2024 19:07 - defs.defs - Index sync complete. INFO: 20-09-2024 19:07 - defs.defs - Making adjustments for splits and bonus CRITICAL: 20-09-2024 19:07 - root - Adjustment Error - Context PHOENIXLTD - Bonus - 20-Sep-2024 ERROR: 20-09-2024 19:07 - main - Error while making adjustments. All adjustments have been discarded. Traceback (most recent call last): File "/Users/saurabhgarg/shivshakti/eod2/src/init.py", line 125, in defs.adjustNseStocks() File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 833, in adjustNseStocks raise e File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 813, in adjustNseStocks commit = makeAdjustment( ^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 656, in makeAdjustment df = pd.read_csv(file, index_col="Date", parse_dates=["Date"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv return _read(filepath_or_buffer, kwds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 626, in _read return parser.read(nrows) ^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1923, in read ) = self._engine.read( # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 234, in read chunks = self._reader.read_low_memory(nrows) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "parsers.pyx", line 838, in pandas._libs.parsers.TextReader.read_low_memory File "parsers.pyx", line 905, in pandas._libs.parsers.TextReader._read_rows File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status File "parsers.pyx", line 2061, in pandas._libs.parsers.raise_parser_error pandas.errors.ParserError: Error tokenizing data. C error: Expected 9 fields in line 4312, saw 17 INFO: 20-09-2024 19:07 - defs.defs - Rolling back changes from 2024-09-20: /Users/saurabhgarg/shivshakti/eod2/src/eod2_data/daily INFO: 20-09-2024 19:07 - defs.defs - Rollback successful Thanks Saurabh -- Reply to this email directly or view it on GitHub: #184 You are receiving this because you are subscribed to this thread. Message ID: @.***> Regards, Benny Thadikaran

.venv) saurabhgarg@MacBook-Pro daily % tail -10 phoenixltd.csv
2024-09-06,1827.45,1895.95,1795.6,1803.4,738915,77559.0,9.53,375385.0 2024-09-09,1817.6,1837.2,1731.3,1748.9,247675,37562.0,6.59,152866.0 2024-09-10,1762.45,1806.0,1762.0,1782.9,196083,40000.0,4.9,98060.0 2024-09-11,1793.5,1804.5,1715.0,1719.85,191423,28819.0,6.64,122336.0 2024-09-12,1735.0,1746.45,1672.0,1682.1,238262,29898.0,7.97,153350.0 2024-09-13,1697.0,1699.9,1645.2,1671.4,543217,60139.0,9.03,352718.0 2024-09-16,1689.45,1775.0,1647.8,1766.1,358166,42577.0,8.41,168362.0 2024-09-17,1777.45,1789.7,1739.6,1746.35,251446,39562.0,6.36,126985.0 2024-09-18,1759.9,1782.5,1680.6,1722.85,228944,40642.0,5.63,109858.0 2024-09-19,1745.0,1777.5,1686.85,1765.9,263004,49066.0,5.36,122032.02024-09-20,1820.0,1950.0,1722.05,1844.05,1296237,110874,11.69,5117482024-09-20,1820.0,1950.0,1722.05,1844.05,1296237,110874,11.69,511748 (.venv) saurabhgarg@MacBook-Pro daily %

You are correct the issue is in this file.

Thanks Saurabh

cotton1234 commented 1 day ago

phoenixltd.csv Attached is the file.

cotton1234 commented 1 day ago

Also there was issue on 18 also. Uploaing error.log for your review.

Thanks Saurabh

cotton1234 commented 1 day ago

error.log

cotton1234 commented 1 day ago

Benny,

I checked other files also (HDFCBANK/Reliance), seems all are corrupt and need to remove 18 and 19th data and resync it.

Thanks Saurabh

cotton1234 commented 1 day ago

.venv) saurabhgarg@MacBook-Pro defs % python3 diagnostic.py

File or Pandas exceptions SETFNN50.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2264, saw 33\n') INTLCONV.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 748, saw 33\n') DIAMONDYD.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 1722, saw 33\n') AEROFLEX.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 263, saw 33\n') BHARTIARTL.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 4524, saw 33\n') SBIETFPB.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 970, saw 33\n') INFINIUM_SME.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 353, saw 33\n') BAWEJA_SME.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 156, saw 33\n') HDFCNEXT50.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 523, saw 33\n') PRIVISCL.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2320, saw 33\n') GABRIEL.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 4814, saw 33\n') AVROIND.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 853, saw 33\n') 3PLAND.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3255, saw 33\n') ZENTEC.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2345, saw 33\n') TATASTEEL.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 4690, saw 33\n') SENCO.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 296, saw 33\n') OMFURN_SME.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 571, saw 33\n') BLBLIMITED.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 5436, saw 33\n') RELAXO.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3280, saw 33\n') LIQUIDADD.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 117, saw 33\n') CELLO.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 219, saw 33\n') HDFCVALUE.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 490, saw 33\n') INFOMEDIA.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3483, saw 33\n') LUPIN.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 5722, saw 33\n') SHARIABEES.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3498, saw 33\n') JITFINFRA.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 1867, saw 33\n') BFINVEST.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3372, saw 33\n') NIFTY PHARMA.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2191, saw 33\n') NIFTY MIDSMALLCAP 400.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2095, saw 33\n') (.venv) saurabhgarg@MacBook-Pro defs %

BennyThadikaran commented 1 day ago

I have fixed a KeyError when verifying adjustment for bonus in UEL share. It was temporarily suspended, so today's data was not available. So trying to access today's Close results in the error.

Going forward this will simply throw a warning to the user like below. It means you need to check the charts to see if the adjustment is correct.

WARNING: 20-09-2024 23:40 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually.
BennyThadikaran commented 1 day ago

Now this error you mentioned - I feel like I've already dealt with this issue before. Can you confirm the EOD2 you're using with py init.py -v

Also if you open src/defs/defs.py and search for def updateNseSymbol. Scroll down a few lines below and you should see the below line. Are you missing the \n at the end? That has to be the cause of the issue.

text += bytes(
    f"{dates.pandasDt},{open},{high},{low},{close},{volume},{trdCnt},{avgTrdCnt},{dq}\n",
    encoding="utf-8",
)

If you cant find this, just rename defs.py to defs.txt and drop the file to me here.

Just make sure you have the latest version by running git pull in the EOD2 folder.

To resolve the corrupted data. Open eod2_data folder in your terminal and run the following commands.

git checkout main

git reset --hard HEAD

git clean -f

Now run init.py and let me know if the issue persists.

I will push the other update tomorrow morning.

cotton1234 commented 23 hours ago

Now this error you mentioned - I feel like I've already dealt with this issue before. Can you confirm the EOD2 you're using with py init.py -v

Also if you open src/defs/defs.py and search for def updateNseSymbol. Scroll down a few lines below and you should see the below line. Are you missing the \n at the end? That has to be the cause of the issue.

text += bytes(
    f"{dates.pandasDt},{open},{high},{low},{close},{volume},{trdCnt},{avgTrdCnt},{dq}\n",
    encoding="utf-8",
)

If you cant find this, just rename defs.py to defs.txt and drop the file to me here.

Just make sure you have the latest version by running git pull in the EOD2 folder.

To resolve the corrupted data. Open eod2_data folder in your terminal and run the following commands.

git checkout main

git reset --hard HEAD

git clean -f

Now run init.py and let me know if the issue persists.

I will push the other update tomorrow morning.

Attached is the file. This code is there in the file I have. I keep updated the repo once a week.

defs_sk.txt

Thanks Saurabh

cotton1234 commented 19 hours ago

Benny,

I cloned the repo again and i see the same issue on 19 Sep file. Its not putting "/n" at the end of the file.

(.venv) saurabhgarg@MacBook-Pro eod2_data % sh test.sh daily/reliance.csv No newline at the end of the file.

If I remove the 19th entry it has a new line.

(.venv) saurabhgarg@MacBook-Pro eod2_data % sh test.sh daily/reliance.csv The file ends with a newline. (.venv) saurabhgarg@MacBook-Pro eod2_data %

Thanks Saurabh

BennyThadikaran commented 14 hours ago

I have gone through your error logs and the CSV file you gave me. From the logs, this error originated on 18th July 2024 (below is the relevant bits)

2024-07-18 20:12:30,872 - __main__ - ERROR - Error while making adjustments.
TypeError: cannot do positional indexing on DatetimeIndex with these indexers [slice(737, 739, None)] of type slice

2024-07-18 20:16:02,170 - __main__ - ERROR - Error while making adjustments.
pandas.errors.ParserError: Error tokenizing data. C error: Expected 9 fields in line 739, saw 17

The slice indicates duplicate indexes (dates) in one of the CSV files. It can happen, if an error occurred and the rollback itself failed ( So the row was never removed). Trying to sync again will result in the same row being added again.

I am unable to reproduce this error. I cloned the repo from scratch and ran init.py. Only error was the latest KeyError. Your error is likely originating from an older version of EOD2.

Here is what i will do:

I will post the update soon and let you know.

BennyThadikaran commented 9 hours ago

Update 7.0.3 has been released. My suggestion to delete your existing local repo and perform a fresh install of EOD2. (Backup your user.json and data folder if you have custom config or line data)

Below you can see I have added the EOD2 and NSE version and last update date before file execution.

WARNING: 2024-09-21 17:02:16,900 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually. - 2024-09-20 00:00:00+05:30 - EOD2 v7.0.2 - NSE v1.0.0 - 2024-09-18T00:00:00+05:30

I cloned the repo and made a successful sync

INFO: 21-09-2024 17:02 - __main__ - Downloading Files
INFO: 21-09-2024 17:02 - defs.defs - Starting Data Sync
INFO: 21-09-2024 17:02 - defs.defs - EOD sync complete
INFO: 21-09-2024 17:02 - defs.defs - ### Nifty PE at 23.64 ###
INFO: 21-09-2024 17:02 - defs.defs - Index sync complete.
INFO: 21-09-2024 17:02 - defs.defs - Making adjustments for splits and bonus
INFO: 21-09-2024 17:02 - defs.defs - NDL: face value split (sub-division) - from rs 10/- per share to re 1/- per share
INFO: 21-09-2024 17:02 - defs.defs - SAKSOFT: bonus 1:4
INFO: 21-09-2024 17:02 - __main__ - 19 Sep 2024: Done
----------------------------------------------------
INFO: 21-09-2024 17:02 - defs.defs - Updating NSE EQUITY actions
INFO: 21-09-2024 17:02 - defs.defs - Updating NSE SME actions
INFO: 21-09-2024 17:02 - __main__ - Downloading Files
INFO: 21-09-2024 17:02 - defs.defs - Starting Data Sync
INFO: 21-09-2024 17:02 - defs.defs - EOD sync complete
INFO: 21-09-2024 17:02 - defs.defs - ### Nifty PE at 23.98 ###
INFO: 21-09-2024 17:02 - defs.defs - Index sync complete.
INFO: 21-09-2024 17:02 - defs.defs - Making adjustments for splits and bonus
INFO: 21-09-2024 17:02 - defs.defs - UEL: bonus 1:4
INFO: 21-09-2024 17:02 - defs.defs - RITES: bonus 1:1
INFO: 21-09-2024 17:02 - defs.defs - PHOENIXLTD: bonus 1:1
INFO: 21-09-2024 17:02 - defs.defs - MINDTECK: bonus 1:4
INFO: 21-09-2024 17:02 - defs.defs - AXITA: bonus 1:3
WARNING: 21-09-2024 17:02 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually. - 2024-09-20 00:00:00+05:30
INFO: 21-09-2024 17:02 - defs.defs - SOLEX_sme: bonus 1:4
INFO: 21-09-2024 17:02 - __main__ - 20 Sep 2024: Done
----------------------------------------------------
INFO: 21-09-2024 17:02 - defs.defs - All Up To Date. Check again after 07:00PM for today's EOD data

The warning message on UEL can be ignored for now. I need to check the charts to verify the adjustment was correct.

Issues will Ambaniorgo and RPPL have already been fixed and published on eod2_data.

rohit1409 commented 9 hours ago

aye aye ben !

cotton1234 commented 9 hours ago

Update 7.0.3 has been released. My suggestion to delete your existing local repo and perform a fresh install of EOD2. (Backup your user.json and data folder if you have custom config or line data)

Below you can see I have added the EOD2 and NSE version and last update date before file execution.

WARNING: 2024-09-21 17:02:16,900 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually. - 2024-09-20 00:00:00+05:30 - EOD2 v7.0.2 - NSE v1.0.0 - 2024-09-18T00:00:00+05:30

I cloned the repo and made a successful sync

INFO: 21-09-2024 17:02 - __main__ - Downloading Files
INFO: 21-09-2024 17:02 - defs.defs - Starting Data Sync
INFO: 21-09-2024 17:02 - defs.defs - EOD sync complete
INFO: 21-09-2024 17:02 - defs.defs - ### Nifty PE at 23.64 ###
INFO: 21-09-2024 17:02 - defs.defs - Index sync complete.
INFO: 21-09-2024 17:02 - defs.defs - Making adjustments for splits and bonus
INFO: 21-09-2024 17:02 - defs.defs - NDL: face value split (sub-division) - from rs 10/- per share to re 1/- per share
INFO: 21-09-2024 17:02 - defs.defs - SAKSOFT: bonus 1:4
INFO: 21-09-2024 17:02 - __main__ - 19 Sep 2024: Done
----------------------------------------------------
INFO: 21-09-2024 17:02 - defs.defs - Updating NSE EQUITY actions
INFO: 21-09-2024 17:02 - defs.defs - Updating NSE SME actions
INFO: 21-09-2024 17:02 - __main__ - Downloading Files
INFO: 21-09-2024 17:02 - defs.defs - Starting Data Sync
INFO: 21-09-2024 17:02 - defs.defs - EOD sync complete
INFO: 21-09-2024 17:02 - defs.defs - ### Nifty PE at 23.98 ###
INFO: 21-09-2024 17:02 - defs.defs - Index sync complete.
INFO: 21-09-2024 17:02 - defs.defs - Making adjustments for splits and bonus
INFO: 21-09-2024 17:02 - defs.defs - UEL: bonus 1:4
INFO: 21-09-2024 17:02 - defs.defs - RITES: bonus 1:1
INFO: 21-09-2024 17:02 - defs.defs - PHOENIXLTD: bonus 1:1
INFO: 21-09-2024 17:02 - defs.defs - MINDTECK: bonus 1:4
INFO: 21-09-2024 17:02 - defs.defs - AXITA: bonus 1:3
WARNING: 21-09-2024 17:02 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually. - 2024-09-20 00:00:00+05:30
INFO: 21-09-2024 17:02 - defs.defs - SOLEX_sme: bonus 1:4
INFO: 21-09-2024 17:02 - __main__ - 20 Sep 2024: Done
----------------------------------------------------
INFO: 21-09-2024 17:02 - defs.defs - All Up To Date. Check again after 07:00PM for today's EOD data

The warning message on UEL can be ignored for now. I need to check the charts to verify the adjustment was correct.

Issues will Ambaniorgo and RPPL have already been fixed and published on eod2_data.

Done as instructed. Further processing the same.

Will update.

Thanks Saurabh

cotton1234 commented 8 hours ago

All set Benny. Thank you.