search

man-group / ArcticDB

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.
http://arcticdb.io
Other
1.48k stars 95 forks source link

Read invalid serialized key #1177

Open philsv opened 10 months ago

philsv commented 10 months ago

Describe the bug

Today I encounter this traceback error:

Traceback (most recent call last): File "", line 1, in File "C:\Users\user\anaconda3\envs\py11\Lib\site-packages\arcticdb\arctic.py", line 342, in list_libraries return self._library_manager.list_libraries() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ arcticdb_ext.exceptions.InternalException: E_ASSERTION_FAILURE Read invalid serialized key

I couldn't find any issue regard this InternalException.

Connecting to the s3 bucket is working but it errors out at ac.list_libraries() - trying to list my libaries.

For example: ac.has_library("equity") returns True so there are libraries in the db but I can't list them somehow

Any hint what might went wrong here?

Steps/Code to Reproduce

ac = Arctic(s3_bucket_uri)
ac.list_libraries()

Expected Results

It should print out the list of my libraries

OS, Python Version and ArcticDB Version

Python: 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:12:32) [MSC v.1929 64 bit (AMD64)] | OS: Windows-10-10.0.22621-SP0 | ArcticDB: 4.2.1

Backend storage used

AWS S3

Additional Context

No response

alexowens90 commented 10 months ago

Hi @philsv, I understand it might be hard to work out now, but do you know the names of all the libraries present in the bucket where the issue is arising? My suspicion is that library names containing certain characters could cause this error. If you have access to a tool like s5cmd then it should be fairly easy to work out the library names using ls.

philsv commented 9 months ago

Hi @alexowens90 thank you for your response.

On one of my libraries I frequently get this warning now when writting data to a symbol.

[2023-12-29 04:44:30.145] [arcticdb] [error] E_ASSERTION_FAILURE Cannot write string symbol name, existing symbols are numeric

[2023-12-29 04:44:30.462] [arcticdb] [warning] Ignoring error while trying to compact the symbol list: E_ASSERTION_FAILURE Cannot write string symbol name, existing symbols are numeric

Could this be related?

I found some symbols with special characters: 'ishares_20+_year_treasury_bond_etf' 's&p_500'

However removing the symbols with the special characters didn't resolve the above warning.

I don't think its because of the library per se as they are all written like e.g. "alternative_data" or "equity".
No numbers or special characters