E_ASSERTION_FAILURE Invalid dtype 'UNKNOWN' in visit dim

[philsv]

Describe the bug

I have the following error arcticdb_ext.exceptions.InternalException: E_ASSERTION_FAILURE Invalid dtype 'UNKNOWN' in visit dim. Reverting back to older version brings no resolution to the issue. I have tried to delete the symbol on Exeception and write it again to db and still get a arcticdb_ext.exceptions.InternalException on another write attempt.

Though I have no problem executing the code locally on my windows machine or on WSL2, but it does not work in my bitbucket pipeline.

bitbucket pipeline stdout:

2024-06-18 11:04:24 - df.index[0]: 2002-11-12 00:00:00

2024-06-18 11:04:24 - df.index[-1]: 2024-06-17 00:00:00

2024-06-18 11:04:24 - df.columns: Index(['ICE BofAML MOVE Index'], dtype='object')

2024-06-18 11:04:24 - len(df): 5352

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 5352 entries, 2024-06-17 to 2002-11-12
Data columns (total 1 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   ICE BofAML MOVE Index  5352 non-null   float64
dtypes: float64(1)
memory usage: 83.6 KB

2024-06-18 11:04:24 - df.head():             ICE BofAML MOVE Index

Date                             

2002-11-12                 120.21

2002-11-13                 122.88

2002-11-14                 130.29

2002-11-15                 132.35

2002-11-18                 129.14

[2024-06-18 11:04:24.613] [arcticdb] [error] E_ASSERTION_FAILURE Invalid dtype 'UNKNOWN' in visit dim

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "/opt/atlassian/pipelines/agent/build/src/modules/macro_dashboard.py", line 168, in <module>

    ac.write_to_db("refinitiv", "ice_bofaml_move_index", df)

  File "/opt/atlassian/pipelines/agent/build/src/arctic/db.py", line 262, in write_to_db

    write_df = library.write(

               ^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/arcticdb/version_store/library.py", line 425, in write

    return self._nvs.write(

           ^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/arcticdb/version_store/_store.py", line 574, in write

    vit = self.version_store.write_versioned_dataframe(

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

arcticdb_ext.exceptions.InternalException: E_ASSERTION_FAILURE Invalid dtype 'UNKNOWN' in visit dim

Another try (haven't changed any types):

[2024-06-18 11:56:31.998] [arcticdb] [warning] Appending a non-timeseries-indexed data to a timeseries symbol is highly likely to cause corruption/unexpected behaviour.

2024-06-18 11:56:32 - Append failed for DataFrame united_states_small_large_cap_indicies in the database refinitiv.

[2024-06-18 11:57:11.939] [arcticdb] [error] E_ASSERTION_FAILURE Invalid dtype 5:1 - 'UNKNOWN' in visit dim

Traceback (most recent call last):

  File "/opt/atlassian/pipelines/agent/build/src/arctic/db.py", line 226, in write_to_db

    append_df = library.append(symbol, remain_df)

                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/arcticdb/version_store/library.py", line 733, in append

    return self._nvs.append(

           ^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/arcticdb/version_store/_store.py", line 708, in append

    vit = self.version_store.append(

          ^^^^^^^^^^^^^^^^^^^^^^^^^^

arcticdb_ext.version_store.StreamDescriptorMismatch: The columns (names and types) in the argument are not identical to that of the existing version: APPEND 

(Showing only the mismatch. Full col list saved in the `last_mismatch_msg` attribute of the lib instance.

'-' marks columns missing from the argument, '+' for unexpected.)

-FD<name=Date, type=TD<type=NANOSECONDS_UTC64, dim=0>>

-FD<name=Wilshire US Small-Cap, type=TD<type=FLOAT64, dim=0>>

-FD<name=Wilshire US Large-Cap, type=TD<type=FLOAT64, dim=0>>

-FD<name=iShares Russell 2000 ETF, type=TD<type=FLOAT64, dim=0>>

-FD<name=SPDR Dow Jones Industrial Average ETF Trust, type=TD<type=FLOAT64, dim=0>>

+FD<name=Date, type=TD<type=UNKNOWN, dim=0>>

+FD<name=Wilshire US Small-Cap, type=TD<type=UNKNOWN, dim=0>>

+FD<name=Wilshire US Large-Cap, type=TD<type=UNKNOWN, dim=0>>

+FD<name=iShares Russell 2000 ETF, type=TD<type=UNKNOWN, dim=0>>

+FD<name=SPDR Dow Jones Industrial Average ETF Trust, type=TD<type=UNKNOWN, dim=0>>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/opt/atlassian/pipelines/agent/build/src/arctic/db.py", line 230, in write_to_db

    append_df = library.write(

                ^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/arcticdb/version_store/library.py", line 455, in write

    return self._nvs.write(

           ^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.11/site-packages/arcticdb/version_store/_store.py", line 587, in write

    vit = self.version_store.write_versioned_dataframe(

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

arcticdb_ext.exceptions.InternalException: E_ASSERTION_FAILURE Invalid dtype 5:1 - 'UNKNOWN' in visit dim

Steps/Code to Reproduce

image: python:3.11.3-slim-bullseye

macro_dashboard:
  - step:
      name: "Update macro dashboard"
      caches:
        - pip
      script:
          - apt-get update && apt-get install -y --no-install-recommends software-properties-common
          - export INSTALL_ON_LINUX=1
          - pip install --upgrade pip
          - pip3 install arcticdb==4.4.2 backoff pandas numpy python-dateutil python-dotenv pydantic-settings statsmodels ta xlwings refinitiv-dataplatform eikon
          - python3.11 -m src.modules.macro_dashboard

from arcticdb import Arctic
...
ac = self.__init_db()
library = ac[lib]

# On either of them I get the error message
append_df = library.append(symbol, remain_df)
write_df = library.write(symbol, df)

Expected Results

Does not throw an "UNKNOWN"

OS, Python Version and ArcticDB Version

Python 3.11.3 | Docker Image: python:3.11.3-slim-bullseye using bitbucket-pipelines.yml | OS: Debian-11 ArcticDB: 4.4.2 - 4.0.0

Backend storage used

AWS S3

Additional Context

No response

[jankleinsorge]

I can confirm this bug coming up for Windows 10, Windows 11 and WSL2.

It seems like a race condition of some sort where there might be a use after free. It is not reliable reproducible.

My test case would be just:

adb = arcticdb.Arctic("lmdb://test.db")
if adb.has_library("test"):
    adb.delete_library("test")
alib = adb.create_library("test")

df_1 = pd.DataFrame({'column': [1.0, 2.0, 3.0]})
#df_1 = pd.DataFrame({'column': [1, 2, 3]})
payload_1 = arcticdb.WritePayload("symbol_1", df_1, metadata={'the': 'metadata'})
alib.write("symbol_1", df_1)

Depending on the choice for 'df_1' errors are different. Assert like above for float but a floating point exception for int.

Float

File "/<...>/trader/.venv/lib/python3.11/site-packages/arcticdb/version_store/library.py", line 455, in write return self._nvs.write( ^^^^^^^^^^^^^^^^ File "/<...>/trader/.venv/lib/python3.11/site-packages/arcticdb/version_store/_store.py", line 587, in write vit = self.version_store.write_versioned_dataframe( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

arcticdb_ext.exceptions.InternalException: E_ASSERTION_FAILURE Invalid dtype 3:1 - 'UNKNOWN' in visit dim

Int

Floating point exception

Modules: deps.txt

Maybe valgrind or ASAN can provide indications?

[jankleinsorge]

Debug build of 4.4.2 with gcc ASAN enabled without issues.

Probably unrelated but running under valgrind yields:

Traceback (most recent call last): File "/<...>/trader/broken.py", line 30, in exit(main()) ^^^^^^ File "/<...>/trader/broken.py", line 14, in main adb.delete_library("test") File "/<...>/trader/venv/lib/python3.11/site-packages/arcticdb/arctic.py", line 232, in delete_library lib = self[name]

  File "/<...>/trader/venv/lib/python3.11/site-packages/arcticdb/arctic.py", line 99, in __getitem__
    self._library_manager.get_library(lib_mgr_name, storage_override),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
arcticdb_ext.exceptions.InternalException: lmdb::runtime_error(mdb_env_open: Invalid argument)

[philsv]

@jankleinsorge anything I could do to prevent the E_ASSERTION_FAILURE within my pipeline running on debian 11.9?

[jankleinsorge]

@philsv I am just some random user. I guess the next best thing is downgrade and see how that turns out.

[jankleinsorge]

Btw, a release (default) built with gcc12 fails with:

ArcticDB/cpp/arcticdb/pipeline/frame_slice.hpp:70:8: error: ‘(arcticdb::pipelines::FrameSlice)((char*)& + offsetof(folly::Try<std::tuple<arcticdb::stream::StreamSink::PartialKey, arcticdb::SegmentInMemory, arcticdb::pipelines::FrameSlice> >,folly::Try<std::tuple<arcticdb::stream::StreamSink::PartialKey, arcticdb::SegmentInMemory, arcticdb::pipelines::FrameSlice> >::.folly::detail::TryBase<std::tuple<arcticdb::stream::StreamSink::PartialKey, arcticdb::SegmentInMemory, arcticdb::pipelines::FrameSlice> >::)).arcticdb::pipelines::FrameSlice::row_range’ may be used uninitialized [-Werror=maybe-uninitialized]

[muhammadhamzasajjad]

@jankleinsorge, I was able to repro the issue you reported on 4.4.2. The issue does not happen on 4.4.3. Can you try that out?

[jankleinsorge]

@muhammadhamzasajjad I gave the 4.4.3 pip release a try and I did not encounter the issue since then. But then again I could not reliably reproduce it so I cannot tell whether it is has been really fixed. Let me know if I can lend a hand here.

[muhammadhamzasajjad]

Thanks @jankleinsorge. I was able to repro it on 4.4.2 every single time. And I didn't see it on 4.4.3. I'd recommend using 4.4.3 for now. We are going to investigate the issue further.

[muhammadhamzasajjad]

@philsv, I couldn't repro the issue you reported. It is hard to investigate it further without knowing what you are trying to append/write. Perhaps sharing the structure of your existing dataframe, remain_df and df (from you code) might help.

[philsv]

Hi @muhammadhamzasajjad,

I have switched to work with update on version 4.4.3 and the error was gone for me mentioned here: https://github.com/man-group/ArcticDB/issues/1627

I will test append again on 4.4.3 if the error still persists there, and let you know.

[philsv]

Heres how my code looks like:

...
existing_df = library.read(symbol).data

# Append the new data to the existing df
append_df = pd.concat([remaining_df, existing_df])
append_df = append_df[~append_df.index.duplicated(keep="last")]

if not append_df.index.is_monotonic_increasing:
    append_df = append_df.sort_index(ascending=True)

try:
  result = library.append(
      symbol,
      append_df,
      prune_previous_versions=prune_previous_versions,
  )
except Exception as e:
  return self.rewrite_to_db(lib, symbol, df, library)  # deletes the symbol and rewrites it to the database
...

this is how my remaining_df looks like:

            CBOE Volatility Index
2024-08-01                   13.3

remaining_df.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 1 entries, 2024-08-01 to 2024-08-01
Data columns (total 1 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   CBOE Volatility Index  1 non-null      float64
dtypes: float64(1)
memory usage: 16.0 bytes

This is how my test went:

+ python3.11 -m tests.arctic.test_db
2024-07-02 23:16:57 - Dataframe cboe_volatility_index already up to date in the database test: 1990-01-02 00:00:00 - 2024-07-01 00:00:00
2024-07-02 23:17:03 - cboe_volatility_index existing df (before append) info:
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 8702 entries, 1990-01-02 to 2024-07-01
Data columns (total 1 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   CBOE Volatility Index  8702 non-null   float64
dtypes: float64(1)
memory usage: 136.0 KB
2024-07-02 23:17:03 - cboe_volatility_index processed df (after append) info:
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 8703 entries, 1990-01-02 to 2024-08-01

Data columns (total 1 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   CBOE Volatility Index  8703 non-null   float64
dtypes: float64(1)

memory usage: 136.0 KB
20240702 23:17:09.266602 15696 E arcticdb.root | E_ASSERTION_FAILURE Can't append dataframe with start index 1990-01-02 00:00:00.0 to existing sequence ending at 2024-07-01 00:00:00.0
2024-07-02 23:17:10 - Rewritten dataframe cboe_volatility_index to the database test: 1990-01-02 00:00:00 - 2024-08-01 00:00:00

df.tail(5)

            CBOE Volatility Index
2024-06-26                  12.55
2024-06-27                  12.24
2024-06-28                  12.44
2024-07-01                  12.22
2024-08-01                  12.30

So with 4.4.3 theres still some issue I don't really understand but it was hitting my try block now in the function and I was able to rewrite the data afterwards.

For now I will stick to library.update as I encounter no problems with that.

[alexowens90]

This seems like the same underlying issue as #1589 (sporadic, spurious rejections of legitimate appends).