search

RelationalAI / rai-sdk-csharp

The RelationalAI Software Development Kit (SDK) for C#
Apache License 2.0
0 stars 1 forks source link

Exception when reading arrow stream (Value cannot be null. Parameter('name')) #25

Open NRHelmi opened 2 years ago

NRHelmi commented 2 years ago

Apache arrow csharp library seems to have an issue when reading specific arrow streams, I was able to reproduce this behavior using this query

value type MyType = Int, Int
def output = ^MyType[1, 2]

Exception

Unhandled exception: System.ArgumentNullException: Value cannot be null. (Parameter 'name')
   at Apache.Arrow.Field..ctor(String name, IArrowType dataType, Boolean nullable)
   at Apache.Arrow.Ipc.MessageSerializer.FieldFromFlatbuffer(Field flatbufField, DictionaryMemo& dictionaryMemo)
   at Apache.Arrow.Ipc.MessageSerializer.FieldFromFlatbuffer(Field flatbufField, DictionaryMemo& dictionaryMemo)
   at Apache.Arrow.Ipc.MessageSerializer.GetSchema(Schema schema, DictionaryMemo& dictionaryMemo)
   at Apache.Arrow.Ipc.ArrowStreamReaderImplementation.<ReadSchema>b__11_0(Memory`1 buff)
   at Apache.Arrow.Ipc.ArrowStreamReaderImplementation.ReadSchema()
   at Apache.Arrow.Ipc.ArrowStreamReaderImplementation.ReadRecordBatch()
   at Apache.Arrow.Ipc.ArrowStreamReaderImplementation.ReadNextRecordBatch()
   at Apache.Arrow.Ipc.ArrowStreamReader.ReadNextRecordBatch()

As far as I do understand the library is complaining about field name being null here, while I was able to read the same binary file using apache arrow for golang and python libraries I created an issue apache arrow side waiting for guidance/fix https://issues.apache.org/jira/browse/ARROW-17644

related: https://github.com/RelationalAI/rai-sdk-csharp/issues/21

cc: @bradlo @larf311 @NHDaly

NRHelmi commented 2 years ago

The query below run without issue

value type MyType = Int, String
def output = ^MyType[1, "foo"]

This makes me feel that probably we are missing something for value types with similar sub types :thinking:

NRHelmi commented 2 years ago

Related: https://github.com/RelationalAI/raicode/issues/10335

billscheidel-rai commented 1 year ago

Note: This issue has been migrated to https://relationalai.atlassian.net/browse/RAI-6829.

This link is only accessible to employees of RelationalAI.