uwdata / arquero

Query processing and transformation of array-backed data tables.
https://idl.uw.edu/arquero
BSD 3-Clause "New" or "Revised" License
1.22k stars 64 forks source link

fromArrow -> Unrecognized type: "undefined" #334

Open carlopi opened 11 months ago

carlopi commented 11 months ago

Hi!

I was updating the benchmarks for @duckdb-wasm, and somehow in our benchmark infrastructure arquero support has somehow broken, either while bumping arrow or arquero itself.

Failure seems to happening in an line like:

this.tables['lineitem'] = aq.fromArrow(lineitem);

(here: https://github.com/duckdb/duckdb-wasm/blob/master/packages/benchmarks/src/system/arquero_benchmarks.ts#L46) with stack trace like:

Error: Unrecognized type: "undefined"
    at typeFromJSON (/node_modules/apache-arrow/srcipc/metadata/json.ts:205:11)
    at Function.fieldFromJSON (/node_modules/apache-arrow/srcipc/metadata/json.ts:109:16)
    at <anonymous> (/node_modules/apache-arrow/srcipc/metadata/json.ts:59:76)
    at Array.map (<anonymous>)
    at schemaFieldsFromJSON (/node_modules/apache-arrow/srcipc/metadata/json.ts:59:54)
    at Function.schemaFromJSON (/node_modules/apache-arrow/srcipc/metadata/json.ts:34:9)
    at Message._createHeader (/node_modules/apache-arrow/srcipc/metadata/message.ts:216:54)
    at Message.header (/node_modules/apache-arrow/srcipc/metadata/message.ts:131:35)
    at RecordBatchJSONReaderImpl.next (/node_modules/apache-arrow/srcipc/reader.ts:430:36)
    at RecordBatchStreamReader.readAll (/node_modules/apache-arrow/srcipc/reader.ts:209:35)

and seems similar to this error to be found here: https://observablehq.com/@jheer/arquero-to-arrow-serialization

Writing to signal this, not particular urgent on our side, but would be nice to have it working properly.

Thanks, let me know if any more informations could be useful.