Open AlJohri opened 1 year ago
I figured out the issue! It has to do with --json
flag:
❯ pqrs head -n1 output.parquet
{passage_id: "asdjfaslkdfasf", title: "adsfasdf", text: "asdfasdfadf", shard_id: 0, embedding: [1,2,3,4]}
❯ pqrs head -n1 --json output.parquet
Error: ParquetError(ArrowError("creating ListArrayReader with type FixedSizeList(Field { name: \"item\", data_type: Float32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }, 256) should be unreachable"))
hi there, ran into a new error today. I'm guessing it might have to do with the fact that the inner list field is called
item
instead ofelement
?EDIT: I tried changing the name from item to element via pyarrow's
use_compliant_nested_type=True
but that didn't fix the issue so it might be something else.