AlenkaF
commented
1 year ago Could you first verify the pyarrow version you are using (check pyarrow.__version__)? Then could you also inspect the columns of the parquet file, you can use pyarrow for that (table=pq.read_table('example.parquet')and then check the schema of the table object table.schema).
I tried couple of examples using map type and none of them error:
>>> import pyarrow as pa
>>> data = [[{'key': 'a', 'value': "1"}, {'key': 'b', 'value': "2"}], [{'key': 'c', 'value': "3"}]]
>>> map_type = pa.map_(pa.string(), pa.string())
>>> table = pa.table([pa.array(data, type=map_type)], names=["array_element"])
>>> table.schema
array_element: map<string, string>
child 0, entries: struct<key: string not null, value: string> not null
child 0, key: string not null
child 1, value: string
>>> table.to_pandas()
array_element
0 [(a, 1), (b, 2)]
1 [(c, 3)]
>>> table = pa.table([pa.array(data, type=pa.list_(map_type))], names=["array_element"])
>>> table.schema
array_element: list<item: map<string, string>>
child 0, item: map<string, string>
child 0, entries: struct<key: string not null, value: string> not null
child 0, key: string not null
child 1, value: string
>>> table.to_pandas()
array_element
0 [[(key, a), (value, 1)], [(key, b), (value, 2)]]
1 [[(key, c), (value, 3)]]
>>> inner = pa.array(data, type=map_type)
>>> array = pa.MapArray.from_arrays([0, 2], ['a', 'b'], inner)
>>> table = pa.table({'array_element': array})
>>> table.schema
array_element: map<string, map<string, string>>
child 0, entries: struct<key: string not null, value: map<string, string>> not null
child 0, key: string not null
child 1, value: map<string, string>
child 0, entries: struct<key: string not null, value: string> not null
child 0, key: string not null
child 1, value: string
>>> table.to_pandas()
array_element
0 [(a, [('a', '1'), ('b', '2')]), (b, [('c', '3'...
PCClimate
Getting the following error when trying to pull in data from a parquet file, is this expected for the data structure, is there a workaround using Arrow?
ArrowNotImplementedError: Not implemented type for Arrow list to pandas: map<string, string ('array_element')>
Full error:
Component(s)
Python