Nested fields not readable by python lib polars

[oscar6echo]

I want to read/write parquet files in go to read/write them in python/polars.
It seems that the nested fields ([]int in my example) written by one lib cannot be read by the other. Then instead it returns empty lists.

I placed an issue with polars, but it concerns segmentio/parquet-go symmetrically.
See https://github.com/pola-rs/polars/issues/6428

I am surprised that parquet compability can be partial.
So I am looking for a way, if possible, to create the nested field in such a way it is understood by polars.

• Here is the parquet file creation
• Is there some other way to create the nested field that would increase compatility with other libs like polars ?
• Where in the source code is the nested field creation performed ?

[achille-roussel]

Hello @oscar6echo, thanks for reporting!

I looked at the issue you opened at https://github.com/pola-rs/polars/issues/6428, it appears that this is due to the underlying pyarrow library not supporting the DELTA_LENGTH_BYTE_ARRAY encoding, which is used by default in parquet-go.

...
  File "pyarrow/_dataset.pyx", line 332, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 2661, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Not yet implemented: DecodeArrow for DeltaLengthByteArrayDecoder.

I am surprised that parquet compability can be partial.

This is somewhat a double-edge sword with parquet, each column can have a different encoding, and clients can choose to support a subset of the parquet spec, which eventually can result in incompatibilities like the one you describe. parquet-go actually follows the spec which says that DELTA_LENGTH_BYTE_ARRAY should be the default encoding for byte array columns.

Is there some other way to create the nested field that would increase compatility with other libs like polars ?

All parquet clients must support the PLAIN encoding, which is less efficient but simpler to implement, so I would recommend in your case that you force the encoding in your schema, for example by adding the plain marker to the parquet tag of Go struct fields:

type Row struct {
    Idx    int       `parquet:"idx,plain"`
    Name   string    `parquet:"name,plain"`
    ...
}

With this approach, you will be trading off storage space for greater chance of compatibility between parquet clients.

Where in the source code is the nested field creation performed ?

This is a broad question as it depends on how the schema is constructed, but here are a few entry points:

• https://github.com/segmentio/parquet-go/blob/main/row.go#L402-L406
• https://github.com/segmentio/parquet-go/blob/main/column_buffer.go

I hope these answers are useful to you, let me know if you have any other question!

[oscar6echo]

@achille-roussel thx for the fast and comprehensive reply !

I tried to use the plain option to see if it would achieve compatibility. But then I realised that I in fact got mixed up in my latest comment.

The files produced by parquet-go and polars in this example are indeed incompatible - due to the nested field. But you gave me tips that helped me maybe pintpoint the difference.

1/ schema produced/read by parquet-go:

• code

type Row struct { Idx int parquet:"idx" Name string parquet:"name" Age int parquet:"age" Sex bool parquet:"sex" Weight float64 parquet:"weight" Time time.Time parquet:"time" Arr []int parquet:"array" } schema := parquet.SchemaOf(new(Row)) fmt.Println(schema)

+ result:
```sh
message Row {
    required int64 idx (INT(64,true));
    required binary name (STRING);
    required int64 age (INT(64,true));
    required boolean sex;
    required double weight;
    required int64 time (TIMESTAMP(isAdjustedToUTC=true,unit=NANOS));
    repeated int64 array (INT(64,true));
}

2/ schema produced/read by polars:

• code

  
  import datetime as dt
  from pathlib import Path
  import polars as pl
  import pyarrow.parquet as pq

now = dt.datetime.now()

s1 = pl.Series("idx", [0, 1], dtype=pl.Int64) s1 = pl.Series("name", ["Masterfog", "Armspice"], dtype=pl.Utf8) s2 = pl.Series("age", [22, 23], dtype=pl.Int64) s3 = pl.Series("sex", [True, False], dtype=pl.Boolean) s4 = pl.Series("weight", [51.2, 65.3], dtype=pl.Float64) s5 = pl.Series("time", [now, now], dtype=pl.Datetime) s6 = pl.Series("array", [[10, 20], [11, 22]], dtype=pl.List(pl.Int64)) df = pl.DataFrame([s1, s2, s3, s4, s5, s6]) path = Path("sample3.pqt") df.write_parquet(path)

h = pq.ParquetFile(path) print(h.schema)

+ result:

<pyarrow._parquet.ParquetSchema object at 0x7fd8fb70d540> required group field_id=-1 schema { optional binary field_id=-1 name (String); optional int64 field_id=-1 age; optional boolean field_id=-1 sex; optional double field_id=-1 weight; optional int64 field_id=-1 time (Timestamp(isAdjustedToUTC=false, timeUnit=microseconds, is_from_converted_type=false, force_set_converted_type=false)); optional group field_id=-1 array (List) { repeated group field_id=-1 list { optional int64 field_id=-1 item; } } }

The inspection of the schemas produced by each lib show the difference in parquet format. There is a `group` notion in the polars generated parquet that does not exist in parquet-go.:

// parquet-go repeated int64 array (INT(64,true));

vs.

// polars optional group field_id=-1 array (List) { repeated group field_id=-1 list { optional int64 field_id=-1 item; }

So if I could create the polars format with parquet-go (possibly using intermediate structs ?) then it should achieve compatibility.   
Does it make sense ?  

 I made an attempt to mirror the parquet structure of polars. But it does not seem to work immediately. See [trial.go](https://github.com/oscar6echo/parquet-go-explo/blob/main/io/trial.go).

Is that possible ?  
I recognize polars struct nested structure seem a bit convoluted but probably makes sense in their context.  
Else I should probably try a lower level lib. In this case, any recommendation ?

I would be grateful for any hint as I think the ability to interact with a polars dataframe from outside its ecosystem is quite interesting - particularly from "Goland" as Go is quite complementary with Python.