Parquet TODO

[nevillelyh]

• [ ] Avro array support in AvroWriteSupport - old TwoLevelListWriter vs new ThreeLevelListWriter
• [ ] Avro nullabe arrays and arrays of nullables
• [x] Fix parquet.avro.data.supplier with generic records in test #278
• [x] Schema compatibility check in ReadSupport https://github.com/spotify/magnolify/commit/2aea4e8de0aaae4db7517e4dd61dbf2df19e20f0
• [x] Schema evolution for enums #290
• [x] Schema evolution for arrays https://github.com/spotify/magnolify/commit/6c00ecbcc6cc5e975676a8f914e407b5d7c956ba

[nevillelyh]

Turns out the new 3 level list is more complex.

With the default 2 level list, myField: List[T] is written as:

required group myField (LIST) {
  repeated T array;
}

But the Avro counter part is still "name": "myField", "type": "array", "items": T

While with 3 level list, the Parquet schema becomes:

required group myField (LIST) {
  repeated group list {
    required T element;
  }
}

And the Avro record becomes [{"element": t1}, {"element": t1}]...

WIP in https://github.com/spotify/magnolify/tree/neville/pq-avro

[nevillelyh]

More on Avro array mapping. The following Avro fields

{"name": "field1", "type:" {"type": "array", "items": "string"}, "default": [] } // required array field that defaults to empty array {"name": "field2", "type:" ["null", {"type": "array", "items": "string"}], "default": null } // nullable array field that defaults to null

map to:

required group field1 (LIST) {
  repeated binary array (STRING);
}
optional group field2 (LIST) {
  repeated binary array (STRING);
}