apache / parquet-java

Apache Parquet Java
https://parquet.apache.org/
Apache License 2.0
2.64k stars 1.41k forks source link

Provide callback to allow user defined key-value metadata merging strategy #1740

Open asfimport opened 9 years ago

asfimport commented 9 years ago

When merging footers, Parquet doesn't know how to merge conflicting user defined key-value metadata entries, and simply throws. It would be better to provide callbacks to let users define metadata merging strategies.

For example, in Spark SQL, we store our own schema information in Parquet files as key-value metadata (similar to parquet-avro). While trying to add schema merging support for reading Parquet files with different but compatible schemas, InitContext.getMergedKeyValueMetaData throws because we have different Spark SQL schemas stored in different Parquet data files. Thus, we have to overwrite ParquetInputFormat and merge the schema within getSplits, which is kinda hacky and inconvenient.

Reporter: Cheng Lian / @liancheng

Related issues:

Note: This issue was originally created as PARQUET-194. Please see the migration documentation for further details.

asfimport commented 9 years ago

Ryan Blue / @rdblue: This should go away because we shouldn't need to merge metadata. PARQUET-139 updates ParquetInputFormat so that the ReadContext is initialized for each file independently on the task side.

asfimport commented 9 years ago

Ryan Blue / @rdblue: Linking to 1.6.0 release issue, which makes this no longer needed.

asfimport commented 9 years ago

Cheng Lian / @liancheng: Thanks  @rdblue, I'm closing this.

asfimport commented 9 years ago

Cheng Lian / @liancheng: @rdblue After double thinking about this, I think providing such a callback is still useful. Because we also need to merge user defined key-value metadata in ParquetOutputCommitter for writing summary files. Currently, if any conflicting user defined key-value pair is found, Parquet simply throws an exception and gives up writing the summary file.

I noticed that in PARQUET-179, Parquet developers tried to retire both _metadata and _common_metadata at first, but later on decided to keep _common_metadata for frameworks and tools like Pig, where no centralized schema management service is available.

asfimport commented 9 years ago

Ryan Blue / @rdblue: True, we decided we needed to keep the code to produce _common_metadata around for a while longer. That isn't to say I think it should be used, except in some specific cases. The problem is that information in that file could easily be incorrect from normal operations on a folder of Parquet data, like adding new files. A better option would be to track the metadata elsewhere, like the Hive metastore or with another dataset management library like Kite. Parquet doesn't manage entire datasets of Parquet files and I'm not sure that it should be expected to.