search

apache / parquet-format

Apache Parquet Format
https://parquet.apache.org/
Apache License 2.0
1.69k stars 422 forks source link

The Parquet Spec doesn't specify whether multiple columns are allowed to have the same name. #421

Open asfimport opened 10 months ago

asfimport commented 10 months ago

The parquet format specification doesn't say whether a Parquet file having columns with the same name (in the same group node, so really exactly the same name) is valid. I.e., say I have a Parquet file with two columns. Both are called x. Is this file a valid Parquet file?

Reporter: Jan Finis / @jfinis

Note: This issue was originally created as PARQUET-2345. Please see the migration documentation for further details.

asfimport commented 10 months ago

Gang Wu / @wgtmac: I didn't find any statement to disallow identical field names in the parquet specs. For engines projecting columns on field ordinals or field ids, identical field names may not be a big issue. It is a good convention to avoid them.

asfimport commented 9 months ago

Micah Kornfield / @emkornfield: I've at least seen in the wild two columns containing the same name only diverging by case sensitivity.  I agree we should recommend against them since its not clear they will be able to handled well.  If we do update docs, we should also recommend against naming columns using "." as a delimeter as this can also lead to ambiguity.