This PR includes additional 3 parameter for timestamps in parquet:
USE_INT64_TIMESTAMPS
Default value is false.
If the parameter is false the timestamps will be saved in parquet as INT96. .
If the parameter is true the TIMESTAMP and TIMESTAMPZ will be saved in parquet as INT64 with TimestampLogicalTypeAnnotation.
USE_LOCAL_PXF_TIMEZONE_WRITE
This parameter work only for WRITABLE external tables only.
Default value is true (for backward compatibility for external tables when this parameter was absent)
If the parameter is true the TIMESTAMP value will be converted to UTC zone regarding to the JVM local time zone on the PXF service. For example, the local time zone of JVM is +03:00. When the timestamp 2005-10-03 15:05:15.123 comes from Greenplum it will be converted to UTC regarding local time zone 2005-10-03 12:05:15.123 (- 3 hours). And then it will be saved to the parquet file.
If the parameter is false the timestamp will be saved without conversion to UTC.
Pay attention, that the parameter doesn’t work for TIMESTAMPZ type as TIMESTAMPZ always has time zone offset and we always convert it to UTC using this offset.
USE_LOCAL_PXF_TIMEZONE_READ
This parameter work only for READABLE external tables only.
Default value is true (for backward compatibility for external tables when this parameter was absent)
If the parameter is true the TIMESTAMP from the parquet file will be converted to the JVM local time zone on the PXF service and then will be sent to Greenplum. For example, the local time zone of JVM is +03:00. When the timestamp 2005-10-03 15:05:15.123 comes from the parquet file it will be converted to the JVM local timezone 2005-10-03 18:05:15.123 (+ 3 hours) and then it will sent to Greenplum. If the column type in the Greenplum is timestamp with time zone, this time will be saved according to the Greenplum time zone. So, if Greenplum has +01:00 time zone, the timestamp in the Greenplum will be saved as 2005-10-03 18:05:15.123+01:00. So, in the parquet file it was 2005-10-03 15:05:15.123 and finally it became 2005-10-03 18:05:15.123+01:00.
If the parameter is false the timestamp will be sent without conversion to JVM local time zone. In the example above, the timestamp 2005-10-03 15:05:15.123 in the parquet file will be saved as 2005-10-03 15:05:15.123+01:00 in Greenplum. If the column type in Greenplum is timestamp (without time zone) the timestamp in Greenplum will be the same as in the parquet file.
This PR includes additional 3 parameter for timestamps in parquet:
USE_INT64_TIMESTAMPS
Default value is false.
If the parameter is
falsethe timestamps will be saved in parquet as INT96. .If the parameter is
truethe TIMESTAMP and TIMESTAMPZ will be saved in parquet as INT64 with TimestampLogicalTypeAnnotation.USE_LOCAL_PXF_TIMEZONE_WRITE
This parameter work only for WRITABLE external tables only.
Default value is true (for backward compatibility for external tables when this parameter was absent)
If the parameter is
truethe TIMESTAMP value will be converted to UTC zone regarding to the JVM local time zone on the PXF service. For example, the local time zone of JVM is +03:00. When the timestamp2005-10-03 15:05:15.123comes from Greenplum it will be converted to UTC regarding local time zone2005-10-03 12:05:15.123(- 3 hours). And then it will be saved to the parquet file.If the parameter is
falsethe timestamp will be saved without conversion to UTC.Pay attention, that the parameter doesn’t work for TIMESTAMPZ type as TIMESTAMPZ always has time zone offset and we always convert it to UTC using this offset.
USE_LOCAL_PXF_TIMEZONE_READ
This parameter work only for READABLE external tables only.
Default value is true (for backward compatibility for external tables when this parameter was absent)
If the parameter is
truethe TIMESTAMP from the parquet file will be converted to the JVM local time zone on the PXF service and then will be sent to Greenplum. For example, the local time zone of JVM is +03:00. When the timestamp2005-10-03 15:05:15.123comes from the parquet file it will be converted to the JVM local timezone 2005-10-03 18:05:15.123 (+ 3 hours) and then it will sent to Greenplum. If the column type in the Greenplum is timestamp with time zone, this time will be saved according to the Greenplum time zone. So, if Greenplum has +01:00 time zone, the timestamp in the Greenplum will be saved as2005-10-03 18:05:15.123+01:00. So, in the parquet file it was2005-10-03 15:05:15.123and finally it became2005-10-03 18:05:15.123+01:00.If the parameter is
falsethe timestamp will be sent without conversion to JVM local time zone. In the example above, the timestamp2005-10-03 15:05:15.123in the parquet file will be saved as2005-10-03 15:05:15.123+01:00in Greenplum. If the column type in Greenplum is timestamp (without time zone) the timestamp in Greenplum will be the same as in the parquet file.