Closed findinpath closed 2 years ago
@alexjo2144 , @findepi FYI
See https://github.com/trinodb/trino/issues/7953 (and the linked issue and PR).
EDIT: Seems unrelated since an existing test fails.
Marking as a release blocker since it's a regression and we'll end up disabling the test to unblock master while a fix is being made - https://github.com/trinodb/trino/pull/10487
If this is just an issue in the experimental writer, I might not block the release on it. Especially since we need to release soon to address a more serious issue with how log rotation is being performed.
I'm fine either way since IIUC this is a pre-existing bug that was uncovered by new tests. @findinpath can correct me if my understanding is wrong.
The PR containing the test code used to discover this issue:
The issue of reading from hdp3 hive incorrectly data written with parquet must be an older issue.
I just realized that I didn’t discover it while writing the tests due to fact that I was using onTrino()
and TestHiveStorageFormat was setting the session properties via defaultQueryExecutor
- reason why the session property hive.experimental_parquet_optimized_writer_enabled
was not actually set in the trino session and therefore the test ran successfully.
So, this is indeed no regression issue, but rather an issue that existed before writing the test.
Caused by: java.sql.SQLException: java.io.IOException: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file hdfs://hadoop-master:9000/user/hive/warehouse/stora....
hadoop 3 is based on parquet-1.10.x and hadoop 2 is based on parquet 1.8.1 As of parquet-1.10.x the reading of decimals in parquet-mr library is done differently. See https://issues.apache.org/jira/browse/PARQUET-884 and https://github.com/apache/parquet-mr/pull/404 for details.
Possible fix: Instead of writing integer or long for small decimals, write always FIXED_LEN_BYTE_ARRAY
. This approach has the downside of using more space than needed, but it is consistent with both hadoop 2 & hadoop 3 hive readers.
The experimental Parquet writer follows however the Parquet specification
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal
DECIMAL can be used to annotate the following types:
int32: for 1 <= precision <= 9
int64: for 1 <= precision <= 18; precision < 10 will produce a warning
fixed_len_byte_array: precision is limited by the array size. Length n can store <= floor(log_10(2^(8*n - 1) - 1)) base-10 digits
binary: precision is not limited, but is required. The minimum number of bytes to store the unscaled value should be used.
IIRC Spark has issues with the byte array encoding. See https://stackoverflow.com/questions/63578928/spark-unable-to-read-decimal-columns-in-parquet-files-written-by-avroparquetwrit and https://github.com/sksamuel/avro4s/issues/271
Closing this in favor of the older issue on the topic: https://github.com/trinodb/trino/issues/6377
It seems that there is an issue in the
trino-parquet
module for writing the data.Failing build:
https://github.com/trinodb/trino/runs/4723741729?check_suite_focus=true
Failing test:
Reproduction scenario:
Run the test
io.trino.tests.product.hive.TestHiveStorageFormats#testInsertAllSupportedDataTypesWithTrino
Executing the query on hive:
is causing the issue.
However, executing the query
works just fine.
The data is written in the test with Trino and read with Hive. It seems that there is a regression issue in the
trino-parquet
module for writing the data.Selecting the
c_timestamp
field on hive also throw the exception: