Support ORC decimal type evolution in ORC reader

[wgzhao]

I have an external hive partitioned table, when I change the table's structure (mostly modify column precision), presto query will trigger the following errors:

io.prestosql.spi.PrestoException: Failed to read ORC file: /tmp/test_part_tbl/logdate=20200911/000000_0
   at io.prestosql.plugin.hive.orc.OrcPageSource.handleException(OrcPageSource.java:138)
   at io.prestosql.plugin.hive.orc.OrcPageSourceFactory.lambda$createOrcPageSource$10
                          (OrcPageSourceFactory.java:344)
    at io.prestosql.orc.OrcBlockFactory$OrcBlockLoader.load(OrcBlockFactory.java:83)
    at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
    at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
    at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
   at io.prestosql.plugin.hive.orc.OrcPageSource$SourceColumn$MaskingBlockLoader.load(OrcPageSource.java:275)
    at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
    at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
    at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
    at io.prestosql.plugin.hive.HivePageSource$CoercionLazyBlockLoader.load(HivePageSource.java:549)
    at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
    at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
    at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
    at io.prestosql.spi.Page.getLoadedPage(Page.java:273)
    at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:304)
    at io.prestosql.operator.Driver.processInternal(Driver.java:379)
    at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
    at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
    at io.prestosql.operator.Driver.processFor(Driver.java:276)
    at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
    at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
    at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
    at io.prestosql.$gen.Presto_340____20200902_012116_2.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalArgumentException: target scale must be larger than source scale
    at io.prestosql.spi.type.Decimals.rescale(Decimals.java:298)
    at io.prestosql.orc.reader.DecimalColumnReader.readShortNotNullBlock(DecimalColumnReader.java:184)
    at io.prestosql.orc.reader.DecimalColumnReader.readNonNullBlock(DecimalColumnReader.java:164)
    at io.prestosql.orc.reader.DecimalColumnReader.readBlock(DecimalColumnReader.java:125)
    at io.prestosql.orc.OrcBlockFactory$OrcBlockLoader.load(OrcBlockFactory.java:76)
    ... 24 more

The above error can be repeated by doing the following instructions:

1. create table and add partitions via beeline

create external table test_part_tbl(id  int, asset decimal(10,2))
partitioned by (logdate string)
stored as orc
location '/tmp/test_part_tbl'
tblproperties('external.table.purge'='true');

alter table test_part_tbl add partition(logdate='20200910');

insert into test_part_tbl partition (logdate='20200910') values(1,10.22);

2. modify column and add another partition and insert a record

alter table test_part_tbl add partition(logdate='20200911');

alter table test_part_tbl change column asset asset decimal(10,3);
insert into test_part_tbl partition (logdate='20200911') values(1,10.23);
insert into test_part_tbl partition (logdate='20200911') values(1,10.233);

3. query test_part_tbl via presto

presto> select * from  default.test_part_tbl;

Query 20200910_053546_00098_cczqj, FAILED, 1 node
Splits: 19 total, 1 done (5.26%)
2.16 [1 rows, 310B] [0 rows/s, 144B/s]

Query 20200910_053546_00098_cczqj failed: Failed to read ORC file: 
hdfs://cfzq/tmp/test_part_tbl/logdate=20200911/000000_0

4. query test_part_tbl via hive

select * from test_part_tbl;
+-------------------+----------------------+------------------------+
| test_part_tbl.id  | test_part_tbl.asset  | test_part_tbl.logdate  |
+-------------------+----------------------+------------------------+
| 1                 | 10.220               | 20200910               |
| 1                 | 10.230               | 20200911               |
| 1                 | 10.233               | 20200911               |
+-------------------+----------------------+------------------------+
3 rows selected (0.487 seconds)

If I drop partition 20200911 and add it again , query will result all data.

I use Hortonworks HDP 3.1.4.0-315 version

• Hadoop HDFS: 3.1.1.3.1
• Hive: 3.1.0
• OS: CentOS 7.7.1908
• PrestoSQL: 340

[sopel39]

Did you reduce decimal scale of your column?

What happens is that Presto reads Parquet file which has decimal data with larger scale, but table/partition schema contains column with smaller scale. In that case decimal data would have to be scaled down and trailing digits would be lost. Presto currently doesn't support rounding in such case.

However, if you increased scale of column that should not happen.

[wgzhao]

I create the above table using stored as parquet instead of stored as orc. when I change the column's precision. presto and hive both query old pricison. 0.003 missing .

like the following :

hive query

hive> select * from test_part_tbl;
+-------------------+----------------------+------------------------+
| test_part_tbl.id  | test_part_tbl.asset  | test_part_tbl.logdate  |
+-------------------+----------------------+------------------------+
| 1                 | 10.220               | 20200910               |
| 1                 | 10.230               | 20200911               |
| 1                 | 10.230               | 20200911               |
+-------------------+----------------------+------------------------+

presto query

presto> select * from hive.default.test_part_tbl;
 id | asset  | logdate
----+--------+----------
  1 | 10.230 | 20200911
  1 | 10.220 | 20200910
  1 | 10.230 | 20200911
(3 rows)

[sopel39]

cc @findepi

[amitds1997]

I would like to take this up, if no one is working on it.

[qibinaoe]

Is this PR merged ?