It would be better to support crc32 function:
steps:
create a hive table:
hive> show create table order_orc4;
OK
CREATE TABLE `order_orc4`(
`order_id` int,
`order_date` string,
`order_customer_id` int,
`order_status` string)
PARTITIONED BY (
`date` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'hdfs://10.19.183.210:9000/user/hive_remote/warehouse/order_orc4'
TBLPROPERTIES (
'bucketing_version'='2',
'orc.compression'='SNAPPY',
'transient_lastDdlTime'='1663645093')
insert some rows:
hive> select * from order_orc4;
OK
1 2022-09-10 1 desc 20220920
2 2022-09-20 2 desc 20220920
run SQL with crc32 function:
spark.sql("select crc32(order_status) from order_orc4").show
driverlogs:
! <Crc32> crc32(cast(order_status#34 as binary)) cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.catalyst.expressions.Crc32
It would be better to support crc32 function: steps: create a hive table:
insert some rows:
run SQL with crc32 function:
driverlogs: