[FEA] Add in tests for special parquet/ORC compression formats

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs

https://nvidia.github.io/spark-rapids

Apache License 2.0

792 stars 231 forks source link

[FEA] Add in tests for special parquet/ORC compression formats #143

Open revans2 opened 4 years ago

revans2 commented 4 years ago

Is your feature request related to a problem? Please describe. Spark does not support some compression types without extra jars. We need a way to know if these are available or not so we can properly test them.

'lzo', 'lz4', 'brotli', 'zstd'

Parquet supports all of these types, ORC only supports 'lzo'

sameerz commented 4 years ago

CuDF only supports brotli at this time.