Describe the bug
I have a databricks job where I am saving my data in json format lzo compressed which requires the library lzo-codec,
on shifting to graviton instances
I noticed that the same job started throwing exception
Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) at com.hadoop.compression.lzo.LzopCodec.getCompressor(LzopCodec.java:171) at com.hadoop.compression.lzo.LzopCodec.createOutputStream(LzopCodec.java:72)
Describe the bug I have a databricks job where I am saving my data in json format lzo compressed which requires the library lzo-codec, on shifting to graviton instances
I noticed that the same job started throwing exception Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) at com.hadoop.compression.lzo.LzopCodec.getCompressor(LzopCodec.java:171) at com.hadoop.compression.lzo.LzopCodec.createOutputStream(LzopCodec.java:72)
Current Config: Worker- c6g.4xlarge Driver- c6g.xlarge Older config: Worker- r4.8xlarge Driver- r4.xlarge
To Reproduce Steps to reproduce the behavior: NA
Expected behavior When shifting back to R class instance it starts working
Screenshots NA
Environment Databricks(AWS)