wangty6
commented
2 years ago same problem
Open robmaz opened 4 years ago
wangty6
commented
2 years ago same problem
gtinjr
commented
1 year ago This is a problem on FourMcInputStream.close() method calling LZ4Decompressor.releaseDirectBuffers. when reset gets called the buffers have been previously set to null and therefore it causes the NullPointerException. Commenting this out on FourMcInputStream.close() method fix this issue. //((Lz4Decompressor)decompressor).releaseDirectBuffers(); I dont now the reason why this is getting called at this. I looked at a similar project hadoop-lzo from twitter and it does not releaseDirectBuffers in its inputstream.close() method. It is very destructive to make this call here since the decompressor will later be return to the hadoop CodecPool and it will be a bad decompressor in the pool since it is internal buffers have been destroyed.
The following scala code can be used to reproduce this issue, using spark 2.4.4-scala-2.11 and hadoop 2.7.0. This will fail here val raw_df = spark.read.option("header", value = true).csv("file:///c:/lift/V1_sep_test_900.psv.4mc") when it uses the inputstream to do a partial read to create the dataframe and its schema, then it was supposed to return the decompressor back to the pool and later reused on this line raw_df.show(5) but it fails after calling calling reset on the decompressor after its directbuffers were released in the close method.
object Decompressor { def createSparkSession(): SparkSession = { System.setProperty("hadoop.home.dir", "C:\opt\mapr\hadoop\hadoop-2.7.0") System.setProperty("SPARK_HOME", "C:\opt\mapr\spark\spark-2.4.4") return SparkSession.builder() .master("local[1]") .appName("Decompressor - 4mc debug") .config("spark.some.config.option", "config-value") .getOrCreate() }
def main(args: Array[String]): Unit = { val spark = createSparkSession() val raw_df = spark.read.option("header", value = true) .csv("file:///c:/lift/V1_sep_test_900.psv.4mc") raw_df.show(5) } }
Not sure if this is the right place to report such issues. I just installed the hadoop-4mc-2.1.0 to my Hadoop 2.7.7 cluster, but get a null pointer exception when it is apparently trying to decompress the input for a streaming job. The same job with the uncompressed input foo.txt runs fine.
foo.txt.4mc.gz