Open faithnh opened 1 year ago
Solved by the following method.
Dockerfile
to install libzstd-devel
and libhadoop
FROM amazon/aws-glue-libs:glue_libs_3.0.0_image_01
USER root
# libzstd-devel
RUN yum install -y libzstd-devel \
&& rm -rf /var/cache/yum/* \
&& yum clean all
# hadoop
RUN curl -s https://dlcdn.apache.org/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz | tar -xz -C /usr/local/
RUN cd /usr/local && ln -s ./hadoop-3.2.4 hadoop
USER glue_user
# To use hadoop command
ENV JAVA_HOME=/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/jre
/home/glue_user/spark/conf/spark-defaults.conf
spark.driver.extraLibraryPath /usr/local/hadoop/lib/native
spark.executor.extraLibraryPath /usr/local/hadoop/lib/native
Reference:
After thinking about it, I thought that officially supporting it would make it easier to introduce zstd, so I reopened it.
I want to convert a json to zstd.parquet locally, so I tried running it on the Docker container of
amazon/aws-glue-libs:glue_libs_3.0.0_image_01
.But the following error occurred and it did not work.
Glue 3.0 on AWS supports ZSTD, but doesn't
amazon/aws-glue-libs:glue_libs_3.0.0_image_01
support it?If there is a way to support zstd compression, please let me know.
Here is the code I used to check the problem.
For using s3 locally, a
minio
container has also been launched and is linked.