search

Pathairush / airflow_hive_spark_sqoop

A docker using the airflow with Hadoop ecosystem (hive, spark, and sqoop)
11 stars 8 forks source link

Can not use docker image linux/arm64 #1

Closed zainal-abidin-assegaf closed 3 years ago

zainal-abidin-assegaf commented 3 years ago

Can we convert ppatcoding/airflow linux/arm64 to linux/amd64 ??

My machine can not run docker image linux/arm64,

Thank you

zainal-abidin-assegaf commented 3 years ago

image

I have recreated your dockerfile in amd64 env , now is working properly, but how to use airflow and sqoop to ETL from postgres to hdfs ??

Thank you

Pathairush commented 3 years ago

@4ss3g4f Thanks for the issue. This docker is intended to only run on linux/arm64. If you need to run on linux/amd64, you can build it yourself, or there are many interesting Airflow with Hadoop components for linux/amd64 out there in the docker hub.

For how to use airflow and sqoop to ETL from Postgres to HDFS. You can take a look at this repository as a reference. It may not be 100% compatible with what you are looking for (It didn't use the sqoopOperator from airflow), but maybe you get some idea.