MacOS M1 support (ARM)

[redcpp]

Following the instructions detailed in here: https://aws.amazon.com/blogs/big-data/building-an-aws-glue-etl-pipeline-locally-without-an-aws-account/

Pulled the only image found in docker hub: docker pull amazon/aws-glue-libs:glue_libs_1.0.0_image_01

Then tried testing the imported image:

➜ docker run -itd --name glue_without_notebook amazon/aws-glue-libs:glue_libs_1.0.0_image_01

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

➜  docker run -itd -p 8888:8888 -p 4040:4040 -v ~/.aws:/root/.aws:ro --name glue_jupyter \amazon/aws-glue-libs:glue_libs_1.0.0_image_01 \
/home/jupyter/jupyter_start.sh

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

I'm running on a MacBook Air (M1, 2020).

[prabuvnc]

Can someone help on this issue. Even tried with - - platform parameter but qemu error occurs when trigfering pyspark.

If you can provide a arm64 version of the image it would be really helpful

[bradsheridanUURD]

Just ran into same issue. Anyone have a solution yet for this?

[rsilvestre]

+1

Is it possible to share the Dockerfile to build my own docker image on my Apple M1 Please ?

[paweljankowski]

Can you help us?

[marius-xentral]

+1

[shawnzam]

+1

[Ana-Borg]

+1

[lilfelix]

+1

[edgarvaldez]

+1

[b-y-f]

+1

[niccatsmy]

+1

[tkansara]

+1 Also was not able to run Jupyter notebook on the Docker container.

[tkansara]

Are there any workarounds for this? Can we use a VM with Linux or something as temp solution?

[dkrystall]

+1 Ran the notebook service using Docker through Rosetta, but glue and spark were giving me errors when running my code in the notebook. Using an M1 Max.

[benymahajan]

any arm64 docker image available?

[rpmunoz]

Same problem here. Has anybody tried compiling the image from the source code?

[dalmad2]

+1

[justincandoit]

@awslabs help us!!!

[clarkebishop]

Like @justincanney said, PLEASE HELP.

This almost works. Almost is so frustrating. This line is essential, DOCKER_DEFAULT_PLATFORM=linux/amd64

Jupyter launches but I get the following error: The code failed because of a fatal error. Some things to try: a) Make sure Spark has enough available resources for Jupyter to create a Spark context. b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.

I'm going to try to max out the resources to see if that helps.

I've attached my docker-compose.yml file. docker-compose-yaml.txt

[clarkebishop]

I'd love to know if Docker's buildx would solve the problem. https://docs.docker.com/desktop/multi-arch/

Have you tried this approach?

[clarkebishop]

I maxed out the resources–8 CPUs and 14GB of RAM but got the same error.

That means the only think that's left is the Spark magics library?

By the way, a Python notebook works just fine. I can create, run, and save notebooks.

[clarkebishop]

I got something to work!

docker run -itd -p 8888:8888 -p 4040:4040 -e DISABLE_SSL="true" -v ~/.aws:/root/.aws:ro --platform linux/amd64 --name glue3_jupyter amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/glue_user/jupyter/jupyter_start.sh

I can't get the equivalent container to launch with docker-compose, and that is puzzling. But this is at least a start.

It's definitely slower than I had hoped, but may be usable.

[danphenderson]

+1

[roygoraposonjr]

amazon/aws-glue-libs just updated their images with arm64 support