Open redcpp opened 3 years ago
Can someone help on this issue. Even tried with - - platform parameter but qemu error occurs when trigfering pyspark.
If you can provide a arm64 version of the image it would be really helpful
Just ran into same issue. Anyone have a solution yet for this?
+1
Is it possible to share the Dockerfile to build my own docker image on my Apple M1 Please ?
Can you help us?
+1
+1
+1
+1
+1
+1
+1
+1 Also was not able to run Jupyter notebook on the Docker container.
Are there any workarounds for this? Can we use a VM with Linux or something as temp solution?
+1 Ran the notebook service using Docker through Rosetta, but glue and spark were giving me errors when running my code in the notebook. Using an M1 Max.
any arm64 docker image available?
Same problem here. Has anybody tried compiling the image from the source code?
+1
@awslabs help us!!!
Like @justincanney said, PLEASE HELP.
This almost works. Almost is so frustrating. This line is essential, DOCKER_DEFAULT_PLATFORM=linux/amd64
Jupyter launches but I get the following error:
The code failed because of a fatal error. Some things to try: a) Make sure Spark has enough available resources for Jupyter to create a Spark context. b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.
I'm going to try to max out the resources to see if that helps.
I've attached my docker-compose.yml file. docker-compose-yaml.txt
I'd love to know if Docker's buildx would solve the problem. https://docs.docker.com/desktop/multi-arch/
Have you tried this approach?
I maxed out the resources–8 CPUs and 14GB of RAM but got the same error.
That means the only think that's left is the Spark magics library?
By the way, a Python notebook works just fine. I can create, run, and save notebooks.
I got something to work!
docker run -itd -p 8888:8888 -p 4040:4040 -e DISABLE_SSL="true" -v ~/.aws:/root/.aws:ro --platform linux/amd64 --name glue3_jupyter amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/glue_user/jupyter/jupyter_start.sh
I can't get the equivalent container to launch with docker-compose, and that is puzzling. But this is at least a start.
It's definitely slower than I had hoped, but may be usable.
+1
amazon/aws-glue-libs just updated their images with arm64 support
Following the instructions detailed in here: https://aws.amazon.com/blogs/big-data/building-an-aws-glue-etl-pipeline-locally-without-an-aws-account/
Pulled the only image found in docker hub:
docker pull amazon/aws-glue-libs:glue_libs_1.0.0_image_01
Then tried testing the imported image:
I'm running on a MacBook Air (M1, 2020).