rcasey-earthscope
commented
2 years ago Note that I am running on an M1 (ARM) Mac. Not sure if that matters.
Closed rcasey-earthscope closed 2 years ago
rcasey-earthscope
commented
2 years ago Note that I am running on an M1 (ARM) Mac. Not sure if that matters.
autumnjohnson
commented
2 years ago Hi @rcasey-iris
Can you confirm that you are using the latest from the branch add-docker-compose-test-to-gh-workflow?
What happens when you first run:
docker compose down
docker compose build --no-cache
autumnjohnson
commented
2 years ago Hi @rcasey-iris
Actually Jean-Marie ran into a similar issue with the commit that I pushed yesterday. Let me see what happened.
rcasey-earthscope
commented
2 years ago Thanks for the suggestion. The docker compose down performs a shutdown call, but the process is not actually running, so a soft warning. The docker compose build seems to yield the same result. The artifacts seem to load in parallel, and the docker.io lines continue to count up to 12s before failing. Same error messages.
Also, yes this is the test_to_gh branch. I downloaded the ZIP instead of cloning and it named the file to the branch name.
autumnjohnson
commented
2 years ago Yes, this error is from an optimization I made yesterday. Let me revert some changes.
autumnjohnson
commented
2 years ago Hi @rcasey-iris
Although yours and Jean-Marie's could be separate issue. I found this thread on Stack Overflow. Given that you mention "M1", it it seems related.
autumnjohnson
commented
2 years ago Hi @rcasey-iris ,
I have reverted the commits from yesterday likely caused the build error, unless the one you are experiencing is different from Jean-Marie's. Mind pulling the latest changes and retrying?
autumnjohnson
commented
2 years ago Hi @rcasey-iris
Also, yes this is the test_to_gh branch. I downloaded the ZIP instead of cloning and it named the file to the branch name.
I am not sure what you mean by this, or if that will work. I recommend cloning the repository and checking out the branch itself. Then, in the root of the repository, run docker-compose down then docker-compose build --no-cache and finally docker-compose up
rcasey-earthscope
commented
2 years ago Still having issues even after doing that. I found that I can search for images from the command line. I am on an arm64 platform, and the results are pretty telling.
$ docker search openjdk:8-jre-stretch
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
$ docker search openjdk
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
openjdk "Vanilla" builds of OpenJDK (an open-source … 3253 [OK]
adoptopenjdk/openjdk11 Docker Images for OpenJDK Version 11 binarie… 194
adoptopenjdk/openjdk8 Docker Images for OpenJDK Version 8 binaries… 111
eclipse-temurin Official Images for OpenJDK binaries built b… 106 [OK]
adoptopenjdk/openjdk8-openj9 Docker Images for Eclipse OpenJ9 Version 8 b… 51
adoptopenjdk/openjdk11-openj9 Docker Images for Eclipse OpenJ9 Version 11 … 46
arm64v8/openjdk "Vanilla" builds of OpenJDK (an open-source … 41
adoptopenjdk/openjdk12 Docker Images for OpenJDK Version 12 binarie… 19
adoptopenjdk/openjdk16 Docker Images for OpenJDK Version 16 binarie… 15
ibm-semeru-runtimes IBM Semeru Runtimes Official Images for Open… 13 [OK]
adoptopenjdk/openjdk13 Docker Images for OpenJDK Version 13 binarie… 11
adoptopenjdk/openjdk14 Docker Images for OpenJDK Version 14 binarie… 11
adoptopenjdk/openjdk15 Docker Images for OpenJDK Version 15 binarie… 10
circleci/openjdk CircleCI images for OpenJDK 10 [OK]
amd64/openjdk "Vanilla" builds of OpenJDK (an open-source … 9
springcloud/openjdk OpenJDK Docker image 3
symphonicsoft/openjdkbase Openjdk base images with dumb-init 1
winamd64/openjdk "Vanilla" builds of OpenJDK (an open-source … 1
cimg/openjdk The CircleCI OpenJDK (Java) Docker Convenien… 1
classmethod/openjdk-with-git docker image for openjdk and git 1 [OK]
cfje/openjdk OpenJDK Builder Image 0
store/microsoft/defaultpublisher Product family for Zulu for Azure builds of … 0
store/microsoft/defaultpublisher Microsoft Build of OpenJDK 0
suranagivinod/openjdk8 openjdk:8-jre-slim, zip & unzip 0
djdapz/openjdk-node-cf openjdk with node (npm) and cloud foundry cl… 0```Never mind. I can use a different utility to search for the tags available and it looks like it's visible.
azimuth-1019:backend rob$ skopeo list-tags docker://docker.io/library/python | grep 3.6.5
"3.6.5",
"3.6.5-alpine",
"3.6.5-alpine3.4",
"3.6.5-alpine3.6",
"3.6.5-alpine3.7",
"3.6.5-jessie",
"3.6.5-onbuild",
"3.6.5-slim",
"3.6.5-slim-jessie",
"3.6.5-slim-stretch",
"3.6.5-stretch",
"3.6.5-windowsservercore",
"3.6.5-windowsservercore-1709",
"3.6.5-windowsservercore-ltsc2016",Okay, some progress. I can attempt a manual load of the image and it gives me some meaningful feedback.
azimuth-1019:yasmine-stationxml-editor rob$ docker image pull docker.io/library/openjdk:8-jre-stretch
8-jre-stretch: Pulling from library/openjdk
no matching manifest for linux/arm64/v8 in the manifest list entriesSo, this has to do with the availability of arm64 implementations of openjdk. Fortunately, there is a 'latest' version that supports arm64.
et voila!
$ docker compose up
[+] Running 0/2
⠿ yasmine-backend Error 1.9s
⠿ yasmine-frontend Error 1.9s
[+] Building 18.8s (14/28)
=> [yasmine/backend internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 32B 0.0s
=> [yasmine/frontend internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.55kB 0.0s
=> [yasmine/backend internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [yasmine/frontend internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [yasmine/backend internal] load metadata for docker.io/library/python:3.6.5-slim 2.4s
=> [yasmine/frontend internal] load metadata for docker.io/library/openjdk:latest 2.2s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [auth] library/openjdk:pull token for registry-1.docker.io 0.0s
=> [yasmine/frontend 1/8] FROM docker.io/library/openjdk:latest@sha256:3b3c313f3ffaca9cad66a6c13cdbb330e310907632685db43f52be8264a4a7e8 16.2s
=> => resolve docker.io/library/openjdk:latest@sha256:3b3c313f3ffaca9cad66a6c13cdbb330e310907632685db43f52be8264a4a7e8 0.0s
=> => sha256:d200eb5167ed838a74239544d81e10c3d37a59ea571f23c1d6ed6e0c207cf997 14.29MB / 14.29MB 1.9s
=> => sha256:0329e180496a34157319b853f09219d48897f2a948a049fb6c92198af2fceff6 186.36MB / 186.36MB 13.1s
=> => sha256:3b3c313f3ffaca9cad66a6c13cdbb330e310907632685db43f52be8264a4a7e8 1.04kB / 1.04kB 0.0s
=> => sha256:0722e5cd28b8834d2c2e6a3659ba4631c6f6aea6aa88361feff58032bb3514e3 954B / 954B 0.0s
=> => sha256:7255975f2e1569db12b87a5d831de1c9d8e97377268a1eab9129160aa51f13df 4.46kB / 4.46kB 0.0s
=> => sha256:293fbd461d2c2a94e5d457aa3c7b18429b8457b06d5c2ad1a57113b1cac6d657 42.02MB / 42.02MB 5.1s
=> => extracting sha256:293fbd461d2c2a94e5d457aa3c7b18429b8457b06d5c2ad1a57113b1cac6d657 1.4s
=> => extracting sha256:d200eb5167ed838a74239544d81e10c3d37a59ea571f23c1d6ed6e0c207cf997 0.4s
=> => extracting sha256:0329e180496a34157319b853f09219d48897f2a948a049fb6c92198af2fceff6 2.9s
=> [yasmine/backend 1/11] FROM docker.io/library/python:3.6.5-slim@sha256:c838ee57410cfa16163524ffaa069397df2def8611ae3ea998ad5244f1c7e275 10.1s
=> => resolve docker.io/library/python:3.6.5-slim@sha256:c838ee57410cfa16163524ffaa069397df2def8611ae3ea998ad5244f1c7e275 0.0s
=> => sha256:c838ee57410cfa16163524ffaa069397df2def8611ae3ea998ad5244f1c7e275 2.37kB / 2.37kB 0.0s
=> => sha256:f3504a93e05d7dffa3f7cdb0ef730f598fa4dc88542130dea7abb5db9ba82422 1.37kB / 1.37kB 0.0s
=> => sha256:1f20f9befb9e25a48baeb85b4932914b11718c7f81fb54e08a56786d135a0829 5.99kB / 5.99kB 0.0s
=> => sha256:153362478bb06475bede3bd6da2ff08a91dd9fc6f245c70bb5a033b65f38ac14 20.35MB / 20.35MB 8.2s
=> => sha256:850b3af51e965c345636f3711bbad32357ddad888662bd5a5809796c04722b3b 3.10MB / 3.10MB 6.4s
=> => sha256:7b8d4ab1d3a31c98120d9b3095e0da61964017ee9999e03c1dfbbf8be86d1b7d 21.24MB / 21.24MB 8.9s
=> => sha256:522972f933b4299fb55ac2ed97f6b227e6c0d3ba13ac568f863c103b093ddbb3 239B / 239B 8.6s
=> => extracting sha256:153362478bb06475bede3bd6da2ff08a91dd9fc6f245c70bb5a033b65f38ac14 0.7s
=> => sha256:0e43946358a51f18015e108e6c97330390b84e74938e042e1310863fb01c20d7 2.07MB / 2.07MB 9.7s
=> => extracting sha256:850b3af51e965c345636f3711bbad32357ddad888662bd5a5809796c04722b3b 0.1s
=> => extracting sha256:7b8d4ab1d3a31c98120d9b3095e0da61964017ee9999e03c1dfbbf8be86d1b7d 0.6s
=> => extracting sha256:522972f933b4299fb55ac2ed97f6b227e6c0d3ba13ac568f863c103b093ddbb3 0.0s
=> => extracting sha256:0e43946358a51f18015e108e6c97330390b84e74938e042e1310863fb01c20d7 0.1s
=> [yasmine/backend internal] load build context 1.9s
=> => transferring context: 57.18MB 1.9s
=> [yasmine/backend 2/11] WORKDIR /opt/yasmine/src 0.4s
=> CANCELED [yasmine/backend 3/11] RUN apt-get update && apt-get install -y wget unzip gnupg gcc 5.9s
=> ERROR [yasmine/frontend 2/8] RUN apt-get update && apt-get install -y wget unzip 0.3s
------
> [yasmine/frontend 2/8] RUN apt-get update && apt-get install -y wget unzip:
#0 0.155 /bin/sh: apt-get: command not found
------
failed to solve: executor failed running [/bin/sh -c apt-get update && apt-get install -y wget unzip]: exit code: 127
Now there is just a new problem, which may require its own new issue. For the purposes of release, intel based systems should not run into this problem, but those running a Mac M1 will need guidance on what to change.
rcasey-earthscope
commented
2 years ago So, the issue above is due to the fact that the image loaded was not Debian based. Probably Alpine based. Tried modifying commands to apk but you still run into issues further down the line.
Instead, went back to the frontend/Dockerfile and changed the image to a more generic
FROM openjdk:jre-stretch
I suspect that only more recent versions of Java are built for the arm64 platform. I was able to get it to go pretty far. Now ran into an issue with installing an instance of google-chrome on the backend side. I think that is used for unit testing, but I am not sure.
=> [yasmine/frontend 2/8] RUN apt-get update && apt-get install -y wget unzip 9.9s
=> [yasmine/backend 4/11] RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - 3.6s
=> [yasmine/frontend 3/8] RUN useradd -m sencha && cd && cp -R .bashrc .profile /home/sencha && mkdir -p /code /opt/Sencha/SDK /opt/ 0.3s
=> CANCELED [yasmine/frontend 4/8] RUN cd /home/sencha && wget http://cdn.sencha.com/cmd/6.5.3.6/no-jre/SenchaCmd-6.5.3.6-linux-amd64.sh.zi 4.4s
=> [yasmine/backend 5/11] RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' 0.2s
=> [yasmine/backend 6/11] RUN apt-get update 1.5s
=> ERROR [yasmine/backend 7/11] RUN apt-get -y install google-chrome-stable 0.7s
------
> [yasmine/backend 7/11] RUN apt-get -y install google-chrome-stable:
#0 0.197 Reading package lists...
#0 0.568 Building dependency tree...
#0 0.628 Reading state information...
#0 0.635 Package google-chrome-stable is not available, but is referred to by another package.
#0 0.635 This may mean that the package is missing, has been obsoleted, or
#0 0.635 is only available from another source
#0 0.635
#0 0.639 E: Package 'google-chrome-stable' has no installation candidate
------
failed to solve: executor failed running [/bin/sh -c apt-get -y install google-chrome-stable]: exit code: 100On the backend Dockerfile, I had to change the browser install to
RUN apt-get -y install chromium
This got things moving along pretty good until the pip install attempt for matplotlib. Apparently this is causing issues for people on M1 macs, and probably arm64 platforms in general.
#0 185.6 IMPORTANT WARNING:
#0 185.6 pkg-config is not installed.
#0 185.6 matplotlib may not be able to find some of its dependencies
#0 185.6 ============================================================================
#0 185.6 Edit setup.cfg to change the build options
#0 185.6
#0 185.6 BUILDING MATPLOTLIB
#0 185.6 matplotlib: yes [2.0.0]
#0 185.6 python: yes [3.6.5 (default, Jun 27 2018, 11:19:31) [GCC
#0 185.6 6.3.0 20170516]]
#0 185.6 platform: yes [linux]
#0 185.6
#0 185.6 REQUIRED DEPENDENCIES AND EXTENSIONS
#0 185.6 numpy: yes [not found. pip may install it below.]
#0 185.6 six: yes [six was not found.pip will attempt to install
#0 185.6 it after matplotlib.]
#0 185.6 dateutil: yes [dateutil was not found. It is required for date
#0 185.6 axis support. pip/easy_install may attempt to
#0 185.6 install it after matplotlib.]
#0 185.6 functools32: yes [Not required]
#0 185.6 subprocess32: yes [Not required]
#0 185.6 pytz: yes [pytz was not found. pip will attempt to install
#0 185.6 it after matplotlib.]
#0 185.6 cycler: yes [cycler was not found. pip will attempt to
#0 185.6 install it after matplotlib.]
#0 185.6 tornado: yes [tornado was not found. It is required for the
#0 185.6 WebAgg backend. pip/easy_install may attempt to
#0 185.6 install it after matplotlib.]
#0 185.6 pyparsing: yes [pyparsing was not found. It is required for
#0 185.6 mathtext support. pip/easy_install may attempt to
#0 185.6 install it after matplotlib.]
#0 185.6 libagg: yes [pkg-config information for 'libagg' could not
#0 185.6 be found. Using local copy.]
#0 185.6 freetype: no [The C/C++ header for freetype2 (ft2build.h)
#0 185.6 could not be found. You may need to install the
#0 185.6 development package.]
#0 185.6 png: no [pkg-config information for 'libpng' could not
#0 185.6 be found.]
#0 185.6 qhull: yes [pkg-config information for 'qhull' could not be
#0 185.6 found. Using local copy.]
#0 185.6
#0 185.6 OPTIONAL SUBPACKAGES
#0 185.6 sample_data: yes [installing]
#0 185.6 toolkits: yes [installing]
#0 185.6 tests: no [skipping due to configuration]
#0 185.6 toolkits_tests: no [skipping due to configuration]
#0 185.6
#0 185.6 OPTIONAL BACKEND EXTENSIONS
#0 185.6 macosx: no [Mac OS-X only]
#0 185.6 qt5agg: no [PyQt5 not found]
#0 185.6 qt4agg: no [PySide not found; PyQt4 not found]
#0 185.6 gtk3agg: no [Requires pygobject to be installed.]
#0 185.6 gtk3cairo: no [Requires cairocffi or pycairo to be installed.]
#0 185.6 gtkagg: no [Requires pygtk]
#0 185.6 tkagg: yes [installing; run-time loading from Python Tcl /
#0 185.6 Tk]
#0 185.6 wxagg: no [requires wxPython]
#0 185.6 gtk: no [Requires pygtk]
#0 185.6 agg: yes [installing]
#0 185.6 cairo: no [cairocffi or pycairo not found]
#0 185.6 windowing: no [Microsoft Windows only]
#0 185.6
#0 185.6 OPTIONAL LATEX DEPENDENCIES
#0 185.6 dvipng: no
#0 185.6 ghostscript: no
#0 185.6 latex: no
#0 185.6 pdftops: no
#0 185.6
#0 185.6 OPTIONAL PACKAGE DATA
#0 185.6 dlls: no [skipping due to configuration]
#0 185.6
#0 185.6 ============================================================================
#0 185.6 * The following required packages can not be built:
#0 185.6 * freetype, png#0 263.2 ERROR: Could not find a version that satisfies the requirement matplotlib>=1.1.0 (from obspy) (from versions: 0.86, 0.86.1, 0.86.2, 0.91.0, 0.91.1, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1rc1, 1.4.1, 1.4.2, 1.4.3, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 2.0.0b1, 2.0.0b2, 2.0.0b3, 2.0.0b4, 2.0.0rc1, 2.0.0rc2, 2.0.0, 2.0.1, 2.0.2, 2.1.0rc1, 2.1.0, 2.1.1, 2.1.2, 2.2.0rc1, 2.2.0, 2.2.2, 2.2.3, 2.2.4, 2.2.5, 3.0.0rc2, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.1.0rc1, 3.1.0rc2, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.2.0rc1, 3.2.0rc3, 3.2.0, 3.2.1, 3.2.2, 3.3.0rc1, 3.3.0, 3.3.1, 3.3.2, 3.3.3, 3.3.4)
#0 263.2 ERROR: No matching distribution found for matplotlib>=1.1.0Hi @rcasey-iris
I just merged the pull request for this branch into main. I will make another branch to address this issue once I find out what it is and what is going on.
I can test the implementation on my Mac later today.
rcasey-earthscope
commented
2 years ago Thank you. It may turn out that it will be hard for you to replicate this issue, as I think it relates to the platform I am running on.
It may be simply that the arm64 macs will have difficulty carrying out the build themselves, but a pre-built docker image should just simply run. Can we attempt to put out a docker image with the complete build from your platform and we can see if the M1 Mac can simply run it? I don't know if there is a fundamental kernel issue at the container interface between Intel and Arm.
autumnjohnson
commented
2 years ago Hi @rcasey-iris
Can we attempt to put out a docker image with the complete build from your platform and we can see if the M1 Mac can simply run it?
Are you asking me to deploy the image to a container registry like Docker Hub or GitHub Container Registry? The latter would be simplest, and I can make the update in .github/yasmine.yml to do so upon merges with the main branch.
autumnjohnson
commented
2 years ago Hi @rcasey-iris
Do you think this is related to the problem: https://stackoverflow.com/questions/69708866/how-to-make-docker-compose-work-on-m1-chip
If so, looks like a simply fix in the docker-compose file.
rcasey-earthscope
commented
2 years ago Good find. I tried this exact approach last week and it didn't seem to resolve the issue, at least with the JRE image.
I found that I had to select a different image, which was a non-versioned 'jre-stretch' tag. I think that you have to look to newer builds of images to start seeing the arm64 support.
Honestly, until I can get the executable to run, I won't be entirely sure that my selection of JRE image will even work. We'll see. Still working on the python/pip/matplotlib angle to see what it takes to get it to build.
-RobOn May 2, 2022, at 2:27 PM, Autumn Johnson @.***> wrote:
Hi @rcasey-iris https://github.com/rcasey-iris Do you think this is related to the problem: https://stackoverflow.com/questions/69708866/how-to-make-docker-compose-work-on-m1-chip https://stackoverflow.com/questions/69708866/how-to-make-docker-compose-work-on-m1-chip If so, looks like a simply fix in the docker-compose file.
— Reply to this email directly, view it on GitHub https://github.com/iris-edu/yasmine-stationxml-editor/issues/9#issuecomment-1115386579, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL4VN7JSC2LNNBEPGMRUQDVIBCDDANCNFSM5UPQUZUQ. You are receiving this because you were mentioned.
rcasey-earthscope
commented
2 years ago Hang on. I think I had misapplied this trick in docker-compose.yml incorrectly. Last week, I put in the platform for my workstation. Instead, I should have done exactly as the poster in StackOverflow did (I read it dyslexically or said huh?) and used the linux/amd64 platform. It now makes sense to me: I am trying to build an environment where all of the necessary images can be easily found and constructed. amd64, I believe, is the default architecture for docker.io <http://docker.io/> builds. Being a container, I should be able to run it in any Docker installation, regardless of platform.
So, with that, I was able to get my initial build effort to complete without complaint.
Now, I will change the Dockerfiles back to their default settings and try the build again.
Thanks for pointing me back to this suggestion.
-RobOn May 2, 2022, at 2:51 PM, Robert Casey @.***> wrote:
Good find. I tried this exact approach last week and it didn't seem to resolve the issue, at least with the JRE image.
I found that I had to select a different image, which was a non-versioned 'jre-stretch' tag. I think that you have to look to newer builds of images to start seeing the arm64 support.
Honestly, until I can get the executable to run, I won't be entirely sure that my selection of JRE image will even work. We'll see. Still working on the python/pip/matplotlib angle to see what it takes to get it to build.
-Rob
On May 2, 2022, at 2:27 PM, Autumn Johnson @. @.>> wrote:
Hi @rcasey-iris https://github.com/rcasey-iris Do you think this is related to the problem: https://stackoverflow.com/questions/69708866/how-to-make-docker-compose-work-on-m1-chip https://stackoverflow.com/questions/69708866/how-to-make-docker-compose-work-on-m1-chip If so, looks like a simply fix in the docker-compose file.
— Reply to this email directly, view it on GitHub https://github.com/iris-edu/yasmine-stationxml-editor/issues/9#issuecomment-1115386579, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL4VN7JSC2LNNBEPGMRUQDVIBCDDANCNFSM5UPQUZUQ. You are receiving this because you were mentioned.
autumnjohnson
commented
2 years ago Hi @rcasey-iris
That's awesome to hear. I think this is a great fix to include in the docker-compose file as it will ensure others will not run into this problem in the future.
If all goes well with your run, do you wan to share the fix as a commit? Otherwise, which lines were changed and I can do so.
rcasey-earthscope
commented
2 years ago Thanks, Autumn. It's clear that I need to learn more about the intra-container to extra-container interface regarding different host platforms.
When I rolled back the Dockerfiles to their original state, I ran into some issues, so not out of the woods yet. It feels close, though. Even a workaround procedure for M1 Mac users would be good, so want to see just how much needs to be changed.
Still picking at it.On May 3, 2022, at 10:17 AM, Autumn Johnson @.***> wrote:
Hi @rcasey-iris https://github.com/rcasey-iris That's awesome to hear. I think this is a great fix to include in the docker-compose file as it will ensure other's will not run into this problem in the future.
If all goes well with your run, do you wan to share the fix as a commit? Otherwise, which lines were changed and I can do so.
— Reply to this email directly, view it on GitHub https://github.com/iris-edu/yasmine-stationxml-editor/issues/9#issuecomment-1116348646, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL4VN3OOYHU4PUY75Z5VSLVIFNSVANCNFSM5UPQUZUQ. You are receiving this because you were mentioned.
rcasey-earthscope
commented
2 years ago So going back to the original Dockerfiles (but using the platform: linux/amd64 entry for both frontend and backend in docker-compose.yml), I initially ran into an issue with getting a signing key from Google, which apparently was an intermittent thing.
wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
Running
docker compose build --no-cache
...this morning seemed to get past this hangup. The build completes without incident.
Now, the moment of truth is if I can actually run this thing.[+] Running 2/2
⠿ Container yasmine-backend Recreated 0.2s
⠿ Container yasmine-frontend Recreated 0.2s
Attaching to yasmine-backend, yasmine-frontend
yasmine-backend | INFO [alembic.runtime.migration] Context impl SQLiteImpl.
yasmine-backend | INFO [alembic.runtime.migration] Will assume non-transactional DDL.
yasmine-frontend | Sencha Cmd v6.5.3.6
yasmine-frontend | [INF] Using GPL version of Ext JS version 6.2.0.981 from /code/frontend/ext.
yasmine-frontend | [INF] The implications of using GPL version can be found here (http://www.sencha.com/products/extjs/licensing).
yasmine-frontend | [INF] Processing Build Descriptor : classic (development environment)
yasmine-backend | INFO:apscheduler.scheduler:Scheduler started
yasmine-backend | INFO:apscheduler.scheduler:Added job "Application.sync_nrl" to job store "default"
yasmine-backend | INFO:apscheduler.scheduler:Added job "Application.sync_ial" to job store "default"So far so good, until the Sencha stuff starts up the JVM, which is Java 8 for linux/amd64:
yasmine-frontend | #
yasmine-frontend | # A fatal error has been detected by the Java Runtime Environment:
yasmine-frontend | #
yasmine-frontend | # Internal Error (safepoint.cpp:310), pid=10, tid=0x000000403f12b700
yasmine-frontend | # guarantee(PageArmed == 0) failed: invariant
yasmine-frontend | #
yasmine-frontend | # JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-b08)
yasmine-frontend | # Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops)
yasmine-frontend | # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
yasmine-frontend | #
yasmine-frontend | # An error report file with more information is saved as:
yasmine-frontend | # /code/frontend/hs_err_pid10.log
yasmine-frontend | #
yasmine-frontend | # If you would like to submit a bug report, please visit:
yasmine-frontend | # http://bugreport.java.com/bugreport/crash.jsp
yasmine-frontend | #
yasmine-frontend | qemu: uncaught target signal 6 (Aborted) - core dumped
yasmine-frontend | Aborted
yasmine-frontend exited with code 134It took me a bit to get some information that made sense with what I was seeing, but this StackOverflow page describes the condition well.https://stackoverflow.com/questions/66834864/how-to-determine-when-docker-containers-on-an-m1-macbook-are-running-via-qemu https://stackoverflow.com/questions/66834864/how-to-determine-when-docker-containers-on-an-m1-macbook-are-running-via-qemu
So, qemu ( in this case: /usr/bin/qemu-x86_64 ) kicks in when the host architecture is different than the build architecture. This is the next sticking point to look at.Example internal exception:
Event: 7.836 Thread 0x000000400400b000 Exception <a 'java/lang/ClassNotFoundException': com/sencha/command/BasePluginCommands$BasePluginCommandCust
omizer> (0x00000000ecaab820) thrown at [/home/openjdk/jdk8u/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]-RobOn May 2, 2022, at 3:13 PM, Robert Casey @.***> wrote:
Hang on. I think I had misapplied this trick in docker-compose.yml incorrectly. Last week, I put in the platform for my workstation. Instead, I should have done exactly as the poster in StackOverflow did (I read it dyslexically or said huh?) and used the linux/amd64 platform. It now makes sense to me: I am trying to build an environment where all of the necessary images can be easily found and constructed. amd64, I believe, is the default architecture for docker.io http://docker.io/ builds. Being a container, I should be able to run it in any Docker installation, regardless of platform.
So, with that, I was able to get my initial build effort to complete without complaint.
Now, I will change the Dockerfiles back to their default settings and try the build again.
Thanks for pointing me back to this suggestion.
-Rob
On May 2, 2022, at 2:51 PM, Robert Casey @. @.>> wrote:
Good find. I tried this exact approach last week and it didn't seem to resolve the issue, at least with the JRE image.
I found that I had to select a different image, which was a non-versioned 'jre-stretch' tag. I think that you have to look to newer builds of images to start seeing the arm64 support.
Honestly, until I can get the executable to run, I won't be entirely sure that my selection of JRE image will even work. We'll see. Still working on the python/pip/matplotlib angle to see what it takes to get it to build.
-Rob
On May 2, 2022, at 2:27 PM, Autumn Johnson @. @.>> wrote:
Hi @rcasey-iris https://github.com/rcasey-iris Do you think this is related to the problem: https://stackoverflow.com/questions/69708866/how-to-make-docker-compose-work-on-m1-chip https://stackoverflow.com/questions/69708866/how-to-make-docker-compose-work-on-m1-chip If so, looks like a simply fix in the docker-compose file.
— Reply to this email directly, view it on GitHub https://github.com/iris-edu/yasmine-stationxml-editor/issues/9#issuecomment-1115386579, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL4VN7JSC2LNNBEPGMRUQDVIBCDDANCNFSM5UPQUZUQ. You are receiving this because you were mentioned.
rcasey-earthscope
commented
2 years ago Related links for more insighthttps://gitlab.com/qemu-project/qemu/-/issues/340 https://gitlab.com/qemu-project/qemu/-/issues/340
https://github.com/docker/for-mac/issues/6110 https://github.com/docker/for-mac/issues/6110
https://github.com/metanorma/metanorma-docker/issues/126 https://github.com/metanorma/metanorma-docker/issues/126
Looking at trying to create a multiplatform build. We'll see how that goes.On May 3, 2022, at 2:27 PM, Robert Casey @.***> wrote:
So going back to the original Dockerfiles (but using the platform: linux/amd64 entry for both frontend and backend in docker-compose.yml), I initially ran into an issue with getting a signing key from Google, which apparently was an intermittent thing.
wget -q -O - https://dl.google.com/linux/linux_signing_key.pub https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
Running
docker compose build --no-cache
...this morning seemed to get past this hangup. The build completes without incident.
Now, the moment of truth is if I can actually run this thing.
[+] Running 2/2 ⠿ Container yasmine-backend Recreated 0.2s ⠿ Container yasmine-frontend Recreated 0.2s Attaching to yasmine-backend, yasmine-frontend yasmine-backend | INFO [alembic.runtime.migration] Context impl SQLiteImpl. yasmine-backend | INFO [alembic.runtime.migration] Will assume non-transactional DDL. yasmine-frontend | Sencha Cmd v6.5.3.6 yasmine-frontend | [INF] Using GPL version of Ext JS version 6.2.0.981 from /code/frontend/ext. yasmine-frontend | [INF] The implications of using GPL version can be found here (http://www.sencha.com/products/extjs/licensing <http://www.sencha.com/products/extjs/licensing>). yasmine-frontend | [INF] Processing Build Descriptor : classic (development environment) yasmine-backend | INFO:apscheduler.scheduler:Scheduler started yasmine-backend | INFO:apscheduler.scheduler:Added job "Application.sync_nrl" to job store "default" yasmine-backend | INFO:apscheduler.scheduler:Added job "Application.sync_ial" to job store "default"So far so good, until the Sencha stuff starts up the JVM, which is Java 8 for linux/amd64:
yasmine-frontend | # yasmine-frontend | # A fatal error has been detected by the Java Runtime Environment: yasmine-frontend | # yasmine-frontend | # Internal Error (safepoint.cpp:310), pid=10, tid=0x000000403f12b700 yasmine-frontend | # guarantee(PageArmed == 0) failed: invariant yasmine-frontend | # yasmine-frontend | # JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-b08) yasmine-frontend | # Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops) yasmine-frontend | # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again yasmine-frontend | # yasmine-frontend | # An error report file with more information is saved as: yasmine-frontend | # /code/frontend/hs_err_pid10.log yasmine-frontend | # yasmine-frontend | # If you would like to submit a bug report, please visit: yasmine-frontend | # http://bugreport.java.com/bugreport/crash.jsp <http://bugreport.java.com/bugreport/crash.jsp> yasmine-frontend | # yasmine-frontend | qemu: uncaught target signal 6 (Aborted) - core dumped yasmine-frontend | Aborted yasmine-frontend exited with code 134It took me a bit to get some information that made sense with what I was seeing, but this StackOverflow page describes the condition well.
https://stackoverflow.com/questions/66834864/how-to-determine-when-docker-containers-on-an-m1-macbook-are-running-via-qemu https://stackoverflow.com/questions/66834864/how-to-determine-when-docker-containers-on-an-m1-macbook-are-running-via-qemu
So, qemu ( in this case: /usr/bin/qemu-x86_64 ) kicks in when the host architecture is different than the build architecture. This is the next sticking point to look at.
Example internal exception:
Event: 7.836 Thread 0x000000400400b000 Exception <a 'java/lang/ClassNotFoundException': com/sencha/command/BasePluginCommands$BasePluginCommandCust omizer> (0x00000000ecaab820) thrown at [/home/openjdk/jdk8u/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]-Rob
On May 2, 2022, at 3:13 PM, Robert Casey @. @.>> wrote:
Hang on. I think I had misapplied this trick in docker-compose.yml incorrectly. Last week, I put in the platform for my workstation. Instead, I should have done exactly as the poster in StackOverflow did (I read it dyslexically or said huh?) and used the linux/amd64 platform. It now makes sense to me: I am trying to build an environment where all of the necessary images can be easily found and constructed. amd64, I believe, is the default architecture for docker.io http://docker.io/ builds. Being a container, I should be able to run it in any Docker installation, regardless of platform.
So, with that, I was able to get my initial build effort to complete without complaint.
Now, I will change the Dockerfiles back to their default settings and try the build again.
Thanks for pointing me back to this suggestion.
-Rob
On May 2, 2022, at 2:51 PM, Robert Casey @. @.>> wrote:
Good find. I tried this exact approach last week and it didn't seem to resolve the issue, at least with the JRE image. I found that I had to select a different image, which was a non-versioned 'jre-stretch' tag. I think that you have to look to newer builds of images to start seeing the arm64 support. Honestly, until I can get the executable to run, I won't be entirely sure that my selection of JRE image will even work. We'll see. Still working on the python/pip/matplotlib angle to see what it takes to get it to build. -RobOn May 2, 2022, at 2:27 PM, Autumn Johnson @. @.>> wrote:
Hi @rcasey-iris https://github.com/rcasey-iris Do you think this is related to the problem: https://stackoverflow.com/questions/69708866/how-to-make-docker-compose-work-on-m1-chip https://stackoverflow.com/questions/69708866/how-to-make-docker-compose-work-on-m1-chip If so, looks like a simply fix in the docker-compose file.
— Reply to this email directly, view it on GitHub https://github.com/iris-edu/yasmine-stationxml-editor/issues/9#issuecomment-1115386579, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL4VN7JSC2LNNBEPGMRUQDVIBCDDANCNFSM5UPQUZUQ. You are receiving this because you were mentioned.
autumnjohnson
commented
2 years ago Hi @rcasey-iris
Which container is failing to run the frontend or the backend? Looks like frontend then.
I just confirmed with Gillian that she was able to run it on her Mac. We could do a debugging session so I can see what's going on better in your terminal?
rcasey-earthscope
commented
2 years ago Sorry for the slow response.
It's the frontend that is now the issue. To catch up on where I am at.
I find that I can get a JRE 10 instance that is arm64 compatible. The issue now is that JAXB is not included in the JRE (by design), so I am stuck at what appears to be the sencha builder, specifically the Ant builder. It needs XML binding capability. So, I am attempting to inject a jaxb bind jar file into the build and point sencha to that file as part of its classpath.
Gillian is running an Intel Mac. I am glad that she is able to get it to run fine. My issue is specifically related to running on an M1 (arm64/v8) architecture. I feel like I am getting close to something. Soooo close.
-ROn May 4, 2022, at 1:34 PM, Autumn Johnson @.***> wrote:
Hi @rcasey-iris https://github.com/rcasey-iris Which container is failing to run the frontend or the backend?
I just confirmed with Gillian that she was able to run it on her Mac. We could do a debugging session so I can see what's going on better in your terminal?
— Reply to this email directly, view it on GitHub https://github.com/iris-edu/yasmine-stationxml-editor/issues/9#issuecomment-1117888221, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL4VN77MUSHBLW25K2TU43VILNOVANCNFSM5UPQUZUQ. You are receiving this because you were mentioned.
rcasey-earthscope
commented
2 years ago I have a working solution for arm64. Involves a couple of lines added to docker-compose.yml, where the frontend must build as platform linux/arm64 and the backend can continue to run as linux/amd64.
The Dockerfile for frontend requires pretty significant changes, including the use of a different source image, in order to run as arm64 AND support Java 8. The image that I found naturally is missing some key libraries, but I figured out how to get them and load it. Sencha Cmd is still the amd build, so it's an interesting cross-compile problem. I'm going to evaluate the running tool and see if it's stable and functional, then consult on the best process to introduce this alternative path to those running M1 Macs.
rcasey-earthscope
commented
2 years ago I think we can consider this issue resolved with the commits that Autumn is putting in place.
rcasey-earthscope
commented
1 year ago Yes, if you can do that, I think it would be a good experiment.
-RobOn May 2, 2022, at 12:27 PM, Autumn Johnson @.***> wrote:
Hi @rcasey-iris https://github.com/rcasey-iris Can we attempt to put out a docker image with the complete build from your platform and we can see if the M1 Mac can simply run it?
Are you asking me to deploy the image to a container registry like Docker Hub or GitHub Container Registry? The latter would be simplest and I can make up an update to the .github/yasmine.yml file to do so upon merges with the main branch.
— Reply to this email directly, view it on GitHub https://github.com/iris-edu/yasmine-stationxml-editor/issues/9#issuecomment-1115276669, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFL4VNYE4WI6QNEKHDUBOM3VIAUBNANCNFSM5UPQUZUQ. You are receiving this because you were mentioned.
Trying to build using the test-to-gh-workflow branch with docker compose. I'm not sure what the dependency issue is that is causing the failure.