Closed throwaway-a closed 2 years ago
Hi @throwaway-a! It's not clear to me which image is causing the "pull" command to fail. Can you please try to run:
tutor images pull openedx
tutor images pull mysql
...
until you find a culprit?
It appears to fail on "tutor images pull mysql" I tried all the other things in docker-compose.yml and docker-compose.prod.yml and they seemed to pull fine, with only mysql failing.
This issue seems to match the problem reported elsewhere: https://github.com/docker-library/mysql/issues/778 Let's try to narrow it down. Does any one of those commands succeed?
docker pull mysql:5.7.35
docker pull --platform linux/amd64 mysql:5.7.35
What I do not understand is why other images do not cause the same issue.
EDIT: It turns out that mysql is one of the few images that do not provide arm-compatible images.
Not all images are available for ARM64 architecture. You can add --platform linux/amd64 to run an Intel image under emulation. In particular, the mysql image is not available for ARM64. You can work around this issue by using a mariadb image. (source)
See the upstream issues: https://github.com/docker-library/mysql/issues/318 https://bugs.mysql.com/bug.php?id=103462
docker pull --platform linux/amd64 mysql:5.7.35
does work on both macOS and linux, but then when I go back to running tutor quickstart, it will just complain again about there being no linux/arm64/v8 image.
mariadb is a fork of mysql so it should be compatible right? Maybe it's worth trying that out in the future?
@throwaway-a are you sure myswl works for you? I have the same issue with Apple M1 but I am getting the same problem with pulling mysql image directly:
$docker pull mysql:5.7.35
5.7.35: Pulling from library/mysql
no matching manifest for linux/arm64/v8 in the manifest list entries
EDIT: Sorry @throwaway-a I misread your message. I will also specify platform
. MariaDB also works on M1
I am not so keen on migrating from MySQL to Mariadb. We are currently running MySQL 5.7.33 in Open edX, and according to my research (including the mariadb docs and this comment) there are compatibility issues between this version and Mariadb.
We cannot switch to the mysql-server Docker image because the 5.7 tags do not support arm64.
I am not keen either on building the mysql image ourselves from arm64-compatible base images, because we would have to replicate the complex entrypoint mechanism from the base mysql image.
Thus, I suggest that we force linux/amd64 emulation on M1. These laptops are not meant for production anyway, so we shouldn't care too much about an eventual performance drop.
@Alain1405 @throwaway-a please test the following change. In $(tutor config printroot)/env/local/docker-compose.yml, replace:
mysql:
image: docker.io/mysql:5.7.33
by:
mysql:
image: docker.io/mysql:5.7.33
platform: linux/amd64
(ref: docker-compose spec)
Then, run tutor local start -d
and tutor local init
. (NOT tutor local quickstart
or tutor config save
as these commands would override your changes)
Please report here if this works for you. If yes, I'll open a PR.
@Alain1405 @throwaway-a please test the following change. In $(tutor config printroot)/env/local/docker-compose.yml, replace:
@regisb I tested that today (platform: linux/amd64
for the MySQL service in the Tutor compose file) and it does work.
However, if you do open a PR to add that, please make it a setting that can be controlled by a variable (DOCKER_IMAGE_PLATFORM_MYSQL
? USE_QEMU_MYSQL
? DOCKER_COMPOSE_EXTRA_MYSQL
?). As I would still prefer to use mariadb
for now, at least until I encounter any actual incompatibilities, and a variable that can be configured with Tutor is much easier than a custom docker-compose.override.yml
file.
For reference, tutor local init
(i.e. running migrations) took 8m41s with mysql amd64 and qemu emulation, whereas the same command took only 4m48s with mariadb and a native arm64/v8 image. I expect tests and other platform functionality would have a similar slowdown (almost 50% slower).
Today I got Tutor running on my M1 Mac. It was delightfully easy (other than this minor issue).
Assuming one wants Tutor installed for dev and the latest Open edX named release available in prod or dev mode, here are the complete instructions, which is what I used:
# Set up a new folder for the tutor devstack
mkdir tutor-devstack
cd tutor-devstack
# Set up python - assuming you have python 3.8 installed (brew install python@3.8)
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
# Install Tutor
git clone https://github.com/overhangio/tutor.git
cd tutor
pip install -r requirements/dev.txt
pip install -e .
# Patch Tutor - unless https://github.com/overhangio/tutor/pull/543 is merged
git checkout -b braden/arm64buildfix master
git pull https://github.com/open-craft/tutor.git braden/arm64buildfix
# Configure Tutor - say no to the first question
tutor config save --interactive
# Build the Open edX images for ARM64
tutor images build openedx
tutor images build openedx-dev
# Switch to MariaDB since it supports ARM64 (MySQL only supports it for v8+, e.g. with
# image "mysql/mysql-server:8.0-aarch64" which I tried but it didn't work out of the box.)
tutor config save --set DOCKER_IMAGE_MYSQL=mariadb:10.4
# Configure Open edX (set database auth, run migrations, etc.)
tutor local init
tutor local stop
# Run Open edX in dev mode
tutor dev runserver lms
# Now access it at http://local.overhang.io:8000/
After I went through all of this, I realized that I should have been using the nightly
version because what I'm really trying to do is local dev, so next up on my list is to try nightly. I'll report back with any findings.
Hey @bradenmacdonald thanks for the comprehensive explanations! Considering the results of the benchmarks you made, I agree with you that the best way to support the arm64 platform is to advise users to switch to MariaDB. They would also have to build the tutor-specific Docker images themselves, thanks to that #543 PR you opened. Would you have time to write a tutorial on how to run Tutor on arm64/Apple M1? If not I can do it myself, based on your testing and reports.
@regisb Yes, I was thinking of writing a tutorial anyways. I just want to try the nightly mode, and then I can hopefully write a combined tutorial that works for people who want the stable version or nightly. Would you prefer I write it as a forum post or open a PR against the documentation?
That's great, thanks a lot.
Would you prefer I write it as a forum post or open a PR against the documentation?
Whichever you are most comfortable with. You have a good understanding of how Tutor works, so I guess you don't necessarily have to go through the forum for validation first. Feel free to add a commit on top of your PR, or open a different one.
@Alain1405 @throwaway-a please test the following change. In $(tutor config printroot)/env/local/docker-compose.yml, replace:
@regisb I tested that today (
platform: linux/amd64
for the MySQL service in the Tutor compose file) and it does work.However, if you do open a PR to add that, please make it a setting that can be controlled by a variable (
DOCKER_IMAGE_PLATFORM_MYSQL
?USE_QEMU_MYSQL
?DOCKER_COMPOSE_EXTRA_MYSQL
?). As I would still prefer to usemariadb
for now, at least until I encounter any actual incompatibilities, and a variable that can be configured with Tutor is much easier than a customdocker-compose.override.yml
file.For reference,
tutor local init
(i.e. running migrations) took 8m41s with mysql amd64 and qemu emulation, whereas the same command took only 4m48s with mariadb and a native arm64/v8 image. I expect tests and other platform functionality would have a similar slowdown (almost 50% slower).Guide to running Tutor in dev mode on an M1 Mac
Today I got Tutor running on my M1 Mac. It was delightfully easy (other than this minor issue).
Assuming one wants Tutor installed for dev and the latest Open edX named release available in prod or dev mode, here are the complete instructions, which is what I used:
# Set up a new folder for the tutor devstack mkdir tutor-devstack cd tutor-devstack # Set up python - assuming you have python 3.8 installed (brew install python@3.8) python3 -m venv venv source venv/bin/activate pip install --upgrade pip # Install Tutor git clone https://github.com/overhangio/tutor.git cd tutor pip install -r requirements/dev.txt pip install -e . # Patch Tutor - unless https://github.com/overhangio/tutor/pull/543 is merged git checkout -b braden/arm64buildfix master git pull https://github.com/open-craft/tutor.git braden/arm64buildfix # Configure Tutor - say no to the first question tutor config save --interactive # Build the Open edX images for ARM64 tutor images build openedx tutor images build openedx-dev # Switch to MariaDB since it supports ARM64 (MySQL only supports it for v8+, e.g. with # image "mysql/mysql-server:8.0-aarch64" which I tried but it didn't work out of the box.) tutor config save --set DOCKER_IMAGE_MYSQL=mariadb:10.4 # Configure Open edX (set database auth, run migrations, etc.) tutor local init tutor local stop # Run Open edX in dev mode tutor dev runserver lms # Now access it at http://local.overhang.io:8000/
After I went through all of this, I realized that I should have been using the
nightly
version because what I'm really trying to do is local dev, so next up on my list is to try nightly. I'll report back with any findings.
I just tried and this worked great for me on maple
branch. Thanks @bradenmacdonald !
I am trying to create a superuser with tutor local createuser --staff --superuser username email
now but I am stumbling upon:
django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'mysql:3306' (111)")
and I think it might be related with the name of the mysql container being different, but I am not sure since LMS seems to run just fine.
and I think it might be related with the name of the mysql container being different, but I am not sure since LMS seems to run just fine.
@Alain1405 This is a completely unrelated issue. The mysql container should still have the same name, despite being a different image. I suspect that you are running this "local" command when mysql is actually run in "dev" mode. Since "dev" and "local" containers live in different networks (because the docker-compose project names are different), they cannot communicate with one another. Check which containers are running with docker ps
. Stop all "local" containers (with tutor local stop
). Then, run the command in dev mode: tutor dev createuser ...
. If this does not resolve your issue, open a new topic in the forum.
and I think it might be related with the name of the mysql container being different, but I am not sure since LMS seems to run just fine.
@Alain1405 This is a completely unrelated issue. The mysql container should still have the same name, despite being a different image. I suspect that you are running this "local" command when mysql is actually run in "dev" mode. Since "dev" and "local" containers live in different networks (because the docker-compose project names are different), they cannot communicate with one another. Check which containers are running with
docker ps
. Stop all "local" containers (withtutor local stop
). Then, run the command in dev mode:tutor dev createuser ...
. If this does not resolve your issue, open a new topic in the forum.
You are so true @regisb 🤦🏻 . That resolved but I stumbled upon a new issue:
Created new user: "alain"
Setting is_staff for user "alain" to "True"
Setting is_superuser for user "alain" to "True"
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 393, in __get__
rel_obj = self.related.get_cached_value(instance)
File "/openedx/venv/lib/python3.8/site-packages/django/db/models/fields/mixins.py", line 13, in get_cached_value
return instance._state.fields_cache[cache_name]
KeyError: 'profile'
...
django.db.utils.OperationalError: (1054, "Unknown column 'auth_userprofile.allow_certificate' in 'field list'")
I'm using all default flags to run this dev environment. I guess I could try disabling Course Certificates. It might also be that some migrations did not run correctly.
FYI, I was able to get both tutor/maple and tutor/maple-mfe up and running this weekend; will follow up with details later today. Required adding a few dependencies for MFE to build properly (and excluding Pact which doesn't yet have an arm64 binary), but overall extremely easy with @bradenmacdonald's how-to.
OK, instructions in case anyone is trying to replicate a Maple build on M1:
mkdir tutor-devstack
cd tutor-devstack
# Set up Python environment (assumes you have Python 3.8 installed; I like using pyenv for this)
python -m venv venv
source venv/bin/activate
pip install --upgrade pip
# Build exim-relay image for ARM64 (used by smtp service)
git clone https://github.com/devture/exim-relay
cd exim-relay
git checkout 4.94.2-r0-4
make build IMAGE_VERSION=4.94.2-r0-4
cd ..
# Clone patched tutor and tutor-mfe branches
git clone -b maple-m1 https://github.com/msegado/tutor
git clone -b maple-m1 https://github.com/msegado/tutor-mfe
# Install tutor and tutor-mfe
pip install -e tutor -e tutor-mfe
# Configure tutor
tutor plugins enable mfe
tutor config save --interactive # answer no to first question
# Switch to MariaDB (see @bradenmacdonald's post)
tutor config save --set DOCKER_IMAGE_MYSQL=mariadb:10.4
# Build images for ARM64
tutor images build openedx
tutor images build permissions
tutor images build mfe
# Initialize and start platform
tutor local init
tutor local start
After this, you should have a running Maple (beta) platform, complete with working MFEs. Haven't done too much testing but everything seems to work, including the Learning MFE.
The patches themselves are pretty simple. For tutor
, all that's required is https://github.com/overhangio/tutor/pull/543 (rebased on top of maple). For tutor-mfe
, some more build requirements and a couple of environment variables are needed:
diff --git a/tutormfe/templates/mfe/build/mfe/Dockerfile b/tutormfe/templates/mfe/build/mfe/Dockerfile
index 726d9ca..b996eea 100644
--- a/tutormfe/templates/mfe/build/mfe/Dockerfile
+++ b/tutormfe/templates/mfe/build/mfe/Dockerfile
@@ -3,7 +3,18 @@ FROM docker.io/node:12-bullseye-slim AS base
RUN apt update \
&& apt install -y git \
# required for cwebp-bin
- gcc git libgl1 libxi6 make
+ gcc git libgl1 libxi6 make \
+ # additionally required for gifsicle, mozjpeg, and optipng
+ autoconf libtool pkg-config zlib1g-dev \
+ # additionally required for node-sass
+ python g++
+
+# Required for building optipng on M1
+ENV CPPFLAGS=-DPNG_ARM_NEON_OPT=0
+
+# Required for avoiding failed frontend-app-learning build due to missing binary on M1
+# (breaks pact tests)
+ENV PACT_SKIP_BINARY_INSTALL=true
RUN mkdir -p /openedx/app /openedx/env
WORKDIR /openedx/app
The first environment variable is required to fix the issue described here: https://github.com/imagemin/optipng-bin/issues/117#issuecomment-953182111
Unfortunately the second is a hack that allows MFE to build but will break any Pact tests. I don't see much alternative until there's an ARM binary for Pact though. Relevant issue: https://github.com/pact-foundation/pact-js-core/issues/347
Hope that helps someone, Martin
@bradenmacdonald @msegado @Alain1405 Your efforts to get Tutor to work on ARM64 have convinced me that we need to support this platform. However, while I appreciate the effort to write a comprehensive tutorial, we need to get to a point where users do not have to apply complex manual patches to their installation to get it to work. At some point we may have to make upstream changes, for instance to MFEs, to make sure that they install properly on ARM64. One option would be to get rid of some packages that do not support ARM64, such as pact.
I'm also trying to get Tutor working on my Apple M1 machine, and running into this error after making the suggested change to docker-compose.yml
(as described here: https://github.com/overhangio/tutor/issues/510#issuecomment-957253941)
$ tutor local start -d
docker-compose -f /Users/nateaune/Library/Application Support/tutor/env/local/docker-compose.yml -f /Users/nateaune/Library/Application Support/tutor/env/local/docker-compose.prod.yml --project-name tutor_local up --remove-orphans --build -d
tutor_local_elasticsearch_1 is up-to-date
tutor_local_smtp_1 is up-to-date
Recreating tutor_local_mysql_1 ...
tutor_local_mongodb_1 is up-to-date
tutor_local_redis_1 is up-to-date
tutor_local_caddy_1 is up-to-date
Recreating tutor_local_mysql_1 ... done
tutor_local_lms_1 is up-to-date
tutor_local_lms-worker_1 is up-to-date
tutor_local_cms_1 is up-to-date
tutor_local_cms-worker_1 is up-to-date
tutor_local_nginx_1 is up-to-date
$ tutor local init
Initialising all services...
docker-compose -f /Users/nateaune/Library/Application Support/tutor/env/local/docker-compose.yml -f /Users/nateaune/Library/Application Support/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /Users/nateaune/Library/Application Support/tutor/env/local/docker-compose.jobs.yml run --rm mysql-job sh -e -c echo "Initialising MySQL..."
mysql_connection_max_attempts=10
mysql_connection_attempt=0
until mysql -u root --password="NME7mivi" --host "mysql" --port 3306 -e 'exit'
do
mysql_connection_attempt=$(expr $mysql_connection_attempt + 1)
echo " [$mysql_connection_attempt/$mysql_connection_max_attempts] Waiting for MySQL service (this may take a while)..."
if [ $mysql_connection_attempt -eq $mysql_connection_max_attempts ]
then
echo "MySQL initialisation error" 1>&2
exit 1
fi
sleep 10
done
echo "MySQL is up and running"
# edx-platform database
mysql -u root --password="NME7mivi" --host "mysql" --port 3306 -e 'CREATE DATABASE IF NOT EXISTS openedx;'
mysql -u root --password="NME7mivi" --host "mysql" --port 3306 -e 'GRANT ALL ON openedx.* TO "openedx"@"%" IDENTIFIED BY "NME7mivi";'
Creating tutor_local_mysql-job_run ... done
Initialising MySQL...
mysql: [Warning] Using a password on the command line interface can be insecure.
MySQL is up and running
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 3009 (HY000) at line 1: Column count of mysql.user is wrong. Expected 45, found 43. Created with MySQL 50636, now running 50733. Please use mysql_upgrade to fix this error.
ERROR: 1
Error: Command failed with status 1: docker-compose -f /Users/nateaune/Library/Application Support/tutor/env/local/docker-compose.yml -f /Users/nateaune/Library/Application Support/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /Users/nateaune/Library/Application Support/tutor/env/local/docker-compose.jobs.yml run --rm mysql-job sh -e -c echo "Initialising MySQL..."
mysql_connection_max_attempts=10
mysql_connection_attempt=0
until mysql -u root --password="NME7mivi" --host "mysql" --port 3306 -e 'exit'
do
mysql_connection_attempt=$(expr $mysql_connection_attempt + 1)
echo " [$mysql_connection_attempt/$mysql_connection_max_attempts] Waiting for MySQL service (this may take a while)..."
if [ $mysql_connection_attempt -eq $mysql_connection_max_attempts ]
then
echo "MySQL initialisation error" 1>&2
exit 1
fi
sleep 10
done
echo "MySQL is up and running"
# edx-platform database
mysql -u root --password="NME7mivi" --host "mysql" --port 3306 -e 'CREATE DATABASE IF NOT EXISTS openedx;'
mysql -u root --password="NME7mivi" --host "mysql" --port 3306 -e 'GRANT ALL ON openedx.* TO "openedx"@"%" IDENTIFIED BY "NME7mivi";'
@natea It looks to me like you have used two different versions of MySQL (Created with MySQL 50636, now running 50733
), and the on-disk/on-volume data file format is not compatible. If there's no important data on your tutor instance, the easiest fix is just to stop everything, delete the MySQL data folder/volume and then run the init again. If there is important data you'll need to use mysql_upgrade.
@regisb
However, while I appreciate the effort to write a comprehensive tutorial, we need to get to a point where users do not have to apply complex manual patches to their installation to get it to work.
Agreed :)
At least for the non-MFE parts, if you'd be willing to publish arm64v8 images on Docker Hub, no special instructions would be necessary, it would "just work". The instructions mostly cover how to build the docker images from scratch manually. I believe docker buildx
is the tool for building multi-platform images, and I believe it does not require you to use an ARM device to build the images, but I haven't tried it myself.
(Edit: Oh, I forgot about MySQL - I guess a workaround would still be needed there unless we add one by default).
@regisb i suspect that the MySQL running in the Docker container is from a previous version of Tutor, which I don't need anymore. How do I delete that MySQL data folder/volume?
I think it's acceptable if people just have to perform the following steps:
Rebuilding images is something that many people have to do anyway, because of some theming or because they are running on nightly for instance. I tried building the openedx image with buildx and it failed, for some reason, so we would have to setup a cross-architecture CI to publish ARM64 images -- and this is going to take a while, if ever... What I would like to avoid is people applying custom patches on top of tutor or its plugins.
EDIT: I just noticed that pyenv gained support for arm64 in v2.2.2: https://github.com/pyenv/pyenv/releases/tag/v2.2.2 this might make it slightly more easy to cross-build images.
EDIT2: nope. Build of "openedx" image is failing on apt install:
...
#7 220.0 Processing was halted because there were too many errors.
#7 220.1 E: Sub-process /usr/bin/dpkg returned an error code (1)
------
Dockerfile:6
--------------------
5 | ENV DEBIAN_FRONTEND=noninteractive
6 | >>> RUN apt update && \
7 | >>> apt install -y build-essential curl git language-pack-en
8 | ENV LC_ALL en_US.UTF-8
--------------------
error: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c apt update && apt install -y build-essential curl git language-pack-en" did not complete successfully: exit code: 100
Would it also be something to think about switching from MySQL to MariaDB, or what would it be possible to upgrade to MySQL 8 to better support arm and amd64?
As support arm would be great for those running Tutor in AWS, where we could switch to amazon's arm offering.
@regisb: Not to belabor the obvious, but simple, out-of-the-box support for ARM64 could help build up a user base of edX employees who don't really need all the devstack edge cases and don't want to fight with ARM64+devstack on their shiny new M1 laptops over the coming months. 😛
I mean, don't get me wrong, it's already pretty great with @bradenmacdonald's PR. I didn't mean this to imply that you needed to do a lot of work here–I just wanted to highlight the marketing opportunity.
I'm not an edX employee, but what @ormsbee describes is what happened to me - because of my M1 MacBook I gave up on edx/devstack and switched to Tutor.
For those coming late to this thread, the current most up-to-date docs are in this PR: https://github.com/overhangio/tutor/pull/549
Let's close this now that #549 is merged. Please comment here or open a new issue if problems arise on arm64.
Hi I would like to add something that related to this issue. Everybody is not technical persons , some are users like me. This issue sucks on M1, and M2 chips .. I'm very new to M2 and this issue sucks me. I was using Ubuntu and that was fucking good in my previous days. When I sent my ubuntu laptop to repair shop, the time has come to use M2 mac, this issue comes and fucks me. My client fucking hate me and wanted to kill me. I was being late fucking 3 days
Hi @kamkyi, thanks for your constructive comment. It is very helpful. We'll pass it on to our CHO🍎U (Chief Happiness Officer for Apple Users). Next time you feel the need to rant please don't hesitate to post on Twitter, Facebook, or your social networks of choice. Basically anywhere except here on GitHub.
I recently tried installing on Ubuntu ARM64 (On AWS) and tutor tried to install MySQL 5.7 which is unsupported although open edX already supports MySQL which has a great ARM64 support, the installation failed for many reasons and now I don't have access to the server anymore so I need to do this whole thing from the top.
Is your feature request related to a problem? Please describe. When I tried to install tutor on an Apple M1 Mac, I got the following error:
Which I assume just means there's no preconfigured docker containers which run on ARM64.
Describe the solution you'd like
Support for ARM64 macOS native or Linux VM installs.
Describe alternatives you've considered
Only run tutor on x86 systems.
Additional context Logs: