v3.0-rc.4

[RaymondMouthaan]

v3.0-rc.4 for amd64, arm32v6, arm64v8

Hai all,

I just succeeded to build ARM32v6, ARM64v8 and AMD64 images for EMQX 3.0-rc.4. These images are based on Alpine 3.8 and Erlang 21.1.3 and includes tzdata #70 & #75

To build these images from Travis, please sent an email to support@travis-ci.com to request to increase the overall timeout! This is default set to 50 minutes and needs to be extended to at least 120 minutes!!! The build runs 3 jobs parallel, each for one image (arm32v6, arm64v8, amd64). The arm32v6 and arm64v8 take about 120 minutes to build, while amd64 takes about 15 minutes. This is because arm gets emulated by qemu and is slower. The final step is to create and push the manifest lists for these images. When using manifest list for your docker images there is no need to tag the image, since the images are manifest listed, which means, docker detect the architecture and pulls corresponding image automatically.

Howto release:

1. modify EMQX_VERSION and/or OTP_VERSION in .travis.yml
2. if needed modify OTP_DOWNLOAD_SHA256 in Dockerfile.alpine-tmpl
3. create a github release with a tag like:
   • v3.0-rc.4
   • v3.0-rc.4-testing
   • v3.0-rc.4-beta

The Docker.sh script check for testing and beta in the build version, if it exists it will create associated manifest list for it and leaves latest tag untouched.

Find below an example of a docker compose file I use to deployed with docker stack (note TZ is not used yet):

################################################################################
# MQTT Stack
################################################################################
#$ docker stack deploy mqtt --compose-file docker-compose-mqtt.yml
################################################################################
version: "3.6"

services:
  emqx-master:
    image: emqx/emqx
    hostname: emqx-master
    environment:
      - "EMQX_NAME=emq"
      - "EMQX_HOST=master.mq.tt"
      - "EMQX_NODE__COOKIE=ef16498f66804df1cc6172f6996d5492"
      - "EMQX_WAIT_TIME=60"
    networks:
      emqx-net:
        aliases:
          - master.mq.tt
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/TZ:ro
    deploy:
      placement:
        constraints: [node.role == manager]

  emqx-worker:
    image: emqx/emqx
    hostname: emqx-worker
    environment:
      - "EMQX_JOIN_CLUSTER=emq@master.mq.tt"
      - "EMQX_NODE__COOKIE=ef16498f66804df1cc6172f6996d5492"
      - "EMQX_WAIT_TIME=60"
    depends_on:
     - emqx-master
    networks:
      - emqx-net
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/TZ:ro
    deploy:
      placement:
        constraints: [node.role == worker]

networks:
  emqx-net:
    external: false

Hope you appreciate all the work and don't hesitate to contact me.

Regards, Raymond

[gilbertwong96]

Thanks for your pull requests. Maybe the base branch your forked is too old, we can not choose to rebase and merge. @zhanghongtong would review the code as soon as possible.

[RaymondMouthaan]

Conflicts!

should be fixed!

[gilbertwong96]

Anyway, appreciate your great contribution

[RaymondMouthaan]

Thanks @Gilbert-Wong :-)

As we speak I sent a mail to travis support to increase the overal build time out. When it's set to 150 min then the CI check should pass.

I'll keep you informed.

[RaymondMouthaan]

The hard time-out has been set to 150 min by Travis Team.

[gilbertwong96]

Cool!

[RaymondMouthaan]

Can we restart the Travis-CI, so we can pass the check?

[gilbertwong96]

Yes, I have restarted this job.

[RaymondMouthaan]

Nice, let's wait until merge and first release 👍🏽

[Rory-Z]

HI, sorry to let you wait so long, I executed./.docker/docker.sh build, but it seems that there are some problems

$ sudo -E ./.docker/docker.sh build
DOCKER BUILD: Build Docker image.
DOCKER BUILD: build version -> beta.
DOCKER BUILD: build from -> amd64/alpine:3.8.
DOCKER BUILD: os -> alpine.
DOCKER BUILD: arch - amd64.
DOCKER BUILD: qemu arch - x86_64.
DOCKER BUILD: otp version - 21.1.3.
DOCKER BUILD: emqx version - v3.0-rc.4.
DOCKER BUILD: docker file - Dockerfile.alpine-tmpl.
Sending build context to Docker daemon 249.3 kB
Step 1/26 : ARG BUILD_FROM
Please provide a source image with `from` prior to commit

[Rory-Z]

HI, sorry to let you wait so long, I executed./.docker/docker.sh build, but it seems that there are some problems

$ sudo -E ./.docker/docker.sh build
DOCKER BUILD: Build Docker image.
DOCKER BUILD: build version -> beta.
DOCKER BUILD: build from -> amd64/alpine:3.8.
DOCKER BUILD: os -> alpine.
DOCKER BUILD: arch - amd64.
DOCKER BUILD: qemu arch - x86_64.
DOCKER BUILD: otp version - 21.1.3.
DOCKER BUILD: emqx version - v3.0-rc.4.
DOCKER BUILD: docker file - Dockerfile.alpine-tmpl.
Sending build context to Docker daemon 249.3 kB
Step 1/26 : ARG BUILD_FROM
Please provide a source image with `from` prior to commit

I upgraded docker to the latest version, this problem does not appear again

[RaymondMouthaan]

Hi @zhanghongtong,

I assume you try it locally / manually?

As described in the first section of this PR, the project builds the docker images using TravisCi. Please read the description how to release a set of images for different architectures. Also you can have a look at the Travis jobs, since the jobs already ran once for checking.

Another thing to notice is that 4days ago a new commit has been done for Dockerfile in the root dir of the project. This file is no more needed when this PR is merged.

If you need more assistance, please let me know :-)

[Rory-Z]

Yes, I am trying to build an image locally, we will run the built image on the arm device to test the function, after that we will merge PR, thank you for your contribution

[RaymondMouthaan]

Yes, I am trying to build an image locally, we will run the built image on the arm device to test the function, after that we will merge PR, thank you for your contribution

If you want to build and run it locally before you merge it, then you make sure you set the appropriate environment vars before you run the command (normally travis.yml will do that for you).

docker build --no-cache \
    --build-arg BUILD_REF=${TRAVIS_COMMIT} \
    --build-arg BUILD_DATE=$(date +"%Y-%m-%dT%H:%M:%SZ") \
    --build-arg BUILD_VERSION=${BUILD_VERSION} \
    --build-arg BUILD_FROM=${BUILD_FROM} \
    --build-arg OS=${OS} \
    --build-arg ARCH=${ARCH} \
    --build-arg QEMU_ARCH=${QEMU_ARCH} \
    --build-arg OTP_VERSION=${OTP_VERSION} \
    --build-arg EMQX_VERSION=${EMQX_VERSION} \
    --file ./.docker/${DOCKER_FILE} \
    --tag ${TARGET}:build-${OS}-${ARCH} .

You just could run above command and fill in all the variables.

Another way is to use my ARM image (tag is optional) from here, which is exactly the same.

[Rory-Z]

Thanks for your reminder, I have successfully built the image of amd64, we are building the image of v3.0-rc.5, we added an environment variable EMQX_DEPS_DEFAULT_VSN in v3.0-rc.5, you can see here

[RaymondMouthaan]

If I may suggest, merge this PR and release with "testing" in it's version (see description), then test on AMD and/or ARM for its functionally.

Then make your changes environment variable EMQX_DEPS_DEFAULT_VSN, this has impact on travis.yml, docker.sh and the docker.alpine-tmpl. Then you could release another testing image for rc-5 to see if the build jobs still work.

Finally if all works well you could release a "final" version for v3.0-rc.5 and delete all testing images from Docker Hub if you want too :)

[spring2maz]

@RaymondMouthaan Thank you for your contribution. Any good reason to keep docker files and build scripts in a hidden dir ?

[RaymondMouthaan]

I am happy to help 😇

Hiding the docker dir has no special reason, but came from another project where the docker files are part of the application files and therefore we didn't want the dir visible. In your case you could remove the dot from its name and modify the scripts accordingly.

[spring2maz]

I'd prefer to have it in a regular dir. We would like to keep your name in the commit history, could you update it?

Technically we can merge and make the changes and squash the commits into your commit, However, I tried to fetch the commit from your repo but I can't seem to find the branch emqx30 in your repo, have you deleted it already ?

[RaymondMouthaan]

I am omw, but this? https://github.com/RaymondMouthaan/emqx-docker-1

[spring2maz]

right, I didn't notice it's the -1 repo. ;)

[spring2maz]

Changed base to build-args branch and merged this PR. I'll rename dir and perhaps apply some minor updates on top and send another PR to have it merged to emqx30

[RaymondMouthaan]

Glad to see it merged 😇.

Earlier I noticed something goes wrong with the persistence of the emqx dirs. I will investigate and do a PR if needed. I'll keep you guys informed.

Let me know when the first release to docker hub is done, then I can remove my own repos on github and docker hub and use the official 👍🏽

Btw thanks for this great peace of software!

[Rory-Z]

@RaymondMouthaan Hi, I set the following environment variables, then I ran nohup sudo -E ./.docker/docker.sh build&, now the compilation has been running for 9 hours, but still no success and no error, is this normal? My machine is AWS EC2 t2.medium

export  EMQX_VERSION=v3.0-rc.5
export  BUILD_VERSION=v3.0-rc.5
export  OTP_VERSION="21.1.3"
export  TARGET=emqx/emqx
export  QEMU_VERSION=v3.0.0
export  DOCKER_FILE=Dockerfile.alpine-tmpl
export  OS=alpine 
export  BUILD_FROM=arm32v6/alpine:3.8  
export  QEMU_ARCH=arm 
export  ARCH=arm32v6

[RaymondMouthaan]

@zhanghongtong hi, no that's not normal... From my experience building the arm images could take longer than amd because the emulation done by qemu, but not as long as 9 hour and still ongoing. Rc-4 for arm took about 2 hours on the Travis server. This difference of 7hours could be either the changes between rc-4 and rc-5 or between the Travis and AWS server. Reducing the build time could be accomplished by using the Erlang base image instead of compiling Erlang it self, this process is quite time consuming. But I understood earlier that using the Erlang base image increases the final Emqx image.

You might want to build with the Erlang base image to test whether it's faster to build and check it's final size. Maybe "things" could be removed from it which are not needed by emqx? Also building the current docker file as is but on Travis to see if it is still about 2 hours with rc-5, in that case it must be the difference between AWS and Travis.

A side note: the max time-out that the Travis team can set is 3 hours, so each job should finish before otherwise it gets aborted.

Compared to other projects I build for arm Emqx takes the longest time so far. And I have no idea if it can be faster somehow...

Greets, Raymond

[RaymondMouthaan]

@zhanghongtong As we speak I have Travis jobs running to build rc-5, to check it's build time for arm. You can check it's process here https://travis-ci.org/RaymondMouthaan/emqx-docker-1/builds/463696072

[spring2maz]

agree to use base image instead. The size increase is mostly because of docker layer increase (delete things would help, but very little) We may choose to squash the layers after build

[Rory-Z]

@RaymondMouthaan We have added an environment variable EMQX_DEPS_DEFAULT_VSN to compile v3.0-rc.5, you can see this change in this PR,

[RaymondMouthaan]

@zhanghongtong yes I know :) but as you can see the job fails on my travis https://travis-ci.org/RaymondMouthaan/emqx-docker-1/jobs/463696073, somewhere around line 9089. This shouldn't have to do anything with new EMQX_DEPS_DEFAULT_VSN environment var right?

[Rory-Z]

@RaymondMouthaan I am not sure if this failure is caused by the lack of EMQX_DEPS_DEFAULT_VSN, but I suggest you try it after adding EMQX_DEPS_DEFAULT_VSN.

[RaymondMouthaan]

The earlier reason of failure was indeed the EMQX_DEPS_DEFAULT_VSN env variable. I added it to my own project to test whether the images get build for rc-5. As I am writing this a release is running to push rc-5 to my Docker hub. The progress can be seen here

@zhanghongtong Just curious, why did you add docker save in here?

[Rory-Z]

@RaymondMouthaan In addition to publishing our images on the docker hub, we will also provide downloads of docker images on our website, so we need to export the docker image as a file.

[RaymondMouthaan]

Just to keep you updated from my site -- I've successfully build v3.0-rc.5 images for arm32v6, arm64v8 and amd64. They are available at https://hub.docker.com/r/raymondmm/emqx/tags/ for testing purpose. While building the images, I faced the issue of max silence timeout on the travis server and requested Travis support team to increase the timeout to 20 minutes (default 10 minutes). After these changes the build went successful. I requested the Travis team to do the same for emqx/emqx-docker. Waiting for there feedback :)

[Rory-Z]

Thank you for your help, we have successfully built the image on travis, and we have successfully built the image on the local server. In addition, I noticed that most of the time I was building the image wasted in the preparation environment. I tried to compile emqx with erlang:21-alpine, and then took the compiled emqx file and put it in another image built by alpine:3.8. On, just like this Github repository does. I am thinking, is it possible to arm32 and arm64 can also do this, we can use arm64v8:21.1-alpine to compile emqx, and then run the compiled file in arm64v8/alpine:3.8

[RaymondMouthaan]

@zhanghongtong, sounds like a great idea! The binaries for emqx could be available too on emqx.io website for anyone who doesn't use Docker.

Meanwhile I just got confirmation from the Travis support team, that the silence timeout has been increase to 20 min, although this might not be needed anymore with your new idea. I think all travis modified timeouts should be reset once the new solution works.

In my humble opinion, I think It would be nice that the docker repo should be merged with the main emqx repo (emqx-rel). When this repo releases a new GitHub release, it should compile all binaries for each platform you support (including arm). Also the docker images are build and pushed from this repo using the binaries. Main advantage of this, is that all are triggered by one repo. Anyway, its just an idea ...

-- off-topic Meanwhile I tried to persist the emqx directories below, but got errors trying.

• /opt/emqx/log
• /opt/emqx/data
• /opt/emqx/lib
• /opt/emqx/etc

I created empty directories log, data, lib and etc on my host system and used the following command to start emqx:

Test log persistence only: docker run --rm -ti -v $(pwd)/emqx/log:/opt/emqx/log --name emqx -p 18083:18083 raymondmm/emqx:latest

Test data persistence only: docker run --rm -ti -v $(pwd)/emqx/data:/opt/emqx/data --name emqx -p 18083:18083 raymondmm/emqx:latest

Test lib persistence only: docker run --rm -ti -v $(pwd)/emqx/lib/opt/emqx/lib --name emqx -p 18083:18083 raymondmm/emqx:latest

Test etc persistence only: docker run --rm -ti -v $(pwd)/emqx/etc:/opt/emqx/etc --name emqx -p 18083:18083 raymondmm/emqx:latest

Only log persistence worked, all other 3 commands failed because emqx couldn't find certain files/directory. I was expecting that emqx created these, when they don't exist and when they do that it uses it. I might do something wrong here, could you point me in the right direction if I do?

[Rory-Z]

@RaymondMouthaan Hi, We try to compile the image of arm32 on the local server, it will run for more than one day, and it will always be stuck in the step of compiling erlang, neither successful nor failure, which makes me particularly confused, what advice do you have?

[Rory-Z]

@RaymondMouthaan In addition, I don't understand what does prepare_qemu() in docker.sh. I saw that he resets multiarch/qemu-user-static:register (although I don't know when to create multiarch/qemu-user-static). Then downloaded and unzipped the three tar files in the tmp directory, but I didn't see where the three files were used.

[RaymondMouthaan]

@zhanghongtong , I have no idea why it takes so long on your local machine. I haven't tried locally my self yet. I am always using TravisCi for building the images.

One way or another, compiling Erlang takes a lot of time and like I said earlier I recommend the Erlang base image since it saves a lot of time.

You might wanna read this article http://www.hotblackrobotics.com/en/blog/2018/01/22/docker-images-arm/ which explains qemu.

Btw any idea about my question regarding volume persistence?

Cheers, Ray

[Rory-Z]

@RaymondMouthaan Sorry, I remember I replied to your question, but I just saw no reply. When the docker run -v command is executed, docker will overwrite the container's directory with the host's directory. When the mounted host directory is empty, emqx will not start properly.

Also, I have tried using arm32v7 / erlang:21-slim and arm64v8 / erlang:21-slim, but they all report an error:standard_init_linux.go:190:exec user process causes "no such file or directory", I guess they May not be able to run directly on amd64 devices

Thank you for your article

[RaymondMouthaan]

@zhanghongtong, no problem :-) Regarding emqx persistence, I would expect a little different behavior. I know about the override and would expect a check whether required files exists, if not then emqx creates them and starts. On the other hand, if they do, then creation is not required and emqx uses the existing files/folders. This way one doesn't need to supply required files/folders, because emqx creates them initially. Based on these one can change / add specific requirements. If you agree I can do PR for start.sh.

Can you send me a docker file which uses Erlang as a base image + requirements needed for emqx to run properly? Then I can have a look as well.

Greetz, Ray

[Rory-Z]

@RaymondMouthaan The files in the emqx directory are created when the code is compiled, so when emqx starts, it does not automatically create the required files. For this question, you need to consult our developers. @Gilbert-Wong Regarding the docker image running emqx, if you only install the emqx package, you do not need to install erlang. Erlang is used to compile emqx code. As I said before, if you separate emqx from build and build, This Dockerfile can provide a build environment. This Dockerfile is the simplest runtime environment.

[RaymondMouthaan]

@zhanghongtong thanks for your reply. Hope @Gilbert-Wong has a nice idea regarding persistence as described I described in my previous post.

So basically your idea here would be a very nice solution, this way the docker images won't need Erlang for compiling Emqx, only the location where the binaries are available. This would make building the Emqx images superfast and my guess a very small image size. Any idea how to implement this approach?

[Rory-Z]

@RaymondMouthaan My previous idea was to create a Dockerfile to prepare the erlang build environment, then use this image to create the container, clone the code in the container and compile the code, copy the compiled code to the host by container mount or scp. Then, create a Dockerfile, COPY the compiled directory on the host machine, just like this shell script does.

But now there is a problem, if my server is amd64, then I can't run the container of arm32 and arm64.

[RaymondMouthaan]

@zhanghongtong

But now there is a problem, if my server is amd64, then I can't run the container of arm32 and arm64.

Why not use Travis for that and store compiled binaries to github, where we can download from to build the docker images?

In steps: 1) emqx creates the binaries for all systems that are supported (ARM included). Use Travis for that. 2) emqx-docker uses those binaries to build associated docker-images and pushes them to Docker Hub and make them available too on the main website.

[Rory-Z]

Because we have been using Jenkins to compile code on our own server, such as nightly build, but it seems to be a good idea to use Travis, which can save our server resources. I can try to use Travis instead of Jenkins. We can download our compiled packages on our github release page, but there is no alpine version at the moment, and the packages we are currently publishing are compiled under the amd64 environment.

[RaymondMouthaan]

Aha, now I understand - did the article I send you earlier, help you to build for ARM? Two weeks ago I succeeded to build an ARM image on macOS, which runs fine on my Raspberry Pi Docker swarm. The is a Java based application -- I can give it a try to do the same for Emqx ... I can imagine that moving from Jenkins to Travis gives a lot of work, maybe the whole pipeline (if there is any) should be created in Travis. One approach could be to do only ARM binaries by Travis and leave all others to Jenkins for the time being. Although I am convinced that if it works on Travis, it also should work on Jenkins or even locally on AMD64.

[Rory-Z]

There is still a lot of work to do to migrate the work of compiling code from Jenkins to Travis, but it's still a good idea to try this, we'll put it into the plan and make sure it's executed.

I successfully compiled emqx with arm32v7/erlang:21-slim as the base image, thanks to the article you sent me. But I don't have an arm32 device to run him now, I will test it tomorrow for working hours. But as I tried before, I still can't install erlang on arm32 / alpine:3.8.

I look forward to your running emqx code on the Raspberry Pi, thank you for your participation.

[RaymondMouthaan]

hi @zhanghongtong, @Gilbert-Wong

Any idea when branch build-args-new-zhanght gets merged with emqx30? The latest docker version emqx/emqx doesn't support manifest list :(

[Rory-Z]

@RaymondMouthaan Thank you for your attention, this is my work mistake, I did not upload the manifest list, I will upload the manifest list in the work behind. About the branch build-args-new-zhanght merged with emqx30, the current branch build-args-new-zhanght still needs to be modified. We will build the emqx edge version later. I plan to use branch build-args-new-zhanght to support build. The edge version and the normal version of emqx, in addition, the size of the docker image compiled by the branch build-args-new-zhanght I hope will be smaller, these are the work that needs to be done. Finally, thank you for your contribution, emqx/emqx:v3.0.0-arm64, emqx/emqx:v3.0.0-arm32v6, emqx/emqx:v3.0.0-arm64v8 is the code build you contributed, thank you very much

[RaymondMouthaan]

@zhanghongtong, Glad I could help to improve Emqx Docker images! I hope to see branch build-args-new-zhanght soon, so that all releases are manifest listed. I'd like to make use of the official emqx docker repo and delete my own emqx stuff here, here and here.

Greetz, Ray