jukie commented 3 years ago

I'd like to contribute to this project by adding multi-arch build support instead of individual arch tagged images. I run a mixed arch homelab cluster and having the ability to run this on any node would be great.

What's the current process you take to build/push images to DockerHub? All of the recent builds have failed through TravisCI and I didn't see a push happening anywhere there so I was having trouble finding this piece.

Would you be open to me implementing something like https://www.docker.com/blog/multi-arch-build-what-about-travis/?

jacobalberty commented 3 years ago

What's the current process you take to build/push images to DockerHub?

I am building on the docker hub itself using the automated builds or whatever they're calling it this week.

All of the recent builds have failed through TravisCI

Yeah the Travis builds broke a bit ago, IIRC the unifi webserver started taking longer to bring come up and I couldn't decide on a reliable way of determining a working build so I just hadn't fixed it.

sleep 10 && curl -kILs --fail https://127.0.0.1:8443 || exit 1 curl: (35) gnutls_handshake() failed: The TLS connection was non-properly terminated.

Shouldn't be too hard to fix, easiest options are just increase the sleep before hand and continue using that command, or drop the curl altogether and rely on ps and logs as indications of a working build

Would you be open to me implementing something like https://www.docker.com/blog/multi-arch-build-what-about-travis/?

Using travis would be the easiest route by far

jacobalberty commented 3 years ago

I believe I have the curl issue fixed on travis, I added some retries to it and removed ssl from the check, between those two it should be a lot more reliable about succeeding. I'm open to improving them down the line but for now I just want a working build check to proceed with multiarch builds.

I'm opening a new branch multiarch as well to put work towards travis multi arch builds into. Since multiarch isn't set up on the docker hub as a build we should be able to push images into it on the hub without worrying about one build overwriting the other.

jukie commented 3 years ago

Alright great, I'll test changes in my own hub to avoid any issues as well but will have a PR ready soon.

jacobalberty commented 3 years ago

I have one big question on the docker blog post, what happens with pull requests? If I follow that exact setup it looks like any pull request will attempt to overwrite the image on the docker hub with the pull request.

I believe we should use $TRAVIS_BRANCH for the tag name as well, that way rc and beta can continue to function,

jukie commented 3 years ago

Agreed I wouldn't follow that, I think branch tags make the most sense and when a PR comes in it should only overwrite that branch tag.

Maybe you could configure the latest tag in hub to track master just retag that image somewhere? I have limited Travis experience but I imagine it supports building based on git release tags too so that should continue to function as well.

jacobalberty commented 3 years ago

$TRAVIS_BRANCH seems to be the variable to use to build based on git tags/branches but it appears to contain even the target branch I'm looking into it now to figure out the best way to handle it

jacobalberty commented 3 years ago

Ok so the docker hub guide seems to be missing a step https://medium.com/@quentin.mcgaw/cross-architecture-docker-builds-with-travis-ci-arm-s390x-etc-8f754e20aaef covers what it was missing I needed to add qemu-user-static to the apt install line (may even be able to skip this step but I didn't test yet and elsewhere i saw posts saying its needed) and use docker run --rm --privileged multiarch/qemu-user-static --reset -p yes before the buildx create line

jacobalberty commented 3 years ago

The next issue I found is gosu crashes under qemu's arm emulation. It should be safe to just remove

# verify that the binary works
    && gosu nobody true \

from the Dockerfile. That's how I handled it for the arm cross build stuff. I'm also opening a new issue regarding removing gosu entirely and will be deprecating the features that require gosu shortly followed by complete removal.

jukie commented 3 years ago

Alright I've got something that builds but I also need to overcome the build log size in travis. See https://travis-ci.org/github/Jukie/unifi-docker/builds/735860281

I'm looking for ways to remove the progress status from the output or remove non-essential logging. I'll let you know if I figure it out.

jacobalberty commented 3 years ago

use --progress plain in the build comman

jukie commented 3 years ago

Yup I’m there already, thanks! Still working through some issues but almost there.

jacobalberty commented 3 years ago

I think the big issue is going to be arm v7 support, mongo db doesn't have official packages for armv7 and nothing newer than ubuntu 16.04 includes mongodb on armv7

jacobalberty commented 3 years ago

So here's what I'm thinking. I need to modify the build script and remove all mongo db stuff.

Then have the build script execute anything under say pre-build/ when its done installing pre-requisites but hasn't installed the unifi package, On platforms like arm64 and amd64 mongodb will install on its own through the distrobutions apt repo, then on armhf (armv7) We can either install a dummy mongodb package and just let users install their own mongodb service in another container ala the included docker-compose.yml, or add a complete build script there for mongodb.

jukie commented 3 years ago

That's exactly what I was attempting to solve. Although my setup already uses an external mongo and it wouldn't affect me, I think dropping support for it altogether or only for specific architectures could create problems for users so I'd personally push for adding a mongo build script that works in armhf/armv7 alongside your pre-build/ idea.

jukie commented 3 years ago

Alternatively, you could run everything on 16.04 which is still supported through next April. Targeting all platforms while on the arm32v7 branch with buildx succeeds - albeit the temporary removal of gosu nobody true.

jacobalberty commented 3 years ago

is mongodb version 3.6 on 16.04?

jacobalberty commented 3 years ago

I just checked it looks like mongodb on 16.04 is 2.6 the version regression could break existing installs.

another option is not include arm32v7 for now while testing and don't bring it out to the master until I get a working build script for mongodb on 18.04 (or maybe 20.04), my only concern there would be time limits on travis-ci building a whole mongodb package

jacobalberty commented 3 years ago

https://github.com/ddcc/mongodb/releases This has mongodb 3.2 prebuilt for armhf, which is better than nothing, the armhf build script could just pull and install that while other architectures use the distro included ones.

jukie commented 3 years ago

Yeah that could work. I assumed it was there based on my misreading of https://github.com/jacobalberty/unifi-docker/blob/arm32v7/docker-build.sh#L40 but you're correct that 2.6 is what actually is being installed on the arm32v7 image right now.

jacobalberty commented 3 years ago

Ok prebuild system has been added into beta. It can be pulled into your tree by doing a git pull then git cherry-pick 290029d7557ace802e0bf5841c984e742ad9a11b. I will start testing to get a script for pulling 3.2 into armhf

jacobalberty commented 3 years ago

If you put the following in pre_build/armhf/mongodb.sh this should allow armv7 to build:

#!/usr/bin/env bash
VER=3.2.22-2

TMP=$(mktemp -d)

curl -sL "https://github.com/ddcc/mongodb/releases/download/v${VER}/mongodb_${VER}_armhf.deb" -o "${TMP}/mongodb_${VER}_armhf.deb"
curl -sL "https://github.com/ddcc/mongodb/releases/download/v${VER}/mongodb-server_${VER}_all.deb" -o "${TMP}/mongodb-server_${VER}_all.deb"
for f in clients server-core; do
  pn="mongodb-${f}_${VER}_armhf.deb"
  curl -sL "https://github.com/ddcc/mongodb/releases/download/v${VER}/${pn}" -o "${TMP}/${pn}"
done

apt -qy install "$TMP/"*
rm -rf "$TMP"

I have not tested it in docker yet just on a host system and it appeared to work fine.

Edit just confirmed it works on a ubuntu 18.04 docker image, does not work on 20.04. Would like to find a solution that gets 3.6 working and works on ubuntu 20.04, but this is a good stopgap for now

jacobalberty commented 3 years ago

I have a test multiarch image up now, the code isn't checked into the repository yet as I still have some logistical hub issues to work out.

jukie commented 3 years ago

Okay great, I don’t think we’ll ever get mongo +3.2 running on armhf/arm32v7 due to underlying dependencies.

jacobalberty commented 3 years ago

Yeah I fear armhf support needs to be deprecated in the long run. But if I can I'd like to get at least 3.2 running on 20.04. I'll likely have to work out building 3.2 packages myself. Should be able to do it with just github actions though. this post seems to contain all of the instructions I need to do it and links to a fork of mongo 3.2 with the work done, just need to set up a build environment for it.

jacobalberty commented 3 years ago

And the multiarch tag is live on the docker hub as well as checked into github.

I'm thinking when I go to move latest from stable-5 to stable-6 that should be a good time to merge this into latest as well. if 6.0.28 makes it out of beta and then rc without issue I'm hoping to move latest with 6.0.28.

ghost commented 3 years ago

@jacobalberty Not to rush or anything, but this is the only image that for now works on my odroid C4 (S905X3 chip), all the other ones crash on mongo with a bus error. 6.0.23 isn't my version of choice, I'd love to stay on 5.14, but if I'm to go with 6.0, I'd love to have the freshest stable

jacobalberty commented 3 years ago

@elfhack on the freshest stable the consensus on community.ubnt.com seems to be that what is on https://www.ui.com/download/unifi/ is the latest stable the ones newer on the community downloads are rc, so i'm going with that for now. If you'd like to use 5.14 with the multiarch images I just built a multiarch-5 tag you can try out with 5.14.23 in it

ghost commented 3 years ago

@elfhack on the freshest stable the consensus on community.ubnt.com seems to be that what is on https://www.ui.com/download/unifi/ is the latest stable the ones newer on the community downloads are rc, so i'm going with that for now. If you'd like to use 5.14 with the multiarch images I just built a multiarch-5 tag you can try out with 5.14.23 in it

Thanks for the quick response, it works wondefully now!

jacobalberty commented 3 years ago

I have now pushed these changes into beta and rc as well, fingers crossed that 6.0.28 makes it onto the main download page soon without issue the 6.0.28-rc tag is built against the new multiarch stuff so as of this comment I'm officially dogfooding this instead of just testing builds out in isolation.

323 may still be an issue, Hoping to get someone else to chime in on it. If it is I think I know the fix (and the multiarch stuff makes the fix easy now) but I don't want to throw random fixes in without confirming it's still an issue.

jukie commented 3 years ago

Awesome, thanks for the quick implementation here!

jukie commented 3 years ago

From the PR:

This should fully implement multi-arch images via travis-ci, all that is left is to ensure armhf (arm32v7) works properly

Does that mean you're waiting for an armhf user to validate? I can attest to the both the rc and 6.0.23-ma images running successfully in my environment on armhf.

jacobalberty commented 3 years ago

@Jukie that is indeed what I mean. Good to know then

jukie commented 3 years ago

I did see #323 and at least wanted to clarify that I have a pretty simple setup and haven’t tried anything with remote access though.

jacobalberty commented 3 years ago

Ok I fired up my raspberry pi and confirmed it works with multiarch

jacobalberty / unifi-docker

Multi-arch image support #360

323 may still be an issue, Hoping to get someone else to chime in on it. If it is I think I know the fix (and the multiarch stuff makes the fix easy now) but I don't want to throw random fixes in without confirming it's still an issue.