the-paperless-project / paperless

Scan, index, and archive all of your paper documents
GNU General Public License v3.0
7.84k stars 501 forks source link

Build and publish Multi-Arch Docker Images #674

Closed MarkSchmitt closed 3 years ago

MarkSchmitt commented 4 years ago

First things first: I love paperless, it's awesome!

In order to use it on multiarch (amd64, arm64, arm32v7) kubernetes cluster, I've created docker images for all archs that I think are useful and added a manifest file, so docker/containerd can automatically pick and choose the correct operating system/architecture.

It'll detect if it's on a tag and if so, tag the docker image like so.

You'll need to set the following travis environment varibles:

As far as I can tell, all but the DOCKER_REPO are already set, that's what used to be the "tag" in the existing CI script. I wanted to have that flexible, so I can set it to my tag in my pipeline instead.

I've extended the travis-ci pipeline to use arm64 for the arm64 build, also see: https://docs.travis-ci.com/user/multi-cpu-architectures#using-docker-in-multiple-cpu-architecture-based-builds-within-lxd-containers

For arm32v7 and arm32v6 there is afaik currently no native support, so instead, I'm using qemu userland emulation to build it, executing on the standard amd64 hardware. See also: https://github.com/multiarch/qemu-user-static#getting-started That has one drawback: Because docker is not aware that the code executed in the docker container is arm32v7/v6, it'll still pull the amd64 base image (btw, a multi arch image too). That is why there's a sed-replace, setting the base image to a specific version that I know is arm32v7/v6.

Same can be done for any arch that is supported in qemu, has a base image we can use and is supported in paperless and all it's dependencies.

Hope you like the pull request. Let me know if you need anything changed to accept it. I'd like to get my things back in upstream and stop maintaining my own branch :).

MarkSchmitt commented 4 years ago

A couple more things: the build in travis is failing because of the missing DOCKER_REPO environment variable, setting it to "the-paperless-project/paperless" should do the trick.

That issue should get resolved if we can get this Pull Request merged: #337

Docker buildx seems to be discussed as solution to build cross architecture sometimes, also in that issue #337 . I have not had any luck getting that to work, not in a pipeline, not locally on a linux box and I've never seen a build pipeline using it.

whyicantusemyemailasusername commented 3 years ago

Are there any eta for this? I'm waiting to be able to use an official image on RPI4 Thanks!

MarkSchmitt commented 3 years ago

From where does come from ${DOCKER_REPO}?

that's a travis-ci environment variable you need to set. it's not part of the pipeline, that way I can push into my repo in my travis build. we can go back to hardcoding it, though I don't see the advantage (forks will break in either case and making it harder to maintain forks)

We need images without ${TRAVIS_COMMIT}.

that's always the git revision, so imho this is always set. we could not push non-tagged, non-master (latest) builds though.

We need to have images with ${TRAVIS_TAG}.

that's supported, those lines are responsible: https://github.com/the-paperless-project/paperless/pull/674/files#diff-354f30a63fb0907d4ad57269548329e3R118-R127

We need to have the image with tag latest for backward compatibility.

that's supported, those lines are reponsible: https://github.com/the-paperless-project/paperless/pull/674/files#diff-354f30a63fb0907d4ad57269548329e3R111-R116

If you don't use the file ci/deploy-docker he should be removed but I think that he should be adapted to don't have many copy past code.

I'll remove that file in a follow up commit.

MarkSchmitt commented 3 years ago

@sbrunner

Here you can find a latest image generated with the changed travis-ci pipeline: https://hub.docker.com/layers/moztr/paperless-travis/latest/images/sha256-dec1c803503a52d4f1c6a981453e4c5c95dcc8aa2d523afe2981da6ef5bbb7d2?context=explore

And here one for a tag (I created for testing purposes) https://hub.docker.com/layers/moztr/paperless-travis/0.0.2/images/sha256-61c8449316fa63b74536f75c47080cbe4f63d33af3fb07695f53f4dfaf83a306?context=explore

Anything else I can do to help you accept the pull request?

MarkSchmitt commented 3 years ago

Moving to jonaswinkler/paperless-ng. Looks like this project is dead.