No ARM Dockerfile available

rhld16 commented 1 year ago

[x] Did you check to see if this issue already exists?
[x] Is this only a single bug? Do not put multiple bugs in one issue.
[ ] Is this a question or discussion? Don't use this, use https://lemmy.ml/c/lemmy_support .
[ ] Is this a UI / front end issue? Use the lemmy-ui repo.

Issue Summary

Following the revert of the multi-arch Dockerfile, the Dockerfile.arm file was not readded meaning there's no way to build for arm in the repo. There are also no arm docker images published on Docker Hub for 1.17.4.

Steps to Reproduce

Attempt to build lemmy on an arm machine
It fails, so you go look for the old arm Dockerfile

Technical details

What OS are you trying to install lemmy on? Linux CentOS aarch64

Nutomic commented 1 year ago

We were having trouble with native ARM builds, so the plan was to cross compile. However that PR ran into problems https://github.com/LemmyNet/lemmy/pull/2806. For now you can install from scratch.

denisgolius commented 1 year ago

I got this error:

ERROR [lemmy builder 4/5] RUN --mount=type=cache,target=/app/target     if [ "debug" = "debug" ] ; then       echo "pub const VERSION: &str = "$(git describe --tag)";" > "crates/utils/sr  0.2s

I've made some changes in Dockerfile^

FROM clux/muslrust:1.70.0 as builder
WORKDIR /app
ARG CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu

lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Neoverse-N1
    Model:               1
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -
    Cluster(s):          1
    Stepping:            r3p1
    BogoMIPS:            50.00
    Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs

Do I need to try build it with lux/muslrust:1.67.0 or it's just a bug on the lemmy side?

samitaaissat commented 1 year ago

From the v0.17.4 branch, I was able to successfully use the docker file in docker/prod/Dockerfile.arm to compile it on my M1 MacBook, push it on a private repo, and use it on a server running arm.

So at least this file works

NettoHikari commented 1 year ago

Hey, there. Any news on this? Would like to update my instance to 0.18.0, but there are still no dockerfiles available.

rhld16 commented 1 year ago

@NettoHikari Retrieving the old Dockerfile from the 0.17.4 release still works. #3322 seems to be adding the functionality back as well.

paulo-roger commented 1 year ago

Someone with knowledge regarding build lemmy/lemmy-ui on arm64 could help me please?

I keep getting errors when trying to build lemmy-ui. This is what I keep getting:

 => [lemmy_lemmy-ui builder 12/18] RUN yarn --production --prefer-offline                        1.3s
 => ERROR [lemmy_lemmy-ui builder 13/18] RUN yarn build:prod                                     2.7s
------
 > [lemmy_lemmy-ui builder 13/18] RUN yarn build:prod:
#0 0.564 yarn run v1.22.19
#0 0.638 $ yarn clean && node generate_translations.js
#0 0.975 $ yarn run rimraf dist
#0 1.306 $ /usr/src/app/node_modules/.bin/rimraf dist
#0 1.558 $ webpack --mode=production
#0 2.619 [webpack-cli] Failed to load '/usr/src/app/webpack.config.js' config
#0 2.626 [webpack-cli] Error: Cannot find module 'webpack-bundle-analyzer'
#0 2.626 Require stack:
#0 2.626 - /usr/src/app/webpack.config.js
#0 2.626 - /usr/src/app/node_modules/webpack-cli/lib/webpack-cli.js
#0 2.626 - /usr/src/app/node_modules/webpack-cli/lib/bootstrap.js
#0 2.626 - /usr/src/app/node_modules/webpack-cli/bin/cli.js
#0 2.626 - /usr/src/app/node_modules/webpack/bin/webpack.js
#0 2.626     at Module._resolveFilename (node:internal/modules/cjs/loader:1070:15)
#0 2.626     at Module._load (node:internal/modules/cjs/loader:923:27)
#0 2.626     at Module.require (node:internal/modules/cjs/loader:1137:19)
#0 2.626     at require (node:internal/modules/helpers:121:18)
#0 2.626     at Object.<anonymous> (/usr/src/app/webpack.config.js:10:3)
#0 2.626     at Module._compile (node:internal/modules/cjs/loader:1255:14)
#0 2.626     at Module._extensions..js (node:internal/modules/cjs/loader:1309:10)
#0 2.626     at Module.load (node:internal/modules/cjs/loader:1113:32)
#0 2.626     at Module._load (node:internal/modules/cjs/loader:960:12)
#0 2.626     at Module.require (node:internal/modules/cjs/loader:1137:19) {
#0 2.626   code: 'MODULE_NOT_FOUND',
#0 2.626   requireStack: [
#0 2.626     '/usr/src/app/webpack.config.js',
#0 2.626     '/usr/src/app/node_modules/webpack-cli/lib/webpack-cli.js',
#0 2.626     '/usr/src/app/node_modules/webpack-cli/lib/bootstrap.js',
#0 2.626     '/usr/src/app/node_modules/webpack-cli/bin/cli.js',
#0 2.626     '/usr/src/app/node_modules/webpack/bin/webpack.js'
#0 2.626   ]
#0 2.626 }
#0 2.649 error Command failed with exit code 2.
#0 2.649 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
------
failed to solve: process "/bin/sh -c yarn build:prod" did not complete successfully: exit code: 2

I am using this docker-compose command:

docker-compose up --build -d

It is failing to build lemmy-ui. this is my lemmi-ui container section of the docker-compose.yml

  lemmy-ui:
    build: ./lemmy-ui
    # image: lemmy-easy-deploy-lemmy-ui
    # image: dessalines/lemmy-ui:0.17.3-linux-arm64
    environment:
      - LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy:8536
      - LEMMY_UI_LEMMY_EXTERNAL_HOST=localhost:1234
      - LEMMY_HTTPS=true
    volumes:
      - ./volumes/lemmy-ui/extra_themes:/app/extra_themes
    depends_on:
      - lemmy
    networks:
      - lemmyinternal
      - lemmybridge
    ports:
      - 1234:1234
    restart: always
    logging: *default-logging

I am stuck with this since saturday, any idea is appreciated.

CorvetteCole commented 1 year ago

yeah I'm still stuck on 0.17.4

snowe2010 commented 1 year ago

@paulo-roger you can use programming.dev's dockerfile here. https://github.com/programming-dot-dev/lemmy-ui/blob/programming.dev/main/Dockerfile#L30

You need to add the webpack analyzer without adding it to the production dependencies.

Also, we have a working github action that deploys a Dockerfile for both arm and amd

https://github.com/programming-dot-dev/lemmy-ui/blob/programming.dev/main/.github/workflows/publish-container.yaml

https://github.com/programming-dot-dev/lemmy-ui/pkgs/container/lemmy-ui

Edit: we have customizations to lemmy in that branch though, so be wary of just deploying it outright. We added a 'Request Community' button to the top.

CorvetteCole commented 1 year ago

@snowe2010 your link to the container (https://github.com/programming-dot-dev/lemmy-ui/pkgs/container/lemmy-ui) is dead, fyi

snowe2010 commented 1 year ago

@CorvetteCole my bad. Package was set to private by default. Updated. you should be able to see it now.

CorvetteCole commented 1 year ago

@snowe2010 are you planning on publishing a package for normal 0.18.0 lemmy? I've managed to get everything building and my ARM instance is running 0.18.0 now. Might publish my own container for those that want to upgrade for now

snowe2010 commented 1 year ago

You go for it. I think it would probably be best to just make a pr to merge the fix though. I'm not sure what system the devs are using to deploy, but it was dead simple with GitHub actions.

paulo-roger commented 1 year ago

@paulo-roger you can use programming.dev's dockerfile here. https://github.com/programming-dot-dev/lemmy-ui/blob/programming.dev/main/Dockerfile#L30

You need to add the webpack analyzer without adding it to the production dependencies.

Also, we have a working github action that deploys a Dockerfile for both arm and amd

https://github.com/programming-dot-dev/lemmy-ui/blob/programming.dev/main/.github/workflows/publish-container.yaml

https://github.com/programming-dot-dev/lemmy-ui/pkgs/container/lemmy-ui

Edit: we have customizations to lemmy in that branch though, so be wary of just deploying it outright. We added a 'Request Community' button to the top.

You're amazing @snowe2010 Thank you so much!

ubergeek77 commented 1 year ago

If anyone is looking for a "vanilla" set of multiarch images, I also build my own here:

https://github.com/ubergeek77/lemmy-docker-multiarch

ghcr.io/ubergeek77/lemmy:<some-tag>
ghcr.io/ubergeek77/lemmy-ui:<some-tag>

I provide x64 images, ARM32v7 images, and ARM64 images.

I have not modified anything apart from making the Dockerfiles more multiarch-friendly. I have images for 0.18.0, the latest rc's, and when 0.18.1 is fully released, a workflow will be kicked off automatically to build it.

Normally I'd submit PRs for these Dockerfiles, but the Lemmy team seems determined to configure their Dockerfiles to cross compile. I disagree with that decision, since it prevents ARM hosts from ever being able to build, and since it is cheap and easy to build on real ARM cloud VMs as my workflow does. Each run finishes in 40 minutes and costs me maybe $0.09 total for both images on all architectures, and only 9 minutes/$0.02 when only building the UI.

Building on ARM is not that much slower than on x64, so I really don't think cross compilation is worth all the trouble it's been causing. I believe this effort would be better spent focusing on architecture-agnostic Dockerfiles, and improving the current CI/CD pipeline for better multiarch support. The Lemmy team is more than welcome to use mine as a reference. To save costs, multiarch Docker images would only need to be built for release or rc tags, leaving the current CI/CD architecture as-is for development builds.

Regardless of my opinion, I think it would be pretty rude of me to submit a PR that clearly conflicts with the direction of the Lemmy team. But at least anyone can use these if they want!

Nutomic commented 1 year ago

@ubergeek77 We previously had an ARM builder from a community member which was extremely slow and unreliable, I think it was something similar to Raspberry Pi. A cloud server might be a better option. However I dont think that woodpecker supports launching builders on demand, so it would have to run 24/7. If you would be willing to donate such a builder and manage it, we could definitely use it for releases instead of cross compiling.

ubergeek77 commented 1 year ago

Is there a reason using woodpecker is a hard requirement for building images? It would be trivial to set up a simple GitHub Actions workflow that launches on AWS like I do just for building release images. Your woodpecker CI/CD could stay as-is for per-commit builds.

snowe2010 commented 1 year ago

Yeah I'm wondering why the usage of woodpecker as well. I'd never even heard of it until this project.

Pinkolik commented 1 year ago

Hello everyone! Any progress on this?

TheSilkky commented 1 year ago

I managed to get cross compiling on x64 working in Docker for x64 and ARM64 musl targets. I'm using the BlackDex/rust-musl image used by the Vaultwarden project, which includes a pre-built static PostgreSQL library and OpenSSL. I've tested it in GitHub actions and it builds working images that I'm currently using on my Lemmy instance, running on an ARM64 server.

Edit: Performance wise compiling both x64 and ARM64 targets at the same time, takes about 33 mins running on a GItHub actions runner.

My Dockerfiles and pre-built images are available at my repository TheSilkky/lemmy-docker. Below is a copy of my working Lemmy Dockerfile, hopefully this is useful (buildx is required):

#syntax=docker/dockerfile:1-labs

ARG RUST_VERSION=1.71.0
ARG ALPINE_VERSION=3.18
ARG LEMMY_VERSION

####################################################################################################
## Get source code
####################################################################################################
FROM scratch AS source

ARG LEMMY_VERSION

ADD --keep-git-dir=true https://github.com/LemmyNet/lemmy.git#${LEMMY_VERSION} /lemmy

####################################################################################################
## AMD64 builder base
####################################################################################################
FROM --platform=${BUILDPLATFORM} blackdex/rust-musl:x86_64-musl-stable-${RUST_VERSION} AS base-amd64

ENV DEBIAN_FRONTEND=noninteractive
ENV CARGO_HOME="/root/.cargo"

RUN apt update && apt install -y \
    --no-install-recommends \
    git

RUN mkdir -pv "${CARGO_HOME}" && \
    rustup set profile minimal && \
    rustup target add x86_64-unknown-linux-musl

####################################################################################################
## ARM64 builder base
####################################################################################################
FROM --platform=${BUILDPLATFORM} blackdex/rust-musl:aarch64-musl-stable-${RUST_VERSION} AS base-arm64

ENV DEBIAN_FRONTEND=noninteractive
ENV CARGO_HOME="/root/.cargo"

RUN apt update && apt install -y \
    --no-install-recommends \
    git

RUN mkdir -pv "${CARGO_HOME}" && \
    rustup set profile minimal && \
    rustup target add aarch64-unknown-linux-musl

####################################################################################################
## AMD64 builder
####################################################################################################
FROM base-amd64 AS build-amd64

WORKDIR /lemmy

COPY --from=source /lemmy ./
RUN echo "pub const VERSION: &str = \"$(git describe --tag)\";" > crates/utils/src/version.rs && \
    cargo build --target=x86_64-unknown-linux-musl --release && \
    mv target/x86_64-unknown-linux-musl/release/lemmy_server ./lemmy

####################################################################################################
## ARM64 builder
####################################################################################################
FROM base-arm64 AS build-arm64

WORKDIR /lemmy

COPY --from=source /lemmy ./
RUN echo "pub const VERSION: &str = \"$(git describe --tag)\";" > crates/utils/src/version.rs && \
    cargo build --target=aarch64-unknown-linux-musl --release && \
    mv target/aarch64-unknown-linux-musl/release/lemmy_server ./lemmy

####################################################################################################
## Get target binary
####################################################################################################
FROM build-${TARGETARCH} AS build

####################################################################################################
### Final image
####################################################################################################
FROM alpine:${ALPINE_VERSION} 

ARG LEMMY_VERSION
ARG UID=911
ARG GID=911
ARG CONFIG_DIRECTORY=/etc/lemmy

ENV LEMMY_CONFIG_LOCATION="${CONFIG_DIRECTORY}/lemmy.hjson"

RUN apk add --no-cache \
    ca-certificates

COPY --from=build --chmod=0755 /lemmy/lemmy /usr/local/bin

RUN addgroup -S -g ${GID} lemmy && \
    adduser -S -H -D -G lemmy -u ${UID} -g "" -s /sbin/nologin lemmy && \
    mkdir -p "${CONFIG_DIRECTORY}"

USER lemmy

CMD ["lemmy"]

EXPOSE 8536

STOPSIGNAL SIGTERM

LABEL org.opencontainers.image.source="https://github.com/LemmyNet/lemmy.git"
LABEL org.opencontainers.image.version=${LEMMY_VERSION}
LABEL org.opencontainers.image.licenses="AGPL-3.0-or-later"
LABEL org.opencontainers.image.title="Lemmy"
LABEL org.opencontainers.image.description="A link aggregator and forum for the fediverse"

Nutomic commented 1 year ago

@TheSilkky That looks good. Could you open a pull request with that dockerfile? Then we can use it for ARM release builds.

TheSilkky commented 1 year ago

@Nutomic Yep I'll open a pull request

kroese commented 1 year ago

@TheSilkky The docker files in https://github.com/ubergeek77/lemmy-docker-multiarch look a lot less complicated? Is there any advantage to the ones you created?

ubergeek77 commented 1 year ago

@kroese They don't have the CI/CD infrastructure for building on ARM, and have decided to stick with Woodpecker in lieu of a CI/CD that can launch ARM runners.

The PR just discussed would build images for ARM, but there is still a gap that people building from ARM will have no official options.

TheSilkky commented 1 year ago

@Nutomic I opened #3810

RocketDerp commented 1 year ago

There are new installs today going up with 0.18.0 with major bugs because they can't find arm docker builds: https://lemmy.ml/post/3328629

EvilOlaf commented 1 year ago

@ubergeek77 We previously had an ARM builder from a community member which was extremely slow and unreliable, I think it was something similar to Raspberry Pi. A cloud server might be a better option. However I dont think that woodpecker supports launching builders on demand, so it would have to run 24/7. If you would be willing to donate such a builder and manage it, we could definitely use it for releases instead of cross compiling.

Hetzner cloud offers ARM64 cloud servers which are billed hourly. So they can be deployed and discarded on demand. Using the hcloud and cloud-init this can be done via Github Actions as well. Armbian is using this to deploy a fully configured Jitsi meet instance on such a cloud server on demand. Instead of paying like 30€ per month it is about 5ct/hour. Depending on how many meetings occur this adds up to like 50ct/month.

Anyway I assume there aren't that much resources needed to build the whole project and its Docker containers the smallest instance would probably do which is, depending on the project's funds, still quite affordable if online 24/7 with less that 5€.

EvilOlaf commented 1 year ago

Just in case somebody is interested in this topic here are sample workflows to create and destroy a CAX11 cloud vm at Hetzner on demand using Github Actions and the hcloud tool. If not, skip reading since this goes slightly into off-topic :grin:

Create: https://paste.armbian.com/pukarasopi.yaml Destroy: https://paste.armbian.com/imidiyidec.yaml

What needs to be prepared beforehand

is generating an API key which will then be added as secret CONTEXT to the repository
a ssh key pair to log into the vm (if needed, because I think everything could be passed via cloud-init too, so no login required at all)
and (also depending of way of deployment) the cloud-init file that is passed with the creation command

There are a ton of more things to say about this topic like pro/con of pre-generated static IPv4 addresses, using ssh -c instead of a script file, deploy a Github Actions Runner or simply execute stuff by hand..... but as stated with the IMHO low cost for 24/7 deployment being less than 5€/month in mind this might not be interesting at all.

Nutomic commented 1 year ago

For 0.19.0-rc.1 there is an arm64 image available. Can someone confirm that it is working?

Nutomic commented 1 year ago

Unfortunately the arm64 which was added recently had to be disabled now as we switched from Alpine to Debian.

https://github.com/LemmyNet/lemmy/pull/4004

raskyld commented 11 months ago

You can assign me the issue if you will. I managed to cross-build on a AMD64 VM (GitHub Codespace) to ARM64, I just need to clean up, document and create a GitHub actions for that.

TheFrenchGhosty commented 11 months ago

For those interested, I can confirm that https://gitlab.com/ravermeister/lemmy-arm works perfectly

kroese commented 10 months ago

This issue has been fixed now by https://github.com/LemmyNet/lemmy/pull/4142

LemmyNet / lemmy