search

valeriansaliou / sonic

🦔 Fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.
https://crates.io/crates/sonic-server
Mozilla Public License 2.0
20.13k stars 578 forks source link

[Docker] Build image for arm64v8 #202

Open poupryc opened 4 years ago

poupryc commented 4 years ago

I think it would be cool if we could use Sonic on raspberry pi (I'm thinking of raspberry pi 3 and up that runs in 64 bits)

Thanks,

valeriansaliou commented 4 years ago

I've not managed to build a MUSL binary for any ARM platform, due to dependencies incompatibility. If someone manages to build it for an ARM target, I'll accept a PR.

darkstar commented 2 years ago

This issue is 2 years old. I'm sure by now musl is buildable on ARM64v8? I#d love to get this running as docker image on my RaspberryPi for ArchiveBox

valeriansaliou commented 2 years ago

I've tried 3 months ago w/ the latest RocksDB and could not successfully have a MUSL build working. I managed to compile it through some tricks, but Sonic segfaulted at runtime, and unfortunately I was out of time and needed to move to other things so I didn't keep on looking for a fix since then. I'll happily take PRs on that one!

jaw3l commented 2 years ago

It can be built now by replacing "__x86_64" to "aarch64__" in the Dockerfile.

FROM rust:slim-bullseye AS build

RUN apt-get update
RUN apt-get install -y build-essential clang

RUN rustup --version
RUN rustup component add rustfmt

RUN rustc --version && \
    rustup --version && \
    cargo --version

WORKDIR /app
COPY . /app
RUN cargo clean && cargo build --release --target aarch64-unknown-linux-gnu
RUN strip ./target/aarch64-unknown-linux-gnu/release/sonic

FROM debian:bullseye-slim

WORKDIR /usr/src/sonic

COPY --from=build /app/target/aarch64-unknown-linux-gnu/release/sonic /usr/local/bin/sonic

CMD [ "sonic", "-c", "/etc/sonic.cfg" ]

EXPOSE 1491
valeriansaliou commented 2 years ago

This is for Apple Silicon CPUs if I understand well the purpose of aarch64, would it also work on ARMv8 CPUs?

jaw3l commented 2 years ago

AArch64 is the 64-bit state introduced in the ARMv8-A architecture. ARM ARMv8 CPU architecture has two ISAs (instruction sets). AArch64 and AArch32 are the two instruction sets that ARMv8-A can support. According to Wikipedia we can say they are the same.

AArch64 or ARM64 is the 64-bit extension of the ARM architecture.

I built the Dockerfile above with my Raspberry Pi 4 64-bit . So yeah it works on ARMv8 as far as i can see.

valeriansaliou commented 2 years ago

Excellent! Will review this ASAP, this is just what I needed :)

JoshMock commented 1 year ago

Would love to see this get merged soon! Also wanting to index my Archivebox data on an RPi. 🖤

as-shura commented 1 year ago

It can be built now by replacing "x86_64" to "aarch64" in the Dockerfile.

FROM rust:slim-bullseye AS build

RUN apt-get update
RUN apt-get install -y build-essential clang

RUN rustup --version
RUN rustup component add rustfmt

RUN rustc --version && \
    rustup --version && \
    cargo --version

WORKDIR /app
COPY . /app
RUN cargo clean && cargo build --release --target aarch64-unknown-linux-gnu
RUN strip ./target/aarch64-unknown-linux-gnu/release/sonic

FROM debian:bullseye-slim

WORKDIR /usr/src/sonic

COPY --from=build /app/target/aarch64-unknown-linux-gnu/release/sonic /usr/local/bin/sonic

CMD [ "sonic", "-c", "/etc/sonic.cfg" ]

EXPOSE 1491

I just tries your code in a SOC arm64 bits and get this result:


#0 6.055     Updating crates.io index
#0 523.3  Downloading crates ...
#0 524.1   Downloaded fst-regex v0.3.0
#0 524.1   Downloaded fxhash v0.2.1
#0 524.1   Downloaded phf_shared v0.11.1
#0 524.2   Downloaded lazycell v1.3.0
#0 524.5   Downloaded rand_chacha v0.3.1
#0 524.8   Downloaded strsim v0.10.0
#0 524.8   Downloaded siphasher v0.3.10
#0 524.8   Downloaded linked_hash_set v0.1.4
#0 524.8   Downloaded lazy_static v1.4.0
#0 524.9   Downloaded version_check v0.9.4
#0 525.1   Downloaded shlex v1.1.0
#0 525.2   Downloaded rustc-hash v1.1.0
#0 525.5   Downloaded phf_generator v0.11.1
#0 525.7   Downloaded clap_lex v0.2.4
#0 526.0   Downloaded peeking_take_while v0.1.2
#0 526.0   Downloaded fst-levenshtein v0.3.0
#0 526.1   Downloaded phf_codegen v0.11.1
#0 526.2   Downloaded glob v0.3.0
#0 526.3   Downloaded utf8-ranges v1.0.5
#0 526.4   Downloaded cfg-if v1.0.0
#0 526.4   Downloaded atty v0.2.14
#0 526.4   Downloaded cfg-if v0.1.10
#0 526.5   Downloaded radix v0.6.0
#0 526.6   Downloaded jobserver v0.1.25
#0 526.6   Downloaded rand_core v0.6.4
#0 526.7   Downloaded once_cell v1.15.0
#0 526.7   Downloaded ppv-lite86 v0.2.16
#0 526.8   Downloaded twox-hash v1.6.3
#0 526.9   Downloaded tikv-jemallocator v0.4.3
#0 526.9   Downloaded cedarwood v0.4.6
#0 527.0   Downloaded smallvec v1.10.0
#0 527.0   Downloaded termcolor v1.1.3
#0 527.1   Downloaded autocfg v1.1.0
#0 527.1   Downloaded byteorder v1.4.3
#0 527.1   Downloaded getrandom v0.2.7
#0 527.2   Downloaded linked-hash-map v0.5.6
#0 527.5   Downloaded phf v0.11.1
#0 527.8   Downloaded cexpr v0.6.0
#0 528.0   Downloaded fs_extra v1.2.0
#0 528.0   Downloaded bitflags v1.3.2
#0 528.1   Downloaded memmap v0.6.2
#0 528.2   Downloaded log v0.4.17
#0 528.4   Downloaded libloading v0.7.3
#0 528.5   Downloaded ahash v0.7.6
#0 528.5   Downloaded unicode-ident v1.0.5
#0 528.5   Downloaded indexmap v1.9.1
#0 528.6   Downloaded clang-sys v1.4.0
#0 528.6   Downloaded static_assertions v1.1.0
#0 528.8   Downloaded os_str_bytes v6.3.0
#0 529.2   Downloaded quote v1.0.21
#0 529.7   Downloaded proc-macro2 v1.0.47
#0 529.8   Downloaded serde_derive v1.0.145
#0 530.3   Downloaded textwrap v0.15.1
#0 530.6   Downloaded hashbrown v0.12.3
#0 530.7   Downloaded memchr v2.5.0
#0 530.8   Downloaded toml v0.5.9
#0 530.9   Downloaded cc v1.0.73
#0 531.1   Downloaded serde v1.0.145
#0 531.6   Downloaded rand v0.8.5
#0 531.7   Downloaded rocksdb v0.17.0
#0 532.1   Downloaded minimal-lexical v0.2.1
#0 532.2   Downloaded whatlang v0.16.1
#0 532.2   Downloaded unicode-segmentation v1.10.0
#0 532.5   Downloaded nom v7.1.1
#0 532.5   Downloaded aho-corasick v0.7.19
#0 533.0   Downloaded regex-syntax v0.3.9
#0 534.1   Downloaded bindgen v0.59.2
#0 535.3   Downloaded nix v0.18.0
#0 535.5   Downloaded clap v3.2.22
#0 537.4   Downloaded syn v1.0.102
#0 538.0   Downloaded regex-syntax v0.6.27
#0 538.1   Downloaded regex v1.6.0
#0 552.4   Downloaded libc v0.2.135
#0 553.6   Downloaded tikv-jemalloc-sys v0.4.3+5.2.1-patched.2
#0 587.6   Downloaded fst v0.3.5
#0 612.0   Downloaded jieba-rs v0.6.7
#0 617.0   Downloaded librocksdb-sys v6.20.3
#0 622.1    Compiling libc v0.2.135
#0 622.1    Compiling cfg-if v1.0.0
#0 622.1    Compiling memchr v2.5.0
#0 628.7    Compiling proc-macro2 v1.0.47
#0 629.1    Compiling unicode-ident v1.0.5
#0 631.9    Compiling glob v0.3.0
#0 633.6 error: could not compile `proc-macro2`
#0 633.6
#0 633.6 Caused by:
#0 633.6   process didn't exit successfully: `rustc --crate-name build_script_build --edition=2018 /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.47/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=d726feb61c0c13f7 -C extra-filename=-d726feb61c0c13f7 --out-dir /app/target/release/build/proc-macro2-d726feb61c0c13f7 -C strip=symbols -L dependency=/app/target/release/deps --cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)
#0 633.6 warning: build failed, waiting for other jobs to finish...
------
Dockerfile:15
--------------------
  13 |     WORKDIR /app
  14 |     COPY . /app
  15 | >>> RUN cargo clean && cargo build --release --target aarch64-unknown-linux-gnu
  16 |     RUN strip ./target/aarch64-unknown-linux-gnu/release/sonic
  17 |
--------------------
ERROR: failed to solve: process "/bin/sh -c cargo clean && cargo build --release --target aarch64-unknown-linux-gnu" did not complete successfully: exit code: 101
repomaa commented 1 year ago

any news on this? I was also able to build the image. Would be nice to have an official one :blush:

valeriansaliou commented 1 year ago

I'll batch-process all my OSS projects MR soon, pbb in a few weeks. Sorry I didn't have any chance to review this before.

bendangelo commented 1 year ago

It can be built now by replacing "x86_64" to "aarch64" in the Dockerfile.

FROM rust:slim-bullseye AS build

RUN apt-get update
RUN apt-get install -y build-essential clang

RUN rustup --version
RUN rustup component add rustfmt

RUN rustc --version && \
    rustup --version && \
    cargo --version

WORKDIR /app
COPY . /app
RUN cargo clean && cargo build --release --target aarch64-unknown-linux-gnu
RUN strip ./target/aarch64-unknown-linux-gnu/release/sonic

FROM debian:bullseye-slim

WORKDIR /usr/src/sonic

COPY --from=build /app/target/aarch64-unknown-linux-gnu/release/sonic /usr/local/bin/sonic

CMD [ "sonic", "-c", "/etc/sonic.cfg" ]

EXPOSE 1491

This works for me when running on github actions.

Also is there an update on this? Adding arm is working for me.

pirate commented 1 week ago

ArchiveBox now provides a multi-arch build of sonic, you're welcome to use it if helpful: https://hub.docker.com/r/archivebox/sonic

JoshMock commented 1 week ago

@pirate happy to see ArchiveBox dev has picked back up! Felt like I was on 0.6.2 forever, and then had to spin down my archive for a bit. :smile: :heart: