Linux arm64/v8 support - Githubissues

coolsober commented 10 months ago

docker run -it -p 8080:8080 -v $HOME/.tabby:/data tabbyml/tabby serve --model TabbyML/StarCoder-1B Unable to find image 'tabbyml/tabby:latest' locally latest: Pulling from tabbyml/tabby docker: no matching manifest for linux/arm64/v8 in the manifest list entries. See 'docker run --help'.

Do you have plans or the capability to build for ARM64?thanks

wsxiaoys commented 10 months ago

The Dockerfile at https://github.com/TabbyML/tabby/blob/main/contrib/jetson/Dockerfile demonstrates how to add support for the Jetson (ARM) platform with CUDA. Adding support for ARM64 should be similar, although it will only run on the CPU.

edith007 commented 10 months ago

Hey @wsxiaoys I would love to work on this issue 🙏🏻 🙇🏻

wsxiaoys commented 10 months ago

Sure - note with refactoring checked-in yesterday, a arm version should be pretty easy - you might wanna fork from https://github.com/TabbyML/tabby/blob/main/Dockerfile directly

edith007 commented 10 months ago

Thanks 🙇🏻 Will make the changes and ask for review on the PR!

sonique6784 commented 9 months ago

I get this error as well on Mac M2Pro when running:

docker run -it -p 8080:8080 -v $HOME/.tabby:/data tabbyml/tabby serve --model TabbyML/StarCoder-1B  --device metal

@edith007 , let me know if you have something, I can test and share feedback

wsxiaoys commented 9 months ago

For Mac silicons, please follow instructions at https://tabby.tabbyml.com/docs/installation/apple

This issue is for linux arm64/v8 support

jasonharrison commented 9 months ago

Hi all - have not seen any activity on this issue for a month+, so I opened a PR that should fix arm64 Docker builds: https://github.com/TabbyML/tabby/pull/1022

Arthessia commented 1 month ago

I just tried to build the ARM one using the dockerfile provided but it is failing on my side

154.5    Compiling time v0.3.26
156.7 error[E0282]: type annotations needed for `Box<_>`
156.7   --> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/time-0.3.26/src/format_description/parse/mod.rs:83:9
156.7    |
156.7 83 |     let items = format_items
156.7    |         ^^^^^
156.7 ...
156.7 86 |     Ok(items.into())
156.7    |              ---- type must be known at this point
156.7    |
156.7 help: consider giving `items` an explicit type, where the placeholders `_` are specified
156.7    |
156.7 83 |     let items: Box<_> = format_items
156.7    |              ++++++++
156.7
159.7 For more information about this error, try `rustc --explain E0282`.
159.7 error: could not compile `time` (lib) due to 1 previous error
159.7 warning: build failed, waiting for other jobs to finish...
------
Dockerfile:38
--------------------
  37 |
  38 | >>> RUN --mount=type=cache,target=/usr/local/cargo/registry \
  39 | >>>     --mount=type=cache,target=/root/workspace/target \
  40 | >>>     cargo build --features cuda --release --package tabby && \
  41 | >>>     cp target/release/tabby /opt/tabby/bin/
  42 |
--------------------
ERROR: failed to solve: process "/bin/sh -c cargo build --features cuda --release --package tabby &&     cp target/release/tabby /opt/tabby/bin/" did not complete successfully: exit code: 101

wsxiaoys commented 1 month ago

Reopening the issue and the build is no longer working

TabbyML / tabby

Linux arm64/v8 support #623