search

browsermt / bergamot-translator

Cross platform C++ library focusing on optimized machine translation on the consumer-grade device.
http://browser.mt
Mozilla Public License 2.0
340 stars 38 forks source link

ARM Support for bergamot-translator matrix multiplies #324

Closed jerinphilip closed 1 year ago

jerinphilip commented 2 years ago

We're going to do this the way it saves time in the long run.

Edit: Looks like I'm going to start doing with browsermt fork first. We'll figure marian-nmt/marian-dev later.

The tasks involve the following:

jerinphilip commented 2 years ago

So previously, we were looking at -DUSE_WASM_COMPATIBLE_SOURCE=on when targeting Mozilla ARM, but we don't want this on native (as it disables threading etc). And enabling it appears to have uncovered some x86 only things in faiss.

https://github.com/browsermt/marian-dev/blob/08b1544636fe13eaf1fbacb17c6fb050abfb8d42/src/3rd_party/faiss/VectorTransform.cpp#L135

More recent faiss has ARM support (but is failing due to absence of BLAS in my android-ndk cross compile).

kpu commented 2 years ago

We don't use faiss currently. It could be in the future if we switch to maximum dot product instead of shortlists.

jerinphilip commented 2 years ago

We've hit Hello world on ARM. 

$ cat $CONFIG
relative-paths: true
models:
  - model.intgemm.alphas.bin
vocabs:
  - vocab.deen.spm
  - vocab.deen.spm
shortlist:
    - lex.s2t.bin
    - false
beam-size: 1
normalize: 1.0
word-penalty: 0
mini-batch: 64
maxi-batch: 1000
maxi-batch-sort: src
workspace: 2000
max-length-factor: 2.5
gemm-precision: int8Alpha
$ ./marian-decoder -c $CONFIG --log-level off <<< "Hello world!"
Hallo Welt!
XapaJIaMnu commented 2 years ago

git diff?

jerinphilip commented 2 years ago

@XapaJIaMnu

https://github.com/jerinphilip/marian/pull/4 (It's in this issue's description). This attempt experimental, in that I started an #ifdef USE_INTGEMM next to the WASM, pulled in https://github.com/jerinphilip/MozIntGemm/ and now got it to work.

There's Android NDK on CI successfully building libmarian (There's a secondary issue of protobuf-> sentencepiece-> android logging causing issues, which I have deferred a solution for). On the oracle cloud ARM machine we have access to all targets compile.

I am not optimistic about the source in the existing state (ifdefs that have to be tracked) being maintainable and am open to suggestions regarding how best to clean it up. I also see conversations of merging with upstream (marian-nmt/marian-dev). I will need to start tracking the other branches (int8 similar to int8Alpha) and fix those up as well.