search

CosmWasm / optimizer

Dockerfile and script to deterministically produce the smallest possible Wasm for your Rust contract
Apache License 2.0
123 stars 61 forks source link

Create docker build for ARM based Macs #49

Closed webmaster128 closed 3 years ago

webmaster128 commented 3 years ago

Right now the docker image is only created for x86_64 which causes problems when users run it on M1 Macs.

orkunkl commented 3 years ago

Chip: m1 8 (4 performance and 4 efficiency) OS: Big Sur 11.4 Memory: 16 GB

Here is a log of cosmwasm-examples/escrow rust-optimizer:


Digest: sha256:8cb710cdc55f5609ef94cb43a417073e7c9a307bf107fc78cf49dedc57afde51
Status: Downloaded newer image for cosmwasm/rust-optimizer:0.11.4
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Info: RUSTC_WRAPPER=sccache
Info: sccache stats before build
Compile requests                      0
Compile requests executed             0
Cache hits                            0
Cache misses                          0
Cache timeouts                        0
Cache read errors                     0
Forced recaches                       0
Cache write errors                    0
Compilation failures                  0
Cache errors                          0
Non-cacheable compilations            0
Non-cacheable calls                   0
Non-compilation calls                 0
Unsupported compiler calls            0
Average cache write               0.000 s
Average cache read miss           0.000 s
Average cache read hit            0.000 s
Failed distributed compilations       0
Cache location                  Local disk: "/root/.cache/sccache"
Cache size                            0 bytes
Max cache size                       10 GiB
Building contract in /code ...
 Downloading crates ...
  Downloaded sha2 v0.9.2
  Downloaded typenum v1.12.0
  Downloaded thiserror-impl v1.0.23
  Downloaded cosmwasm-storage v0.14.0
  Downloaded thiserror v1.0.23
  Downloaded serde_json v1.0.61
  Downloaded syn v1.0.58
  Downloaded version_check v0.9.2
  Downloaded serde_derive v1.0.120
  Downloaded hex v0.4.2
  Downloaded byteorder v1.4.2
  Downloaded proc-macro2 v1.0.24
  Downloaded quote v1.0.8
  Downloaded serde v1.0.120
  Downloaded libc v0.2.82
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.58
   Compiling serde_derive v1.0.120
   Compiling serde v1.0.120
   Compiling ryu v1.0.5
   Compiling serde_json v1.0.61
   Compiling itoa v0.4.7
   Compiling schemars v0.8.3
   Compiling dyn-clone v1.0.4
   Compiling base64 v0.13.0
   Compiling quote v1.0.8
   Compiling serde_derive_internals v0.25.0
   Compiling thiserror-impl v1.0.23
   Compiling cosmwasm-derive v0.14.0
   Compiling schemars_derive v0.8.3
   Compiling thiserror v1.0.23
   Compiling serde-json-wasm v0.3.1
   Compiling cosmwasm-std v0.14.0
   Compiling cosmwasm-storage v0.14.0
   Compiling cw-escrow v0.10.0 (/code)
    Finished release [optimized] target(s) in 3m 42s
Optimizing cw_escrow.wasm ...
Creating hashes ...
Info: sccache stats after build
Compile requests                      32
Compile requests executed             16
Cache hits                             0
Cache misses                          16
Cache misses (Rust)                   16
Cache timeouts                         0
Cache read errors                      0
Forced recaches                        0
Cache write errors                     0
Compilation failures                   0
Cache errors                           0
Non-cacheable compilations             0
Non-cacheable calls                   16
Non-compilation calls                  0
Unsupported compiler calls             0
Average cache write                0.000 s
Average cache read miss           10.128 s
Average cache read hit             0.000 s
Failed distributed compilations        0

Non-cacheable reasons:
crate-type                            12
-                                      4

Cache location                  Local disk: "/root/.cache/sccache"
Cache size                            12 MiB
Max cache size                        10 GiB
done```
ethanfrey commented 3 years ago

I found this article which may be relevant: https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/

If we make a dockerfile just for m1/arm64 and the other one for intel, we could use this approach above to combine them into one Docker tag, that is selected by the user when they download/run

harryscholes commented 3 years ago

@webmaster128 one temporary solution for m1 Macs is to clone this repo and build the Docker image locally, rather than pulling it from the image registry. I found that solved the issues I was having running the container in x86 emulation mode

ethanfrey commented 3 years ago

Thanks for that tip.

We should at least document it. Supporting two architectures in one published docker file would be better I guess, but a fair bit more work.

harryscholes commented 3 years ago

Running make build on v0.11.4 builds rust-optimizer and workspace-optimizer successfully, but on v0.11.5, I get:

...
docker build -t "cosmwasm/rust-optimizer":0.11.5 --file rust-optimizer.Dockerfile .
[+] Building 1.3s (3/3) FINISHED
 => [internal] load build definition from rust-optimizer.Dockerfile                                                                                                   0.0s
 => => transferring dockerfile: 961B                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                       0.0s
 => ERROR [internal] load metadata for docker.io/cosmwasm/base-optimizer:latest                                                                                       1.1s
------
 > [internal] load metadata for docker.io/cosmwasm/base-optimizer:latest:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
make: *** [build-rust-optimizer] Error 1
ethanfrey commented 3 years ago

We added a base-optimizer image. In theory you should check out the repo and do

make build-base-optimizer && make build-rust-optimizer

But I am getting the same error.

@uint or @maurolacy (not sure who worked on this last)? Can you check this? Maybe it is just for M1 Macs?

maurolacy commented 3 years ago

I'll take a look.

maurolacy commented 3 years ago

make build-rust-optimizer should work, as it has build-base-optimizer as a dependency.

Sadly, I don't have an Apple M1 to try this and see why it fails.

ethanfrey commented 3 years ago

Can you try on a linux box and confirm it works there? I can confirm it doesn't on ARM.

We definitely need some better setup for ARM. Maybe I look into it after all these cosmwasm releases, now I have a Mac m1 to try it out on. (And also failed to run rust-optimizer yesterday)

maurolacy commented 3 years ago

Can you try on a linux box and confirm it works there?

It does work fine on a debian stable (buster) machine.

maurolacy commented 3 years ago

Maybe rust:1.54.0-alpine is not available for M1 archs yet?

maurolacy commented 3 years ago

Hmm, there's also a base-optimizer dependency that specifically says "x86_64" (binaryen-version_96-x86_64-linux.tar.gz).

ethanfrey commented 3 years ago

I can build base-optimiser fine. But then when I try to build rust-optimizer, if doesn't find the base-optimizer I just built.

Which is probably due to metadata and such - yup... your example above.

I will leave this issue until I can dig in deeper to the manifests and metadata

maurolacy commented 3 years ago

Try setting DOCKER_NAME_BASE_OPTIMIZER := "base-optimizer" in the Makefile. Perhaps the docker version you are using tries to look for base-optimizer in the docker hub, when it sees the cosmwasm/ prefix / user.

Update: You will also have to remove cosmwasm/ from rust-optimizer.Dockerfile and workspace-optimizer.Dockerfile FROM directives.

Update 2: I'm using

$ docker --version
Docker version 18.09.1, build 4c52b90

and it works just fine.

maurolacy commented 3 years ago

Take a look at https://github.com/CosmWasm/rust-optimizer/pull/52.

badgooooor commented 3 years ago

We added a base-optimizer image. In theory you should check out the repo and do

make build-base-optimizer && make build-rust-optimizer

But I am getting the same error.

@uint or @maurolacy (not sure who worked on this last)? Can you check this? Maybe it is just for M1 Macs?

I've tried this with Mac M1. There is the error prompted. Note :

[+] Building 2.7s (3/3) FINISHED
 => [internal] load build definition from rust-optimizer.Dockerfile                0.0s
 => => transferring dockerfile: 961B                                               0.0s
 => [internal] load .dockerignore                                                  0.0s
 => => transferring context: 2B                                                    0.0s
 => ERROR [internal] load metadata for docker.io/cosmwasm/base-optimizer:latest    2.6s
------
 > [internal] load metadata for docker.io/cosmwasm/base-optimizer:latest:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
make: *** [build-rust-optimizer] Error 1
richance commented 3 years ago

Facing the same issue

failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

webmaster128 commented 3 years ago

Could you try https://github.com/CosmWasm/rust-optimizer/pull/53?

git clone https://github.com/CosmWasm/rust-optimizer.git
cd rust-optimizer
git checkout use-multistage
make build

This should build cosmwasm/rust-optimizer:0.12.0 and cosmwasm/workspace-optimizer:0.12.0 locally. Check with

docker image ls | grep cosmwasm/

Now you can try to use those. I'd appreciate ARM testers before merging #53.

orkunkl commented 3 years ago

Works

docker image ls | grep cosmwasm/

cosmwasm/workspace-optimizer   0.12.0        aa29e67e7612   10 seconds ago   1.03GB
cosmwasm/rust-optimizer        0.12.0        d8d95af0a0c4   17 seconds ago   1.06GB
webmaster128 commented 3 years ago

Thanks @orkunkl! Are the images functional as contract builders?

orkunkl commented 3 years ago

Yes I built with both cw-plus and single contract.

webmaster128 commented 3 years ago

https://github.com/CosmWasm/rust-optimizer/commit/5a519d85ec6a69fe46b7864607b0092267ade209 now describes a workaround for M1 users. However, this is not the end of this ticket. We should get ARM builds to DockerHub.