Srtool does not work with Apple M1 CPUs

[seunlanlege]

➜ srtool build --package picasso-runtime --profile release
Found 1.56.1, we will be using paritytech/srtool:1.56.1 for the build
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
🧰 Substrate Runtime Toolbox - srtool v0.9.18 🧰
              - by Chevdor -
info: using existing install for '1.56.1-x86_64-unknown-linux-gnu'
info: override toolchain for '/build' set to '1.56.1-x86_64-unknown-linux-gnu'

  1.56.1-x86_64-unknown-linux-gnu unchanged - rustc 1.56.1 (59eed8a2a 2021-11-01)

🏗  Building picasso-runtime as release using rustc 1.56.1 (59eed8a2a 2021-11-01)
⏳ That can take a little while, be patient... subsequent builds will be faster.
   Since you have to wait a little, you may want to learn more about Substrate runtimes:
   https://docs.substrate.io/v3/getting-started/architecture/
    Updating crates.io index
/srtool/build: line 118:    63 Killed                  $CMD 2>&1

https://github.com/paritytech/srtool/blob/e8892f0e220a53a0c2fb516e12bace90f8613553/scripts/build?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L109-L116

Looks like this is where the issue is from, how do i get the error message?

[chevdor]

Could you tell me in which repo/branch you were working ? I see picasso-runtime, were you in master ?

Can you try using --runtime-dir as well ? So:

srtool build --package picasso-runtime --runtime-dir some/dir

[chevdor]

I tested the following and it started building the runtime:

cd /tmp
git clone https://github.com/ComposableFi/composable-public.git
cd composable-public
srtool build --package picasso-runtime --runtime-dir runtime/picasso 

Although, I tried srtool build --package picasso-runtime and it worked as well so I may not have spotted your issue.

[seunlanlege]

Maybe worth mentioning I'm on the new m1 macs.

EDIT

Yeah its definitely apple silicon, i was working on another copy of the repo, but it still doesn't build for me

~/Projects
➜ gcl composableFi/composable-public
Cloning into 'composable-public'...
remote: Enumerating objects: 83, done.
remote: Counting objects: 100% (83/83), done.
remote: Compressing objects: 100% (61/61), done.
remote: Total 83 (delta 17), reused 77 (delta 17), pack-reused 0
Receiving objects: 100% (83/83), 1.18 MiB | 1.15 MiB/s, done.
Resolving deltas: 100% (17/17), done.

~/Projects took 6s
➜ ./composable-public

composable-public on  main via 𝗥 v1.56.1 on 🐳 v20.10.8
➜ srtool build --package picasso-runtime --profile release --runtime-dir ./runtime/picasso
Found 1.56.1, we will be using paritytech/srtool:1.56.1 for the build
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
🧰 Substrate Runtime Toolbox - srtool v0.9.18 🧰
              - by Chevdor -
info: using existing install for '1.56.1-x86_64-unknown-linux-gnu'
info: override toolchain for '/build' set to '1.56.1-x86_64-unknown-linux-gnu'

  1.56.1-x86_64-unknown-linux-gnu unchanged - rustc 1.56.1 (59eed8a2a 2021-11-01)

🏗  Building picasso-runtime as release using rustc 1.56.1 (59eed8a2a 2021-11-01)
⏳ That can take a little while, be patient... subsequent builds will be faster.
   Since you have to wait a little, you may want to learn more about Substrate runtimes:
   https://docs.substrate.io/v3/getting-started/architecture/
    Updating crates.io index
/srtool/build: line 118:    63 Killed                  $CMD 2>&1

composable-public on  main via 𝗥 v1.56.1 on 🐳 v20.10.8 took 1m 52s

[chevdor]

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Indeed. I am not sure how/if I can help you. While I do use a Mac, it is an Intel one. However from the message I see, it sounds like you may get away specifying the platform: docker buildx —-platform linux/amd64,linux/arm64...

[seunlanlege]

I'm using the srtool-cli how do I specify the platform using that?

[chevdor]

This won't be supported yet in srtool-cli. But you could try using the image directly and if you find an option that works, I'd be happy to add that to the srtool-clito make it transparent to the user.

The very first points are:

• can it work at all with an M1
• does it lead to the same hash

[decentration]

I'm testing Picasso runtime on m1...

srtool build --package picasso-runtime --runtime-dir runtime/picasso

getting error:

/srtool/build: line 118: 67 Killed $CMD 2>&1

then ran:

docker run \ --platform linux/amd64 \ --name srtool \ --rm \ -e PACKAGE=picasso-runtime \ -e RUNTIME_DIR=runtime/picasso \ -e PROFILE=release \ -v /private/tmp/composable-public:/build \ paritytech/srtool:1.56.1 build

error:

 Downloaded aead v0.3.2
  Downloaded adler v1.0.2
  Downloaded addr2line v0.16.0
   Compiling futures v0.1.31
   Compiling typenum v1.13.0
   Compiling ppv-lite86 v0.2.10
   Compiling futures-core v0.3.16
/srtool/build: line 118:    64 Killed                  $CMD 2>&1

[aliXsed]

We have the same issue building eden runtime for nodle on an apple m1 host. It seems like srtool is recognizing that host as amd64:

[chevdor]

This currently unsupported.

[chevdor]

See also https://github.com/paritytech/ci_cd/issues/261

[chrisdcosta]

Apparently does now run on apple silicon (as I have been lately) but there is a huge performance hit - which is unfortunate because all the massive speed gains you get for compiling on M1 are completely lost.

To use the --platform docker argument I used this:

docker run \
--platform linux/amd64 \
--rm \
-it \
--name srtool-m1 \
-e RUNTIME_DIR=runtime \
-e BUILD_OPTS=" "  \
-e PACKAGE=your-runtime \
-v full-path-to-cloned-repo:/build \
-v /tmp/cargo:/cargo-home  \
paritytech/srtool:1.64.0-0.9.23

In addition you will get this warning on almost every library that gets compiled. I assume this is because srtool uses QEMU to obtain consistent hashes?

<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)

[chevdor]

Could you try building one of the runtime that we built for a release and confirm that you get the same hashes ?

[wischli]

Could you try building one of the runtime that we built for a release and confirm that you get the same hashes ?

Bump @chrisdcosta

[kianenigma]

bump. It seems like there is still no reasonable way to use srtools in apple silicon?