Building failure after switching from 3.1.47 to 3.1.57

I am having issues building ffmpeg after switching from emscripten 3.1.47 to 3.57.

With 3.1.47 I am able to build ffmpeg.wasm with aom and libpvx libraries, however with emscripten 3.1.57 I am fighting some errors.

When building ffmpeg with -O0, the builds finishes ok, but there are runtime issues (in createModule() phase) when running .wasm in browser:

RuntimeError: Aborted(CompileError: WebAssembly.instantiate(): Compiling function #13387:"decoder_get_frame" failed: not enough arguments on the stack for call (need 4, got 3) @+13970659)

Following that, it does appears both aom and libvpx defines the same static decoder_get_frame symbol:

But I do not think I will be able to convince aom / libvpx teams to rename all the conflicting symbols. Actually, I do not think these symbols are eventually needed or used in final .wasm.

When building with -O3 the build fails with

[parse exception: attempted pop from empty stack / beyond block start boundary at 7509472 (at 0:7509472)]
Fatal: error parsing wasm (try --debug for more info)
emcc: error: '/ffmpeg-wasm/modules/emsdk/upstream/bin/wasm-opt --strip-target-features --post-emscripten -O3 --low-memory-unused --zero-filled-memory --pass-arg=directize-initial-contents-immutable /ffmpeg-wasm/wasm/ffmpeg-lgpl-simd.wasm -o /ffmpeg-wasm/wasm/ffmpeg-lgpl-simd.wasm --mvp-features --enable-threads --enable-bulk-memory --enable-multivalue --enable-mutable-globals --enable-sign-ext --enable-simd' failed (returned 1)

running wasm-opt --debug prints a huge log, ending with these lines:

<==
getInt8: 0 (at 7509465)
getS32LEB: 0 ==>
zz recurse from 2 at 7509466
zz recurse into 2 at 7509466
getInt8: 16 (at 7509466)
readExpression seeing 16
zz node: Call
<==
getInt8: 128 (at 7509467)
getInt8: 128 (at 7509468)
getInt8: 128 (at 7509469)
getInt8: 128 (at 7509470)
getInt8: 0 (at 7509471)
getU32LEB: 0 ==>
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
[parse exception: attempted pop from empty stack / beyond block start boundary at 7509472 (at 0:7509472)]
wasm-opt: /usr/local/google/home/dschuff/s/emr/emscripten-releases/binaryen/src/support/name.cpp:44: std::ostream &wasm::Name::print(std::ostream &) const: Assertion `*this && "Cannot print an empty name"' failed.
Aborted

Linking only single aom or vpx at a time, there is no error even with 3.1.57, so my thinking is the problem is some kind of conflict, and 57 being somehow less forgiving than 47.

I am not sure how to proceed from here, I want to stick to the latest emscripten version however:

3.1.47 builds working ffmpeg.wasm
3.1.48 to 3.1.56 are not available for mac M2
3.1.57 fails either in runtime or build time
3.1.58 and newer are not suitable

I wonder what is the breaking change between these versions (47 to 57) and if there is any magical build flag to use with emscripten so the build succeeds?

Both aom/libvpx are build with -DBUILD_SHARED_LIBS=0 or --disable-shared respectively, and linked to final .wasm using these flags:

emcc
-laom
-lvpx
-lworkerfs.js
-msimd128
-s USE_PTHREADS=1 -pthread
-s USE_SDL=2
-s WASM_BIGINT
-s MALLOC=mimalloc
-s INVOKE_RUN=0
-s EXIT_RUNTIME=1
-s MODULARIZE=1
-s EXPORT_NAME="createFFmpeg"
-s EXPORTED_FUNCTIONS="[_main, ___wasm_init_memory_flag]"
-s EXPORTED_RUNTIME_METHODS="[callMain, FS, WORKERFS]"
-s INITIAL_MEMORY=128mb
-s ALLOW_MEMORY_GROWTH=1
-s MAXIMUM_MEMORY=4gb
-s ENVIRONMENT=worker
-s PROXY_TO_PTHREAD=1
-s STACK_SIZE=5MB
-s DEFAULT_PTHREAD_STACK_SIZE=2MB

Can you see which version of decoder_get_frame is being included? You should be able to do this using -Wl,--trace-symbol=decoder_get_frame.

It seems that this function is being miscompiled somehow? Can you attach the resulting wasm file perhaps? Running wasm-dis and looking at the decoder_get_frame could help? Also perhaps you should attach a working wasm file (from 3.1.47)? Does the working build include a definition of decoder_get_frame?

If nothing else helps then bisection may be useful here, https://emscripten.org/docs/contributing/developers_guide.html#bisecting

Using the very same build instructions, produced 3.1.47.wasm (working) is 26MB, 3.1.57.wasm is 23.7MB (failing runtime)

I have tried to dig deeper into the problem using tracing symbols.

3.1.47

3.1.47.zip - attached contains .a, .h, .wasm files and demo.html

emcc -O0 ... -Wl,--trace-symbol=decoder_get_frame --emit-symbol-map
# no symbol related log printed
# also no log for tracing symbols like decoder_get_frame_l , decoder_get_frame.1 , $decoder_get_frame

# produced ffmpeg-lgpl-simd.js.symbols contains these lines:
# 12037:decoder_get_frame
# 13991:decoder_get_frame.1

wasm-dis ffmpeg-lgpl-simd.wasm -o ffmpeg-lgpl-simd.wasm.dis
grep -B 1 -A 1 decoder_get_frame ffmpeg-lgpl-simd.wasm.dis
# (table $0 12914 12914 funcref)
# (elem $0 (i32.const 1) ... $decoder_get_frame ... $decoder_get_frame.1 ...
#
# (func $decoder_get_frame (param $0 i32) (param $1 i32) (result i32)
#  (local $2 i32)
#
# (func $decoder_get_frame.1 (param $0 i32) (param $1 i32) (result i32)
#  (local $2 i32)

3.1.57

3.1.57.zip

emcc -O0 ... -Wl,--trace-symbol=decoder_get_frame
/ffmpeg-wasm/build/lib/libaom.a(av1_dx_iface.c.o): lazy definition of decoder_get_frame

wasm-dis ffmpeg-lgpl-simd.wasm -o ffmpeg-lgpl-simd.wasm.dis
[parse exception: attempted pop from empty stack / beyond block start boundary at 13550816 (at 0:13550816)]
Fatal: error parsing wasm (try --debug for more info)

wasm-dis ffmpeg-lgpl-simd.wasm -o ffmpeg-lgpl-simd.wasm.dis --debug
<==
getInt8: 55 (at 13550807)
getU32LEB: 55 ==>
zz recurse from 3 at 13550808
zz recurse into 3 at 13550808
getInt8: 32 (at 13550808)
readExpression seeing 32
zz node: LocalGet 13550809
<==
getInt8: 52 (at 13550809)
getU32LEB: 52 ==>
zz recurse from 3 at 13550810
zz recurse into 3 at 13550810
getInt8: 16 (at 13550810)
readExpression seeing 16
zz node: Call
<==
getInt8: 128 (at 13550811)
getInt8: 128 (at 13550812)
getInt8: 128 (at 13550813)
getInt8: 128 (at 13550814)
getInt8: 0 (at 13550815)
getU32LEB: 0 ==>
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
== popExpression
[parse exception: attempted pop from empty stack / beyond block start boundary at 13550816 (at 0:13550816)]
wasm-dis: /usr/local/google/home/dschuff/s/emr/emscripten-releases/binaryen/src/support/name.cpp:44: std::ostream &wasm::Name::print(std::ostream &) const: Assertion `*this && "Cannot print an empty name"' failed.
Aborted

It appears that both wasm-opt and wasm.dis fails to parse the created .wasm. So either the produced .wasm is broken or the wasm-** tools are. Considering chrome runtime also throws errors (see original post) I would guess the .wasm is really corrupted.

Any ideas how to proceed further?

Almost certainly the produced .wasm file here is broken.

Can you attach the broken wasm file perhaps?

Also, can you try bisecting between 3.1.47 and 3.1.57 to find the exact change that broke this (See https://emscripten.org/docs/contributing/developers_guide.html#bisecting for details on how to do this).

It is attached as 3.1.57.zip in the previous comment

@sbc100 I would love to narrow down the exact version of emsdk which introduced the issue, but that is not possible since 3.1.48 to 3.1.56 are not available for mac M2 as I mentioned in the original post:

root@8c11e13c2dd5:/ffmpeg-wasm/modules/emsdk# ./emsdk install 3.1.48
Resolving SDK version '3.1.48' to 'sdk-releases-694434b6d47c5f6eff2c8fbd9eeb016c977ae9dc-64bit'
WARNING: arm64-linux binaries are not available for all releases.
See https://github.com/emscripten-core/emsdk/issues/547
Installing SDK 'sdk-releases-694434b6d47c5f6eff2c8fbd9eeb016c977ae9dc-64bit'..
Installing tool 'node-18.20.3-64bit'..
Downloading: /ffmpeg-wasm/modules/emsdk/downloads/node-v18.20.3-linux-arm64.tar.xz from https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/node-v18.20.3-linux-arm64.tar.xz, 23242684 Bytes
Unpacking '/ffmpeg-wasm/modules/emsdk/downloads/node-v18.20.3-linux-arm64.tar.xz' to '/ffmpeg-wasm/modules/emsdk/node/18.20.3_64bit'
Done installing tool 'node-18.20.3-64bit'.
Installing tool 'releases-694434b6d47c5f6eff2c8fbd9eeb016c977ae9dc-64bit'..
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/694434b6d47c5f6eff2c8fbd9eeb016c977ae9dc/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/694434b6d47c5f6eff2c8fbd9eeb016c977ae9dc/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/694434b6d47c5f6eff2c8fbd9eeb016c977ae9dc/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
error: installation failed!
root@8c11e13c2dd5:/ffmpeg-wasm/modules/emsdk# ./emsdk install 3.1.49
Resolving SDK version '3.1.49' to 'sdk-releases-bd0a2e230466dadb36efc71aa7271f17c6c35420-64bit'
WARNING: arm64-linux binaries are not available for all releases.
See https://github.com/emscripten-core/emsdk/issues/547
Installing SDK 'sdk-releases-bd0a2e230466dadb36efc71aa7271f17c6c35420-64bit'..
Skipped installing node-18.20.3-64bit, already installed.
Installing tool 'releases-bd0a2e230466dadb36efc71aa7271f17c6c35420-64bit'..
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/bd0a2e230466dadb36efc71aa7271f17c6c35420/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/bd0a2e230466dadb36efc71aa7271f17c6c35420/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/bd0a2e230466dadb36efc71aa7271f17c6c35420/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
error: installation failed!
root@8c11e13c2dd5:/ffmpeg-wasm/modules/emsdk# ./emsdk install 3.1.50
Resolving SDK version '3.1.50' to 'sdk-releases-2ce4170cef5ce46f337f9fd907b614a8db772c7d-64bit'
WARNING: arm64-linux binaries are not available for all releases.
See https://github.com/emscripten-core/emsdk/issues/547
Installing SDK 'sdk-releases-2ce4170cef5ce46f337f9fd907b614a8db772c7d-64bit'..
Skipped installing node-18.20.3-64bit, already installed.
Installing tool 'releases-2ce4170cef5ce46f337f9fd907b614a8db772c7d-64bit'..
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/2ce4170cef5ce46f337f9fd907b614a8db772c7d/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/2ce4170cef5ce46f337f9fd907b614a8db772c7d/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/2ce4170cef5ce46f337f9fd907b614a8db772c7d/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
error: installation failed!
root@8c11e13c2dd5:/ffmpeg-wasm/modules/emsdk# ./emsdk install 3.1.51
Resolving SDK version '3.1.51' to 'sdk-releases-4f416d92fbff66ce79901cfc8263768f1b25dd3e-64bit'
WARNING: arm64-linux binaries are not available for all releases.
See https://github.com/emscripten-core/emsdk/issues/547
Installing SDK 'sdk-releases-4f416d92fbff66ce79901cfc8263768f1b25dd3e-64bit'..
Skipped installing node-18.20.3-64bit, already installed.
Installing tool 'releases-4f416d92fbff66ce79901cfc8263768f1b25dd3e-64bit'..
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/4f416d92fbff66ce79901cfc8263768f1b25dd3e/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/4f416d92fbff66ce79901cfc8263768f1b25dd3e/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/4f416d92fbff66ce79901cfc8263768f1b25dd3e/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
error: installation failed!
root@8c11e13c2dd5:/ffmpeg-wasm/modules/emsdk# ./emsdk install 3.1.52
Resolving SDK version '3.1.52' to 'sdk-releases-ce2097fb81953331e65543c20b437475f218127c-64bit'
WARNING: arm64-linux binaries are not available for all releases.
See https://github.com/emscripten-core/emsdk/issues/547
Installing SDK 'sdk-releases-ce2097fb81953331e65543c20b437475f218127c-64bit'..
Skipped installing node-18.20.3-64bit, already installed.
Installing tool 'releases-ce2097fb81953331e65543c20b437475f218127c-64bit'..
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/ce2097fb81953331e65543c20b437475f218127c/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/ce2097fb81953331e65543c20b437475f218127c/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/ce2097fb81953331e65543c20b437475f218127c/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
error: installation failed!
root@8c11e13c2dd5:/ffmpeg-wasm/modules/emsdk# ./emsdk install 3.1.53
Resolving SDK version '3.1.53' to 'sdk-releases-e5523d57a0e0dcf80f3b101bbc23613fcc3101aa-64bit'
WARNING: arm64-linux binaries are not available for all releases.
See https://github.com/emscripten-core/emsdk/issues/547
Installing SDK 'sdk-releases-e5523d57a0e0dcf80f3b101bbc23613fcc3101aa-64bit'..
Skipped installing node-18.20.3-64bit, already installed.
Installing tool 'releases-e5523d57a0e0dcf80f3b101bbc23613fcc3101aa-64bit'..
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/e5523d57a0e0dcf80f3b101bbc23613fcc3101aa/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/e5523d57a0e0dcf80f3b101bbc23613fcc3101aa/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/e5523d57a0e0dcf80f3b101bbc23613fcc3101aa/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
error: installation failed!
root@8c11e13c2dd5:/ffmpeg-wasm/modules/emsdk# ./emsdk install 3.1.54
Resolving SDK version '3.1.54' to 'sdk-releases-aa1588cd28c250a60457b5ed342557c762f416e3-64bit'
WARNING: arm64-linux binaries are not available for all releases.
See https://github.com/emscripten-core/emsdk/issues/547
Installing SDK 'sdk-releases-aa1588cd28c250a60457b5ed342557c762f416e3-64bit'..
Skipped installing node-18.20.3-64bit, already installed.
Installing tool 'releases-aa1588cd28c250a60457b5ed342557c762f416e3-64bit'..
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/aa1588cd28c250a60457b5ed342557c762f416e3/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/aa1588cd28c250a60457b5ed342557c762f416e3/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/aa1588cd28c250a60457b5ed342557c762f416e3/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
error: installation failed!
root@8c11e13c2dd5:/ffmpeg-wasm/modules/emsdk# ./emsdk install 3.1.55
Resolving SDK version '3.1.55' to 'sdk-releases-f5557e3b7166d05bddb5977e363ec48cd06e9d32-64bit'
WARNING: arm64-linux binaries are not available for all releases.
See https://github.com/emscripten-core/emsdk/issues/547
Installing SDK 'sdk-releases-f5557e3b7166d05bddb5977e363ec48cd06e9d32-64bit'..
Skipped installing node-18.20.3-64bit, already installed.
Installing tool 'releases-f5557e3b7166d05bddb5977e363ec48cd06e9d32-64bit'..
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/f5557e3b7166d05bddb5977e363ec48cd06e9d32/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/f5557e3b7166d05bddb5977e363ec48cd06e9d32/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/f5557e3b7166d05bddb5977e363ec48cd06e9d32/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
error: installation failed!
root@8c11e13c2dd5:/ffmpeg-wasm/modules/emsdk# ./emsdk install 3.1.56
Resolving SDK version '3.1.56' to 'sdk-releases-9d106be887796484c4aaffc9dc45f48a8810f336-64bit'
WARNING: arm64-linux binaries are not available for all releases.
See https://github.com/emscripten-core/emsdk/issues/547
Installing SDK 'sdk-releases-9d106be887796484c4aaffc9dc45f48a8810f336-64bit'..
Skipped installing node-18.20.3-64bit, already installed.
Installing tool 'releases-9d106be887796484c4aaffc9dc45f48a8810f336-64bit'..
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/9d106be887796484c4aaffc9dc45f48a8810f336/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/9d106be887796484c4aaffc9dc45f48a8810f336/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/9d106be887796484c4aaffc9dc45f48a8810f336/wasm-binaries-arm64.tar.xz': HTTP Error 404: Not Found
error: installation failed!

The attached 3.1.57.zip includes broken .wasm and some more files, and can be compared to working 3.1.47.zip

I have put together min reproducible build setup which I run in debian:12.5 docker container.

#!/bin/bash

# running all the commands in DOCKER:
# docker run -it -v $(pwd):/ffmpeg-wasm -w /ffmpeg-wasm debian:12.5

apt-get update
apt-get install -y git python3.11 build-essential cmake autoconf autogen automake libtool pkg-config ragel wget
git config --global pull.rebase false
ln -sf /usr/bin/python3.11 /usr/bin/python

# EMSDK
git clone --depth=1 --branch main https://github.com/emscripten-core/emsdk/
(cd emsdk && ./emsdk install 3.1.57)
(cd emsdk && ./emsdk activate 3.1.57)
source ./emsdk/emsdk_env.sh

# SETUP
export PATH=$PATH:$EMSDK/upstream/bin
ROOT_DIR=$PWD
WASM_DIR=$ROOT_DIR/wasm
BUILD_DIR=$ROOT_DIR/build
PKG_CONFIG_PATH=$BUILD_DIR/lib/pkgconfig
export EM_PKG_CONFIG_PATH=$PKG_CONFIG_PATH
TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake
export CFLAGS="-O0 -I$BUILD_DIR/include -pthread -msimd128"
export CXXFLAGS=$CFLAGS
export LDFLAGS="$CFLAGS -L$BUILD_DIR/lib"
mkdir -p $WASM_DIR

# AOM
git clone --depth=1 --branch v3.9.0 https://aomedia.googlesource.com/aom/
rm -rf aom/CMakeCache.txt
rm -rf aom/CMakeFiles
rm -rf aom/aom_build
mkdir -p aom/aom_build
(cd aom/aom_build && emmake cmake .. \
  -DCMAKE_TOOLCHAIN_FILE=$TOOLCHAIN_FILE \
  -DAOM_TARGET_CPU=generic \
  -DENABLE_DOCS=0 -DENABLE_TESTS=0 \
  -DCONFIG_RUNTIME_CPU_DETECT=0 \
  -DCONFIG_WEBM_IO=0 \
  -DCMAKE_INSTALL_PREFIX=$BUILD_DIR \
  -DBUILD_SHARED_LIBS=0 \
  -DENABLE_EXAMPLES=0 \
  -DENABLE_TOOLS=0 \
  -DCMAKE_BUILD_TYPE=Release  \
  -DAOM_EXTRA_C_FLAGS="$CFLAGS" \
  -DAOM_EXTRA_CXX_FLAGS="$CXXFLAGS" \
  -G"Unix Makefiles")
emmake make -C aom/aom_build clean
emmake make -C aom/aom_build install -j

# VPX
git clone --depth=1 --branch v1.14.0 https://github.com/webmproject/libvpx/
(cd libvpx && emconfigure ./configure \
  --prefix=$BUILD_DIR \
  --target=generic-gnu \
  --disable-install-bins \
  --disable-examples \
  --disable-tools \
  --disable-docs \
  --disable-unit-tests \
  --disable-dependency-tracking \
  --disable-shared \
  --disable-codec-srcs \
  --disable-debug-libs \
  --extra-cflags="$CFLAGS" \
  --extra-cxxflags="$CXXFLAGS")
emmake make -C libvpx clean
emmake make -C libvpx install STRIP=emstrip -j

## FFmpeg
git clone --depth=1 --branch n7.0.1 https://github.com/FFmpeg/FFmpeg
(cd FFmpeg && emconfigure ./configure \
  --target-os=none \
  --arch=x86_32 \
  --enable-cross-compile \
  --enable-version3 \
  --enable-libaom \
  --disable-encoder=libaom_av1 \
  --enable-libvpx \
  --disable-x86asm \
  --disable-inline-asm \
  --disable-stripping \
  --disable-programs \
  --disable-doc \
  --disable-debug \
  --disable-runtime-cpudetect \
  --disable-autodetect \
  --extra-cflags="$CFLAGS" \
  --extra-cxxflags="$CXXFLAGS" \
  --extra-ldflags="$LDFLAGS" \
  --pkg-config-flags="--static" \
  --nm=emnm \
  --ar=emar \
  --ranlib=emranlib \
  --cc=emcc \
  --cxx=em++ \
  --objcc=emcc \
  --dep-cc=emcc)
emmake make -C FFmpeg -j4
(cd FFmpeg && emcc \
  $LDFLAGS \
  -I. -I./fftools -I$BUILD_DIR/include \
  -Llibavcodec -Llibavdevice -Llibavfilter -Llibavformat -Llibavresample -Llibavutil -Llibswscale -Llibswresample \
  -Wno-deprecated-declarations -Wno-pointer-sign -Wno-implicit-int-float-conversion -Wno-switch -Wno-parentheses -Qunused-arguments \
  -lavdevice -lavfilter -lavformat -lavcodec -lswresample -lswscale -lavutil -lm \
  -laom \
  -lvpx \
  fftools/cmdutils.c \
  fftools/ffmpeg.c \
  fftools/ffmpeg_dec.c \
  fftools/ffmpeg_demux.c \
  fftools/ffmpeg_enc.c \
  fftools/ffmpeg_filter.c \
  fftools/ffmpeg_hw.c \
  fftools/ffmpeg_mux.c \
  fftools/ffmpeg_mux_init.c \
  fftools/ffmpeg_opt.c \
  fftools/ffmpeg_sched.c \
  fftools/objpool.c \
  fftools/opt_common.c \
  fftools/sync_queue.c \
  fftools/thread_queue.c \
  -lworkerfs.js \
  -s USE_SDL=2 \
  -s WASM_BIGINT \
  -s INVOKE_RUN=0 \
  -s EXIT_RUNTIME=1 \
  -s MODULARIZE=1 \
  -s EXPORT_NAME="createFFmpeg" \
  -s EXPORTED_FUNCTIONS="[_main, ___wasm_init_memory_flag]" \
  -s EXPORTED_RUNTIME_METHODS="[callMain, FS, WORKERFS]" \
  -s INITIAL_MEMORY=128mb \
  -s ALLOW_MEMORY_GROWTH=1 \
  -s MAXIMUM_MEMORY=4gb \
  -s ENVIRONMENT=worker \
  -s PROXY_TO_PTHREAD=1 \
  -s STACK_SIZE=5MB \
  -s DEFAULT_PTHREAD_STACK_SIZE=2MB \
  -o $WASM_DIR/ffmpeg-lgpl-simd.js)

The output .wasm and .js is then used in demo.html

<body>
<p>Watch console log.</p>
<script id="worker" type="javascript/worker">
self.onmessage = async (event) => {
    const {args, wasmUrl, ffmpegUrl, ffmpegWorkerUrl} = event.data;
    importScripts(ffmpegUrl);

    const module = await createFFmpeg({
        printErr:console.log,
        locateFile:url => {
            if(url.endsWith(".wasm")) return wasmUrl;
            if(url.endsWith(".worker.js")) return ffmpegWorkerUrl;
            return url;},
        mainScriptUrlOrBlob:ffmpegUrl});

    module.callMain(args);
};
</script>
<script>
(async () => {

if(typeof SharedArrayBuffer === "undefined")
    return document.body.innerHTML = "SharedArrayBuffer is not available. Run <code style='background:#ccc'>chrome --enable-features=SharedArrayBuffer</code> or follow <a href='https://web.dev/cross-origin-isolation-guide/'>A guide to enable cross-origin isolation</a>.";

const wasmUrl = new URL("ffmpeg-lgpl-simd.wasm", document.location).href;
const ffmpegUrl = new URL("ffmpeg-lgpl-simd.js", document.location).href;
const ffmpegWorkerUrl = new URL("ffmpeg-lgpl-simd.worker.js", document.location).href;
const args = ["-codecs"];
const workerBlob = new Blob([document.querySelector('#worker').textContent], {type:"text/javascript"});
const worker = new Worker(URL.createObjectURL(workerBlob));
worker.postMessage({args, wasmUrl, ffmpegUrl, ffmpegWorkerUrl});

})()
</script>
</body>

which eventually prints to console log

failed to asynchronously prepare wasm: CompileError: WebAssembly.instantiate(): Compiling function #12528:"decoder_get_frame" failed: not enough arguments on the stack for call (need 4, got 3) @+13546180
ffmpeg-lgpl-simd.js:520 Aborted(CompileError: WebAssembly.instantiate(): Compiling function #12528:"decoder_get_frame" failed: not enough arguments on the stack for call (need 4, got 3) @+13546180)

The attached outputs are attached in 3.1.57.zip . Please scroll few comments up for more details.

Following emscripten changelog there has been few upgrades to llvm version (between 3.1.47 and 3.1.57) but nothing else that looks suspicious. Considering the wasm-dis log with so many == popExpression could it be that some function is "nested" too deep beyond max allowed stack limit or something? I am just guessing b/c I am not really experiences in related low level languages and systems.

Can you please help me narrow down the problem in emscripten?

That wasm file does indeed look invalid, wasm-opt says

[parse exception: attempted pop from empty stack / beyond block start boundary at 13550816 (at 0:13550816)]

As this is a debug build that likely means a clang or wasm-ld issue.

Bisection may still be best. Even if 3.1.48 to 3.1.56 are not available for mac M2, it sounds like you have this running in Docker so bisection should be possible for you?

Otherwise, if not, then validating the inputs to the final link command may be a good way to narrow down which file is broken. I am actually unaware of a tool that scans a .a file and validates all the wasm object files in it, but @sbc100 do you know of one? Or is there a flag to tell wasm-ld to validate inputs?

(I would try the Docker instructions here but I don't have Docker installed myself.)

Is wasm-ld producing any warnings about signature mismatches at link time by any chance?

I was able to run/bisect builds on few more recent versions of emscripten via shell.cloud.google.com which is x86_64, which led me to a rabbit hole of more build failures and different kind of errors:

3.1.47 OK produces working build
3.1.48 (not tested)
3.1.49 OK produces working build (N/A for aarch64)
3.1.50 OK produces working build (N/A for aarch64)
3.1.51 FAILs to build emcc is unable to create an executable file issue 21070 (fixed by downgrading to 3.1.40)
3.1.52 FAILs to build (same as 3.1.51)
3.1.53 FAILs to build (same as 3.1.51)
3.1.54 FAILs to produce parseable .wasm (same as 3.1.57) build log
3.1.55 FAILs to produce parseable .wasm (same as 3.1.57)
3.1.56 (not tested)
3.1.57 FAILs to produce parseable .wasm issue 22008
3.1.58 FAILs as -pthread + -sENVIRONMENT=worker is broken issue 21989
3.1.59 FAILs (same as 3.1.58)
3.1.60 FAILs (same as 3.1.58)
3.1.61 FAILs (same as 3.1.57)
tot (2024-05-31) FAILs to build wasm-ls Assertion isa<UndefinedData>(sym) failed. build log

Not sure how this bisecting helped here as there is another different error between the originally reported 3.1.47 and 3.1.57.

Considering the most recent reliable/working version of emscripten is ~6 months old (~8 for aarch64), tot introducing new error again, and ffmpeg building is quite common failing use case ( 21155, 21070 ), I was wondering if there should be a bit more focus on building more complex projects and having ffmpeg build scheduled as emscripten unit/integration test for future releases?

@kripken the attached build script can be executed outside the docker, please go ahead and give it a go even without docker being used.

@sbc100 please find attached build log from 3.1.54 (the first emscripten version failing on making parsable .wasm) to verify any suspicious wasm-ld warnings

@jozefchutka

the attached https://github.com/emscripten-core/emscripten/issues/22008#issuecomment-2138783376 can be executed outside the docker

Thanks, I tried that now. I do get the broken final wasm file. I then ran a check on all the object files and they are all valid, so the breakage seems to happen by wasm-ld. That is, the first broken file is emcc-00-base.wasm which wasm-ld emits. @sbc100 can you take a look? If you want I can package up the files for the link command (though just running the script above should work, it was simple).

I was wondering if there should be a bit more focus on building more complex projects and having ffmpeg build scheduled as emscripten unit/integration test for future releases?

I agree, the current situation is in need of improvement. I'm surprised we are seeing so many issues here, but it could be that ffmpeg has a particularly tricky build setup and stresses things we lack coverage for atm in our main test suite. And usually bisection is magical in these situations, but if there are multiple regressions, as you ran into unfortunately, then yeah, it is not that great.

As for how we can improve things, in general there are two ways to go here:

On the emscripten side, add an ffmpeg port, and add a build test using it on our CI.
On the ffmpeg side, add a build with emsdk tot.

Either of those (after we get things working again now) should give us an early warning about possible issues.

My initial was that the linker was attempting the link functions with mismatched signatures. However, in that case the linker should normally give an error.

I think bisecting would certainly help, but I'm a little busy right now preparing the CG meeting next week so I might not have time to get to this until after next week.

Makes sense @sbc100 , no rush.

In hopes of speeding things up when you get back, I bisected now. It turns out that bisecting just on the link command worked (once I added -sERROR_ON_UNDEFINED_SYMBOLS=0 to ignore a missing symbol that I assume is due to stdlib changes between versions). I then see the following:

I started from the last emsdk tot which is 7fc912687ba2077b3aeae472b51c238b3d201c46 (3.1.61) and it hits a wasm-ld crash,

wasm-ld: /b/s/w/ir/cache/builder/emscripten-releases/llvm-project/lld/wasm/SyntheticSections.cpp:522: virtual void lld::wasm::GlobalSection::writeBody():
Assertion `isa<UndefinedData>(sym)' failed.

I bisected that against 3.1.47 which linked ok and produced a valid executable, and the bisection ended on this:

c81cdfba1a3862b05ae43eecb49fd93066732dfb is the first bad commit
commit c81cdfba1a3862b05ae43eecb49fd93066732dfb
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date:   Sat Jan 20 17:21:21 2024 +0000

    Roll llvm-project from 0784b1eefa36 to b9a1e2ab8dea (86 revisions)

    https://chromium.googlesource.com/external/github.com/llvm/llvm-project.git/+log/0784b1eefa36..b9a1e2ab8dea

    2024-01-20 21074287+nsurbay@users.noreply.github.com [AArch64] Rename LDAPR<x>pre to LDAPR<x>post (#77340)
    ...
    2024-01-19 quinn.dawkins@gmail.com [mlir][transform] Add an op for replacing values with function calls (#78398)

86 is a lot, but there are just a few WebAssembly commits there, all from you @sbc100, so hopefully it is one of these:

    2024-01-20 sbc@chromium.org [lld][WebAssembly] Fix regression in function signature checking (#78831)
    2024-01-20 sbc@chromium.org [lld][WebAssembly] Match the ELF linker in transitioning away from archive indexes. (#78658)
    2024-01-19 sbc@chromium.org [lld][WebAssembly] Use the archive offset with --whole-archive (#78791)
    2024-01-19 sbc@chromium.org [lld][WebAssembly] Reset context object after each link (#78770)
    2024-01-19 sbc@chromium.org [lld][ELF] Simplify handleLibcall. NFC (#78659)

Note that the bad releases commit c81cdfba1a3862b05ae43eecb49fd93066732dfb is somewhere in 3.1.53.

I also dug into the broken wasm output. That only occurs in LTO builds, I believe because LTO builds are built without assertions, so they do not hit that Assertion isa<UndefinedData>(sym) from earlier. That is, the broken wasm output is likely a red herring, there is just an internal error in wasm-ld, and without assertions enabled it keeps on truckin and emits a bad output (we should probably recommend people test on emsdk tot when they see odd things, to get the benefit of LLVM assertions).

Great job! So it sounds like a wasm-ld bug for sure.

If you could package up a reproducer (using -Wl,--reproduce) that I can use that would great. Maybe also open an llvm bug for this?

@sbc100 Sure, I opened https://github.com/llvm/llvm-project/issues/94077 and I'll email you the big --reproduce file.

It seems that I've faced similar issue in the project I'm working on.

Are there any hints on when this could be fixed?

emscripten-core / emscripten

Building failure after switching from 3.1.47 to 3.1.57 #22008

3.1.47

3.1.57