[Question] Error running prep_emcc_deps.sh - 'tvm/runtime/object.h' file not found

[Big-Boy-420]

Hi,

I'm trying to compile a llama-3.2. I have followed the setup instructions but before I can get to running the mlc_llm compile command, I am running ./web/prep_emcc_deps.sh which fails with the following output and error:

/content/mlc-llm
+ emcc --version
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.70 (b53978ee3f540dc74761eba127aa7f1b8761a125)
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

+ npm --version
8.19.4
+ TVM_SOURCE_DIR_SET=/content/tvm/3rdparty/tvm
+ git submodule update --init --recursive
++ pwd
+ CURR_DIR=/content/mlc-llm
+ [[ -z /content/tvm/3rdparty/tvm ]]
+ cd web
+ make
emcc -I/content/tvm/3rdparty/tvm -I/content/tvm/3rdparty/tvm/include -I/content/tvm/3rdparty/tvm/3rdparty/dlpack/include -I/content/tvm/3rdparty/tvm/3rdparty/dmlc-core/include -I/content/tvm/3rdparty/tvm/3rdparty/compiler-rt -I/content/tvm/3rdparty/tvm/3rdparty/picojson -I/content/mlc-llm/3rdparty/tokenizers-cpp -I/content/mlc-llm/3rdparty/tokenizers-cpp/include -I/content/mlc-llm/cpp -O3 -std=c++17 -Wno-ignored-attributes -c -MM -MT dist/wasm/mlc_wasm_runtime.bc emcc/mlc_wasm_runtime.cc >dist/wasm/mlc_wasm_runtime.d
clang: warning: argument unused during compilation: '-c' [-Wunused-command-line-argument]
In file included from emcc/mlc_wasm_runtime.cc:38:
In file included from /content/mlc-llm/cpp/grammar/grammar.cc:6:
/content/mlc-llm/cpp/grammar/grammar.h:10:10: fatal error: 
      'tvm/runtime/object.h' file not found
   10 | #include <tvm/runtime/object.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [Makefile:40: dist/wasm/mlc_wasm_runtime.bc] Error 1

I am running this on a google colab.

Below is a summary of my steps (there may be one or 2 unnecessary ones here due to my own learning/debugging but I don't think they are the cause - if they are please tell me).

Install everything

!pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cpu mlc-ai-nightly-cpu
!pip install huggingface_hub
!apt-get install cmake
!pip install --pre -U -f https://mlc.ai/wheels mlc-ai-nightly-cu122
!git lfs install
!git clone https://github.com/emscripten-core/emsdk.git --recursive
!git clone https://github.com/mlc-ai/mlc-llm.git --recursive
!git clone https://github.com/apache/tvm.git --recursive
!git -C mlc-llm pull
!git -C emsdk pull
!git -C tvm pull
!emsdk/emsdk install latest
!emsdk/emsdk activate latest
!source emsdk/emsdk_env.sh

Login to HF

from huggingface_hub import login

login()

Clone target model (this is for test purposes)

!git clone https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct dist/models/Llama-3.2-1B-Instruct

Convert the weights

!mlc_llm convert_weight dist/models/Llama-3.2-1B-Instruct/ --quantization q4f16_1 -o dist/Llama-3.2-1B-Instruct-q4f16_1-MLC --device cuda:0

Generate the config

!mlc_llm gen_config ./dist/models/Llama-3.2-1B-Instruct/ --quantization q4f16_1 --conv-template llama-3 -o dist/Llama-3.2-1B-Instruct-q4f16_1-MLC/

Compile the model (step which breaks)

# !source emsdk/emsdk_env.sh does not seem to work in colab so set the env vars via python which works

import os

os.environ['TVM_SOURCE_DIR'] = '/content/tvm/3rdparty'
os.environ['MLC_LLM_SOURCE_DIR'] = '/content/mlc-llm'
os.environ['PATH'] = (
    "/content/emsdk/node/14.18.2_64bit/bin:"
    "/content/emsdk/upstream/emscripten:"
    "/content/emsdk:" + os.environ.get('PATH', '')
)
os.environ['EMSDK'] = "/content/emsdk"
os.environ['EM_CONFIG'] = "/content/emsdk/.emscripten"
os.environ['EMSDK_NODE'] = "/content/emsdk/node/14.18.2_64bit/bin/node"
os.environ['TVM_SOURCE_DIR'] = '/content/tvm/3rdparty/tvm'
os.environ['MLC_LLM_SOURCE_DIR'] = '/content/mlc-llm'

%cd mlc-llm

# This is the line which fails
!./web/prep_emcc_deps.sh

%cd ../

!mlc_llm compile dist/Llama-3.2-1B-Instruct-q4f16_1-MLC/mlc-chat-config.json --device webgpu -o dist/Llama-3.2-1B-Instruct-q4f16_1-MLC/tl_test_mlc.wasm

Please can someone help/advise?

Many thanks,

[MasterJH5574]

cc @CharlieFRuan Could you take a quick look at your convenience?

[CharlieFRuan]

Likely the TVM is not cloned recursively. Find the TVM repo you are using locally and do git submodule --init --recursive. Or git clone --recursive url-to-repo in the first place

Just saw that it is done in the command. Will revisit later today

[CharlieFRuan]

Could you check whether mlc-llm/3rdparty/tvm is empty? And if /content/tvm is empty? Thanks!

[Big-Boy-420]

Hi,

Thanks for getting back to me.

As requested:

contents of mlc-llm/3rdparty/tvm:

3rdparty  CMakeLists.txt  CONTRIBUTORS.md  golang   LICENSE   NEWS.md         README.md  version.py
apps      conda       docker       include  licenses  NOTICE          rust   vta
ci    configs     docs         jvm      Makefile  pyproject.toml  src    web
cmake     conftest.py     gallery      KEYS     mypy.ini  python          tests

contents of tvm:

3rdparty  CMakeLists.txt  CONTRIBUTORS.md  golang   LICENSE   NEWS.md         README.md  version.py
apps      conda       docker       include  licenses  NOTICE          rust   vta
ci    configs     docs         jvm      Makefile  pyproject.toml  src    web
cmake     conftest.py     gallery      KEYS     mypy.ini  python          tests

[CharlieFRuan]

I see you have TVM_SOURCE_DIR_SET=/content/tvm/3rdparty/tvm. It should be TVM_SOURCE_DIR_SET=/content/tvm

[Big-Boy-420]

That has got it further - I am thinking that was perhaps a misunderstanding from my part based on Step 2 - or perhaps adjust the wording so its more obvious? :p

It gets onto the compile stage now but that fails with the below output:

[2024-11-04 20:38:01] INFO pipeline.py:54: Compiling external modules
[2024-11-04 20:38:01] INFO pipeline.py:54: Compilation complete! Exporting to disk
[20:38:06] /workspace/tvm/src/target/llvm/codegen_llvm.cc:185: Warning: Set native vector bits to be 128 for wasm32
Traceback (most recent call last):
  File "/usr/local/bin/mlc_llm", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/mlc_llm/__main__.py", line 33, in main
    cli.main(sys.argv[2:])
  File "/usr/local/lib/python3.10/dist-packages/mlc_llm/cli/compile.py", line 129, in main
    compile(
  File "/usr/local/lib/python3.10/dist-packages/mlc_llm/interface/compile.py", line 243, in compile
    _compile(args, model_config)
  File "/usr/local/lib/python3.10/dist-packages/mlc_llm/interface/compile.py", line 188, in _compile
    args.build_func(
  File "/usr/local/lib/python3.10/dist-packages/mlc_llm/support/auto_target.py", line 258, in build
    relax.build(
  File "/usr/local/lib/python3.10/dist-packages/tvm/relax/vm_build.py", line 146, in export_library
    return self.mod.export_library(
  File "/usr/local/lib/python3.10/dist-packages/tvm/runtime/module.py", line 628, in export_library
    return fcompile(file_name, files, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tvm/contrib/emcc.py", line 74, in create_tvmjs_wasm
    all_libs += [find_lib_path("wasm_runtime.bc")[0]]
  File "/usr/local/lib/python3.10/dist-packages/tvm/_ffi/libinfo.py", line 166, in find_lib_path
    raise RuntimeError(message)
RuntimeError: Cannot find libraries: wasm_runtime.bc
List of candidates:
/usr/lib64-nvidia/wasm_runtime.bc
/content/emsdk/upstream/emscripten/wasm_runtime.bc
/content/emsdk/wasm_runtime.bc
/opt/bin/wasm_runtime.bc
/usr/local/cuda-12.2/bin/wasm_runtime.bc
/usr/local/sbin/wasm_runtime.bc
/usr/local/bin/wasm_runtime.bc
/usr/sbin/wasm_runtime.bc
/usr/bin/wasm_runtime.bc
/usr/sbin/wasm_runtime.bc
/usr/bin/wasm_runtime.bc
/tools/node/bin/wasm_runtime.bc
/tools/google-cloud-sdk/bin/wasm_runtime.bc
/usr/local/lib/python3.10/dist-packages/tvm/wasm_runtime.bc
/usr/local/lib/wasm_runtime.bc

I ran !ls -l ${TVM_SOURCE_DIR}/web/dist/wasm/*.bc which produced

-rw-r--r-- 1 root root  166140 Nov  4 20:35 /content/tvm/web/dist/wasm/tvmjs_support.bc
-rw-r--r-- 1 root root 4533080 Nov  4 20:35 /content/tvm/web/dist/wasm/wasm_runtime.bc
-rw-r--r-- 1 root root  181292 Nov  4 20:35 /content/tvm/web/dist/wasm/webgpu_runtime.bc

Thank you so much for all your help so far - it looks like we're very close.

[Big-Boy-420]

Hi @CharlieFRuan, sorry to keep on - any ideas?