[Bug] TypeError: argument of type 'LlamaConfig' is not iterable

[daniel-kukiela]

🐛 Bug

Currently the latest comment, which is https://github.com/mlc-ai/mlc-llm/commit/61ce2c226c0c611d00b29ee8db6acb11af93e9d6, causes:

/usr/local/lib/python3.11/dist-packages/mlc_llm/core.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/dist-packages/mlc_llm/build.py", line 13, in <module>
    main()
  File "/usr/local/lib/python3.11/dist-packages/mlc_llm/build.py", line 10, in main
    core.build_model_from_args(parsed_args)
  File "/usr/local/lib/python3.11/dist-packages/mlc_llm/core.py", line 583, in build_model_from_args
    mod = mod_transform_before_build(mod, param_manager, args, model_config)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/mlc_llm/core.py", line 362, in mod_transform_before_build
    if "num_attention_heads" in config and "hidden_size" in config:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'LlamaConfig' is not iterable
``` during model building.
I'm getting it with the `meta-llama/Llama-2-70b-chat-hf` model. Hard-resetting the repository to the previous commit 

## To Reproduce

Steps to reproduce the behavior:

1. Install `mlc-llm` from GitHub.

## Expected behavior

The code does not break.

## Environment

 - Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): CUDA
 - Operating system (e.g. Ubuntu/Windows/MacOS/...): Ubuntu 22.04
 - Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...): RTX 8000
 - How you installed MLC-LLM (`conda`, source): source
 - How you installed TVM-Unity (`pip`, source): pip
 - Python version (e.g. 3.10): 3.11
 - GPU driver version (if applicable): 535.86.10
 - CUDA/cuDNN version (if applicable): 10.8/8
 - TVM Unity Hash Tag (`python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))"`, applicable if you compile models):

USE_GTEST: AUTO SUMMARIZE: OFF USE_IOS_RPC: OFF USE_ETHOSU: CUDA_VERSION: 11.8 USE_LIBBACKTRACE: AUTO DLPACK_PATH: 3rdparty/dlpack/include USE_TENSORRT_CODEGEN: OFF USE_THRUST: OFF USE_TARGET_ONNX: OFF USE_AOT_EXECUTOR: ON BUILD_DUMMY_LIBTVM: OFF USE_CUDNN: OFF USE_TENSORRT_RUNTIME: OFF USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR: OFF USE_CCACHE: AUTO USE_ARM_COMPUTE_LIB: OFF USE_CPP_RTVM: USE_OPENCL_GTEST: /path/to/opencl/gtest USE_MKL: OFF USE_PT_TVMDSOOP: OFF USE_CLML: OFF USE_STACKVM_RUNTIME: OFF USE_GRAPH_EXECUTOR_CUDA_GRAPH: OFF ROCM_PATH: /opt/rocm USE_DNNL: OFF USE_VITIS_AI: OFF USE_LLVM: llvm-config --ignore-libllvm --link-static USE_VERILATOR: OFF USE_TF_TVMDSOOP: OFF USE_THREADS: ON USE_MSVC_MT: OFF BACKTRACE_ON_SEGFAULT: OFF USE_GRAPH_EXECUTOR: ON USE_ROCBLAS: OFF GIT_COMMIT_HASH: be21b378b284eeab5b6a7721bf417ad7445fddf0 USE_VULKAN: ON USE_RUST_EXT: OFF USE_CUTLASS: ON USE_CPP_RPC: OFF USE_HEXAGON: OFF USE_CUSTOM_LOGGING: OFF USE_UMA: OFF USE_FALLBACK_STL_MAP: OFF USE_SORT: ON USE_RTTI: ON GIT_COMMIT_TIME: 2023-08-16 07:41:34 -0400 USE_HEXAGON_SDK: /path/to/sdk USE_BLAS: none USE_ETHOSN: OFF USE_LIBTORCH: OFF USE_RANDOM: ON USE_CUDA: ON USE_COREML: OFF USE_AMX: OFF BUILD_STATIC_RUNTIME: OFF USE_CMSISNN: OFF USE_KHRONOS_SPIRV: OFF USE_CLML_GRAPH_EXECUTOR: OFF USE_TFLITE: OFF USE_HEXAGON_GTEST: /path/to/hexagon/gtest PICOJSON_PATH: 3rdparty/picojson USE_OPENCL_ENABLE_HOST_PTR: OFF INSTALL_DEV: OFF USE_PROFILER: ON USE_NNPACK: OFF LLVM_VERSION: 15.0.7 USE_OPENCL: OFF COMPILER_RT_PATH: 3rdparty/compiler-rt RANG_PATH: 3rdparty/rang/include USE_SPIRV_KHR_INTEGER_DOT_PRODUCT: OFF USE_OPENMP: OFF USE_BNNS: OFF USE_CUBLAS: OFF USE_METAL: OFF USE_MICRO_STANDALONE_RUNTIME: OFF USE_HEXAGON_EXTERNAL_LIBS: OFF USE_ALTERNATIVE_LINKER: AUTO USE_BYODT_POSIT: OFF USE_HEXAGON_RPC: OFF USE_MICRO: OFF DMLC_PATH: 3rdparty/dmlc-core/include INDEX_DEFAULT_I64: ON USE_RELAY_DEBUG: OFF USE_RPC: ON USE_TENSORFLOW_PATH: none TVM_CLML_VERSION: USE_MIOPEN: OFF USE_ROCM: OFF USE_PAPI: OFF USE_CURAND: OFF TVM_CXX_COMPILER_PATH: /opt/rh/gcc-toolset-11/root/usr/bin/c++ HIDE_PRIVATE_SYMBOLS: ON

 - Any other relevant information: -

## Additional context

-

[sudeepag]

@daniel-kukiela Potential fix in https://github.com/mlc-ai/mlc-llm/pull/825#discussion_r1308197341.

[David-Sharma]

having the same issue

TypeError: argument of type 'LlamaConfig' is not iterable

[tstanek390]

File "/Users/admin/AI/mlc-llm/mlc-llm/mlc_llm/core.py", line 584, in build_model_from_args mod = mod_transform_before_build(mod, param_manager, args, model_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/admin/AI/mlc-llm/mlc-llm/mlc_llm/core.py", line 367, in mod_transform_before_build elif "max_sequence_length" in config: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: argument of type 'LlamaConfig' is not iterable

Same here

[jparismorgan]

@daniel-kukiela Potential fix in #825 (comment).

I'm still getting this error with the changes from that PR. Is there a workaround / fix that anyone has found and I could test out? Thanks!

[David-Sharma]

Not working for me either. Hopefully someone has a workaround or fix.

[MasterJH5574]

Just merged #831 which fixes this. Would you folks like to pull and try again? Sorry to see things get messed up these days.

[tstanek390]

Those changes from PR don't solve the issue, would be glad for the fix too ! Thx

[jparismorgan]

Just merged #831 which fixes this. Would you folks like to pull and try again? Sorry to see things get messed up these days.

With this PR I am no longer getting TypeError: argument of type 'LlamaConfig' is not iterable. Thank you!

[David-Sharma]

Please trigger the nightly.