[Bug] Does MLC LLM support function call ？

[Stephen888888]

🐛 Bug

To Reproduce

Steps to reproduce the behavior:

1. mlc_llm convert_weight ./dist/models/Qwen2-1.5B-Instruct/ --model-type qwen2 --quantization q0f16 -o dist/Qwen2-1.5B-Instruct-q0f16-MLC
2. mlc_llm gen_config ./dist/models/Qwen2-1.5B-Instruct/ --model-type qwen2 --quantization q0f16 --conv-template chatml --context-window-size 2048 --max-batch-size 1 -o dist/Qwen2-1.5B-Instruct-q0f16-MLC/
3. mlc_llm package

Expected behavior

Environment

• Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): Android
• Operating system (e.g. Ubuntu/Windows/MacOS/...): Ubuntu
• Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...): Mate50 骁龙8 1 GEN
• How you installed MLC-LLM (conda, source): source
• How you installed TVM-Unity (pip, source): pip
• Python version (e.g. 3.10): 3.10
• GPU driver version (if applicable):
• CUDA/cuDNN version (if applicable):
• TVM Unity Hash Tag (python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))", applicable if you compile models):
• USE_NVTX: OFF USE_GTEST: AUTO SUMMARIZE: OFF TVM_DEBUG_WITH_ABI_CHANGE: OFF USE_IOS_RPC: OFF USE_MSC: OFF USE_ETHOSU: CUDA_VERSION: NOT-FOUND USE_LIBBACKTRACE: AUTO DLPACK_PATH: 3rdparty/dlpack/include USE_TENSORRT_CODEGEN: OFF USE_THRUST: OFF USE_TARGET_ONNX: OFF USE_AOT_EXECUTOR: ON BUILD_DUMMY_LIBTVM: OFF USE_CUDNN: OFF USE_TENSORRT_RUNTIME: OFF USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR: OFF USE_CCACHE: AUTO USE_ARM_COMPUTE_LIB: OFF USE_CPP_RTVM: USE_OPENCL_GTEST: /path/to/opencl/gtest TVM_LOG_BEFORE_THROW: OFF USE_MKL: OFF USE_PT_TVMDSOOP: OFF MLIR_VERSION: NOT-FOUND USE_CLML: OFF USE_STACKVM_RUNTIME: OFF USE_GRAPH_EXECUTOR_CUDA_GRAPH: OFF ROCM_PATH: /opt/rocm USE_DNNL: OFF USE_MSCCL: OFF USE_VITIS_AI: OFF USE_MLIR: OFF USE_RCCL: OFF USE_LLVM: llvm-config --ignore-libllvm --link-static USE_VERILATOR: OFF USE_TF_TVMDSOOP: OFF USE_THREADS: ON USE_MSVC_MT: OFF BACKTRACE_ON_SEGFAULT: OFF USE_GRAPH_EXECUTOR: ON USE_NCCL: OFF USE_ROCBLAS: OFF GIT_COMMIT_HASH: f5f048bbd71513f087799f987019e3931f68a6d9 USE_VULKAN: ON USE_RUST_EXT: OFF USE_CUTLASS: OFF USE_CPP_RPC: OFF USE_HEXAGON: OFF USE_CUSTOM_LOGGING: OFF USE_UMA: OFF USE_FALLBACK_STL_MAP: OFF USE_SORT: ON USE_RTTI: ON GIT_COMMIT_TIME: 2024-06-03 09:18:53 -0400 USE_HEXAGON_SDK: /path/to/sdk USE_BLAS: none USE_ETHOSN: OFF USE_LIBTORCH: OFF USE_RANDOM: ON USE_CUDA: OFF USE_COREML: OFF USE_AMX: OFF BUILD_STATIC_RUNTIME: OFF USE_CMSISNN: OFF USE_KHRONOS_SPIRV: OFF USE_CLML_GRAPH_EXECUTOR: OFF USE_TFLITE: OFF USE_HEXAGON_GTEST: /path/to/hexagon/gtest PICOJSON_PATH: 3rdparty/picojson USE_OPENCL_ENABLE_HOST_PTR: OFF INSTALL_DEV: OFF USE_PROFILER: ON USE_NNPACK: OFF LLVM_VERSION: 15.0.7 USE_MRVL: OFF USE_OPENCL: OFF COMPILER_RT_PATH: 3rdparty/compiler-rt RANG_PATH: 3rdparty/rang/include USE_SPIRV_KHR_INTEGER_DOT_PRODUCT: OFF USE_OPENMP: OFF USE_BNNS: OFF USE_FLASHINFER: USE_CUBLAS: OFF USE_METAL: OFF USE_MICRO_STANDALONE_RUNTIME: OFF USE_HEXAGON_EXTERNAL_LIBS: OFF USE_ALTERNATIVE_LINKER: AUTO USE_BYODT_POSIT: OFF USE_HEXAGON_RPC: OFF USE_MICRO: OFF DMLC_PATH: 3rdparty/dmlc-core/include INDEX_DEFAULT_I64: ON USE_RELAY_DEBUG: OFF USE_RPC: ON USE_TENSORFLOW_PATH: none TVM_CLML_VERSION: USE_MIOPEN: OFF USE_ROCM: OFF USE_PAPI: OFF USE_CURAND: OFF TVM_CXX_COMPILER_PATH: /opt/rh/gcc-toolset-11/root/usr/bin/c++ HIDE_PRIVATE_SYMBOLS: ON
• Any other relevant information:

Additional context

[panghongtao]

Your actions should have generated the mlc model file. You can run it on Android or ios. I thought I'd give your idea a try

[Stephen888888]

@panghongtao yes，already run on android phone. but i need function call

[panghongtao]

You can try using their Android demo app for a while. Make sure that the model you need can be used before developing it. Android is open source. If you want to include this feature in your own APP. Maybe you need to develop. You want a demo interface, right?

[Stephen888888]

@panghongtao yes，can you give me a demo interface ？

[panghongtao]

you can dowland this https://objects.githubusercontent.com/github-production-release-asset-2e65be/633675943/81fe934a-807d-4f70-8d36-84b5e6fc4445?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240620%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240620T025018Z&X-Amz-Expires=300&X-Amz-Signature=b65b2a4fc2e1e8ce4820d5f1a2484947ae46b96279ac283abac2182f3dab17ea&X-Amz-SignedHeaders=host&actor_id=152578991&key_id=0&repo_id=633675943&response-content-disposition=attachment%3B%20filename%3Dmlc-chat.apk&response-content-type=application%2Fvnd.android.package-archive

and this android API you can look it

https://llm.mlc.ai/docs/deploy/android.html

if you want change model. you can look mlc app's files folder. this folder is mlc model.