[Bug] ByT5 not supported

[acalatrava]

🐛 Bug

While playing with PY007/ByteLlama-320M-preview which uses ByT5 tokenizer (AKA no tokenizer, it just operates on bytes) I get the following error:

Stack trace:
  [bt] (0) 1   libmlc_llm.dylib                    0x0000000103ce7e14 tvm::runtime::detail::LogFatal::Entry::Finalize() + 68
  [bt] (1) 2   libmlc_llm.dylib                    0x0000000103ce7dd0 tvm::runtime::detail::LogFatal::Entry::Finalize() + 0
  [bt] (2) 3   libmlc_llm.dylib                    0x0000000103ce6d6c std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::~vector[abi:v160006]() + 0
  [bt] (3) 4   libmlc_llm.dylib                    0x0000000103cfdfa8 mlc::llm::LLMChat::Reload(tvm::runtime::TVMArgValue, tvm::runtime::String, tvm::runtime::String) + 9376
  [bt] (4) 5   libmlc_llm.dylib                    0x0000000103cfb7c8 mlc::llm::LLMChatModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::'lambda'(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const + 488
  [bt] (5) 6   libmlc_llm.dylib                    0x0000000103cfb5d4 tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<mlc::llm::LLMChatModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::'lambda'(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>>::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) + 40
  [bt] (6) 7   mlc_chat_cli                        0x0000000102dd0894 ChatModule::Reload(ModelPaths const&) + 644
  [bt] (7) 8   mlc_chat_cli                        0x0000000102dcfd78 Chat(ChatModule*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, int) + 92
  [bt] (8) 9   mlc_chat_cli                        0x0000000102dd164c main + 1012

To Reproduce

Steps to reproduce the behavior:

1. Try to use any ByT5 tokenizer

Expected behavior

Support for ByT5. I guess that adding ByT5 tokenizer should be easy since, in the end, is to remove the tokenizer, but I'm not sure about it.

Environment

• Platform: Metal
• Operating system: MacOS
• Device: MacBook Pro M1
• How you installed MLC-LLM: source
• How you installed TVM-Unity: source
• Python version: 3.11
• GPU driver version:
• CUDA/cuDNN version:
• TVM Unity Hash Tag (python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))", applicable if you compile models):

  USE_GTEST: AUTO
  SUMMARIZE: OFF
  USE_IOS_RPC: OFF
  USE_MSC: OFF
  USE_ETHOSU: OFF
  CUDA_VERSION: NOT-FOUND
  USE_LIBBACKTRACE: AUTO
  DLPACK_PATH: 3rdparty/dlpack/include
  USE_TENSORRT_CODEGEN: OFF
  USE_THRUST: OFF
  USE_TARGET_ONNX: OFF
  USE_AOT_EXECUTOR: ON
  BUILD_DUMMY_LIBTVM: OFF
  USE_CUDNN: OFF
  USE_TENSORRT_RUNTIME: OFF
  USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR: OFF
  USE_CCACHE: AUTO
  USE_ARM_COMPUTE_LIB: OFF
  USE_CPP_RTVM: OFF
  USE_OPENCL_GTEST: /path/to/opencl/gtest
  USE_MKL: OFF
  USE_PT_TVMDSOOP: OFF
  MLIR_VERSION: NOT-FOUND
  USE_CLML: OFF
  USE_STACKVM_RUNTIME: OFF
  USE_GRAPH_EXECUTOR_CUDA_GRAPH: OFF
  ROCM_PATH: /opt/rocm
  USE_DNNL: OFF
  USE_VITIS_AI: OFF
  USE_MLIR: OFF
  USE_RCCL: OFF
  USE_LLVM: llvm-config --ignore-libllvm --link-static
  USE_VERILATOR: OFF
  USE_TF_TVMDSOOP: OFF
  USE_THREADS: ON
  USE_MSVC_MT: OFF
  BACKTRACE_ON_SEGFAULT: OFF
  USE_GRAPH_EXECUTOR: ON
  USE_NCCL: OFF
  USE_ROCBLAS: OFF
  GIT_COMMIT_HASH: 50a4dce98841ecb825143a79019142dcd1fe9611
  USE_VULKAN: OFF
  USE_RUST_EXT: OFF
  USE_CUTLASS: OFF
  USE_CPP_RPC: OFF
  USE_HEXAGON: OFF
  USE_CUSTOM_LOGGING: OFF
  USE_UMA: OFF
  USE_FALLBACK_STL_MAP: OFF
  USE_SORT: ON
  USE_RTTI: ON
  GIT_COMMIT_TIME: 2023-09-17 12:11:15 -0700
  USE_HEXAGON_SDK: /path/to/sdk
  USE_BLAS: none
  USE_ETHOSN: OFF
  USE_LIBTORCH: OFF
  USE_RANDOM: ON
  USE_CUDA: OFF
  USE_COREML: OFF
  USE_AMX: OFF
  BUILD_STATIC_RUNTIME: OFF
  USE_CMSISNN: OFF
  USE_KHRONOS_SPIRV: OFF
  USE_CLML_GRAPH_EXECUTOR: OFF
  USE_TFLITE: OFF
  USE_HEXAGON_GTEST: /path/to/hexagon/gtest
  PICOJSON_PATH: 3rdparty/picojson
  USE_OPENCL_ENABLE_HOST_PTR: OFF
  INSTALL_DEV: OFF
  USE_PROFILER: ON
  USE_NNPACK: OFF
  LLVM_VERSION: 16.0.6
  USE_OPENCL: OFF
  COMPILER_RT_PATH: 3rdparty/compiler-rt
  RANG_PATH: 3rdparty/rang/include
  USE_SPIRV_KHR_INTEGER_DOT_PRODUCT: OFF
  USE_OPENMP: none
  USE_BNNS: OFF
  USE_CUBLAS: OFF
  USE_METAL: ON
  USE_MICRO_STANDALONE_RUNTIME: OFF
  USE_HEXAGON_EXTERNAL_LIBS: OFF
  USE_ALTERNATIVE_LINKER: AUTO
  USE_BYODT_POSIT: OFF
  USE_HEXAGON_RPC: OFF
  USE_MICRO: OFF
  DMLC_PATH: 3rdparty/dmlc-core/include
  INDEX_DEFAULT_I64: ON
  USE_RELAY_DEBUG: OFF
  USE_RPC: ON
  USE_TENSORFLOW_PATH: none
  TVM_CLML_VERSION:
  USE_MIOPEN: OFF
  USE_ROCM: OFF
  USE_PAPI: OFF
  USE_CURAND: OFF
  TVM_CXX_COMPILER_PATH: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
  HIDE_PRIVATE_SYMBOLS: ON

• Any other relevant information:

Additional context

[tqchen]

This seems to be something that can be contributed to tokenizer cpp