Phi-2 q4f16_1 runs faster when compiled without `tvm.relax.transform.FuseOps()` and `tvm.relax.transform.FuseTIR()` transformations

MMuzzammil1 commented 6 months ago

🐛 Bug

When I compile Phi-2 (https://huggingface.co/microsoft/phi-2) with tvm.relax.transform.FuseOps() and tvm.relax.transform.FuseTIR() transformations commented out (https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/compiler_pass/pipeline.py#L128), I get better prefill and decode speeds on Cuda.

To Reproduce

To compile, I used the following command: mlc_llm compile Phi2/phi-2-q4f16_1-MLC/mlc-chat-config.json --device cuda -o Phi2/phi-2-q4f16_1-MLC/phi-2-q4f16_1-cuda.so

With tvm.relax.transform.FuseOps() and tvm.relax.transform.FuseTIR():

Statistics: ----------- prefill -----------
throughput: 283.674 tok/s
total tokens: 12 tok
total time: 0.042 s
------------ decode ------------
throughput: 101.508 tok/s
total tokens: 31 tok
total time: 0.305 s

Without tvm.relax.transform.FuseOps() and tvm.relax.transform.FuseTIR():

Statistics: ----------- prefill -----------
throughput: 291.720 tok/s
total tokens: 12 tok
total time: 0.041 s
------------ decode ------------
throughput: 129.715 tok/s
total tokens: 31 tok
total time: 0.239 s

Expected behavior

I believe that the expected behavior should be faster performance when FusedOps transformation in performed on the IR.

Environment

Platform: CUDA
Operating system: Ubuntu
Device: PC + RTX 2080
Python version: 3.10
GPU driver version: 530.41.03
CUDA/cuDNN version: 12.1

TVM Unity Hash Tag (python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))", applicable if you compile models):

USE_NVTX: OFF
USE_GTEST: AUTO
SUMMARIZE: OFF
TVM_DEBUG_WITH_ABI_CHANGE: OFF
USE_IOS_RPC: OFF
USE_MSC: OFF
USE_ETHOSU: OFF
CUDA_VERSION: 12.1
USE_LIBBACKTRACE: AUTO
DLPACK_PATH: 3rdparty/dlpack/include
USE_TENSORRT_CODEGEN: OFF
USE_THRUST: ON
USE_TARGET_ONNX: OFF
USE_AOT_EXECUTOR: ON
BUILD_DUMMY_LIBTVM: OFF
USE_CUDNN: ON
USE_TENSORRT_RUNTIME: OFF
USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR: OFF
USE_CCACHE: AUTO
USE_ARM_COMPUTE_LIB: OFF
USE_CPP_RTVM: OFF
USE_OPENCL_GTEST: /path/to/opencl/gtest
USE_MKL: OFF
USE_PT_TVMDSOOP: OFF
MLIR_VERSION: NOT-FOUND
USE_CLML: OFF
USE_STACKVM_RUNTIME: OFF
USE_GRAPH_EXECUTOR_CUDA_GRAPH: OFF
ROCM_PATH: /opt/rocm
USE_DNNL: OFF
USE_VITIS_AI: OFF
USE_MLIR: ON
USE_RCCL: OFF
USE_LLVM: ON
USE_VERILATOR: OFF
USE_TF_TVMDSOOP: OFF
USE_THREADS: ON
USE_MSVC_MT: OFF
BACKTRACE_ON_SEGFAULT: OFF
USE_GRAPH_EXECUTOR: ON
USE_NCCL: OFF
USE_ROCBLAS: OFF
GIT_COMMIT_HASH: d694451c580a931116a2c93571f21f7d791c7fa0
USE_VULKAN: OFF
USE_RUST_EXT: OFF
USE_CUTLASS: ON
USE_CPP_RPC: OFF
USE_HEXAGON: ON
USE_CUSTOM_LOGGING: OFF
USE_UMA: OFF
USE_FALLBACK_STL_MAP: OFF
USE_SORT: ON
USE_RTTI: ON
GIT_COMMIT_TIME: 2024-04-18 10:05:07 -0400
USE_HEXAGON_SDK: /shared-volume/Qualcomm/Hexagon_SDK/3.5.4/
USE_BLAS: none
USE_ETHOSN: OFF
USE_LIBTORCH: OFF
USE_RANDOM: ON
USE_CUDA: ON
USE_COREML: OFF
USE_AMX: OFF
BUILD_STATIC_RUNTIME: OFF
USE_CMSISNN: OFF
USE_KHRONOS_SPIRV: OFF
USE_CLML_GRAPH_EXECUTOR: OFF
USE_TFLITE: OFF
USE_HEXAGON_GTEST: /path/to/hexagon/gtest
PICOJSON_PATH: 3rdparty/picojson
USE_OPENCL_ENABLE_HOST_PTR: ON
INSTALL_DEV: OFF
USE_PROFILER: ON
USE_NNPACK: OFF
LLVM_VERSION: 15.0.7
USE_MRVL: OFF
USE_OPENCL: ON
COMPILER_RT_PATH: 3rdparty/compiler-rt
RANG_PATH: 3rdparty/rang/include
USE_SPIRV_KHR_INTEGER_DOT_PRODUCT: OFF
USE_OPENMP: none
USE_BNNS: OFF
USE_FLASHINFER: OFF
USE_CUBLAS: ON
USE_METAL: OFF
USE_MICRO_STANDALONE_RUNTIME: OFF
USE_HEXAGON_EXTERNAL_LIBS: OFF
USE_ALTERNATIVE_LINKER: AUTO
USE_BYODT_POSIT: OFF
USE_HEXAGON_RPC: OFF
USE_MICRO: OFF
DMLC_PATH: 3rdparty/dmlc-core/include
INDEX_DEFAULT_I64: ON
USE_RELAY_DEBUG: OFF
USE_RPC: ON
USE_TENSORFLOW_PATH: none
TVM_CLML_VERSION: 
USE_MIOPEN: OFF
USE_ROCM: OFF
USE_PAPI: OFF
USE_CURAND: OFF
TVM_CXX_COMPILER_PATH: /usr/bin/c++
HIDE_PRIVATE_SYMBOLS: OFF

0xDEADFED5 commented 6 months ago

i can't reproduce this on Intel hardware, but i was hopeful =) i had to write my own benchmark because the mlc_llm bench command was removed. i averaged the results of 5 sequential benchmarks. i set max_tokens=1000 to try to smooth out results a bit. hardware: Intel Arc A770 16GB model tested: https://huggingface.co/RLHFlow/LLaMA3-iterative-DPO-final quantization: q4f16_1

results with standard compile from nightly wheel (0.1.dev1287):

(completion tokens: 694) Completion tokens/sec =  29.19
(completion tokens: 558) Completion tokens/sec =  28.68
(completion tokens: 462) Completion tokens/sec =  30.96
(completion tokens: 576) Completion tokens/sec =  30.12
(completion tokens: 717) Completion tokens/sec =  29.36
mean tokens = 601.4, mean speed = 29.662

results without tvm.relax.transform.FuseOps() and tvm.relax.transform.FuseTIR():

(completion tokens: 475) Completion tokens/sec =  26.87
(completion tokens: 691) Completion tokens/sec =  27.98
(completion tokens: 568) Completion tokens/sec =  29.80
(completion tokens: 504) Completion tokens/sec =  30.13
(completion tokens: 434) Completion tokens/sec =  30.68
mean tokens = 534.4, mean speed = 29.092

MMuzzammil1 commented 6 months ago

Hi @0xDEADFED5. I created this issue for the "Phi-2" model (https://huggingface.co/microsoft/phi-2). Not sure of the behaviour of the Llama-3.

0xDEADFED5 commented 5 months ago

yes, i know, just adding more data. i'm on cellular as my only internet so I can't test that model.

i bet if you did more benchmarks with more tokens your numbers would stabilize

MMuzzammil1 commented 5 months ago

I'll run the benchmarks to check that. But @0xDEADFED5 isn't the decode speed at least independent of the prompt input?

mlc-ai / mlc-llm