apache / tvm

Open deep learning compiler stack for cpu, gpu and specialized accelerators
https://tvm.apache.org/
Apache License 2.0
11.41k stars 3.4k forks source link

[Bug] Failed to build I-ViT #16358

Open Yanhan-cmd opened 6 months ago

Yanhan-cmd commented 6 months ago

图片 Steps to reproduce the behavior: git clone https://github.com/zkkli/I-ViT.git Just follow the TVM_benchmark.

One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details. Traceback (most recent call last): File "D:/TVM/I-ViT/TVM_benchmark/evaluate_accuracy.py", line 97, in main() File "D:/TVM/I-ViT/TVM_benchmark/evaluate_accuracy.py", line 81, in main lib = relay.build(func, target=target, params=pretrained_params) File "D:\Anaconda3\envs\tvm-build\lib\site-packages\tvm-0.15.dev103+g506eff23b-py3.8-win-amd64.egg\tvm\relay\build_module.py", line 364, in build graph_json, runtime_mod, params = bld_mod.build( File "D:\Anaconda3\envs\tvm-build\lib\site-packages\tvm-0.15.dev103+g506eff23b-py3.8-win-amd64.egg\tvm\relay\build_module.py", line 161, in build self._build( File "D:\Anaconda3\envs\tvm-build\lib\site-packages\tvm-0.15.dev103+g506eff23b-py3.8-win-amd64.egg\tvm_ffi_ctypes\packed_func.py", line 229, in call _LIB.TVMFuncCall( OSError: exception: stack overflow

tvm.version :0.15.dev0

Platform: LAPTOP

Operating system: Windows

Device: PC+RTX 3070

Python version: 3.8

GPU driver version (if applicable):12.0

CUDA/cuDNN version (if applicable):12.0

Jocx-H commented 6 months ago

I have the same problem as you, but the difference is that: When I use Ansor tuning vit_b_32.onnx, the tvm.auto_scheduler.extract_tasks is crushed. I print some messages in tvm.auto_scheduler.relay_integration.auto_schedule_topiimage This information shows that the problem seems to occur when fusing layout_transform, reshape_transpose and concatenate_add. Maybe the underlying reason for the problem you encountered is also this.

twmht commented 2 weeks ago

@Yanhan-cmd @Jocx-H

I have also been testing I-VIT recently, and you can switch the TVM version to v0.9.0. However, I found that its speed is far from as fast as claimed in the paper, and FasterTransformer is not as slow as the paper suggests