search

apache / tvm

Open deep learning compiler stack for cpu, gpu and specialized accelerators
https://tvm.apache.org/
Apache License 2.0
11.73k stars 3.46k forks source link

[Bug] Coredump when import torch after tvm #14682

Closed LoSealL closed 1 year ago

LoSealL commented 1 year ago

Expected behavior

No core dump

Actual behavior

Core dump when program exits

Environment

tangwy @ swalgo-server1 in ~/works/tvm on git:main o [21:41:21] C:134 $ pip show tvm Name: tvm Version: 0.13.dev54+g2bb9698be Summary: TVM: An End to End Tensor IR/DSL Stack for Deep Learning Systems Home-page: https://tvm.apache.org/ Author: Apache TVM Author-email: License: Apache Location: /home/tangwy/works/tvm/python Editable project location: /home/tangwy/works/tvm/python Requires: attrs, cloudpickle, decorator, numpy, psutil, scipy, tornado Required-by: (tvm) tangwy @ swalgo-server1 in ~/works/tvm on git:main o [21:41:25] $ pip show torch Name: torch Version: 2.0.0 Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration Home-page: https://pytorch.org/ Author: PyTorch Team Author-email: packages@pytorch.org License: BSD-3 Location: /home/tangwy/.conda/envs/tvm/lib/python3.8/site-packages Requires: filelock, jinja2, networkx, nvidia-cublas-cu11, nvidia-cuda-cupti-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-runtime-cu11, nvidia-cudnn-cu11, nvidia-cufft-cu11, nvidia-curand-cu11, nvidia-cusolver-cu11, nvidia-cusparse-cu11, nvidia-nccl-cu11, nvidia-nvtx-cu11, sympy, triton, typing-extensions Required-by: triton (tvm) tangwy @ swalgo-server1 in ~/works/tvm on git:main o [21:41:29] $ python -V Python 3.8.2 (tvm) tangwy @ swalgo-server1 in ~/works/tvm on git:main o [21:41:46] $ python -c "import tvm; import torch" free(): invalid pointer [1] 835988 abort (core dumped) python -c "import tvm; import torch" (tvm) tangwy @ swalgo-server1 in ~/works/tvm on git:main o [21:41:58] C:134 $ python -c "import torch; import tvm" (tvm) tangwy @ swalgo-server1 in ~/works/tvm on git:main o [21:46:21] C:1 $ python -c "import tvm;print(tvm.get_global_func('support.GetLibInfo')())" {"USE_GTEST": "AUTO", "SUMMARIZE": "OFF", "USE_IOS_RPC": "OFF", "CUDA_VERSION": "NOT-FOUND", "USE_LIBBACKTRACE": "AUTO", "DLPACK_PATH": "3rdparty/dlpack/include", "USE_TENSORRT_CODEGEN": "OFF", "USE_THRUST": "OFF", "USE_TARGET_ONNX": "OFF", "USE_AOT_EXECUTOR": "ON", "USE_CUDNN": "OFF", "USE_TENSORRT_RUNTIME": "OFF", "USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR": "OFF", "USE_CCACHE": "AUTO", "USE_ARM_COMPUTE_LIB": "OFF", "USE_CPP_RTVM": "", "USE_OPENCL_GTEST": "/path/to/opencl/gtest", "USE_MKL": "OFF", "USE_PT_TVMDSOOP": "OFF", "USE_CLML": "OFF", "USE_STACKVM_RUNTIME": "ON", "USE_GRAPH_EXECUTOR_CUDA_GRAPH": "OFF", "ROCM_PATH": "/opt/rocm", "USE_DNNL": "OFF", "USE_VITIS_AI": "OFF", "USE_LLVM": "ON", "USE_VERILATOR": "ON", "USE_TF_TVMDSOOP": "OFF", "USE_THREADS": "ON", "USE_MSVC_MT": "OFF", "BACKTRACE_ON_SEGFAULT": "OFF", "USE_GRAPH_EXECUTOR": "ON", "USE_ROCBLAS": "OFF", "GIT_COMMIT_HASH": "2bb9698be9876aa1588981a7021b0cb85117d296", "USE_VULKAN": "OFF", "USE_RUST_EXT": "OFF", "USE_CUTLASS": "OFF", "USE_CPP_RPC": "OFF", "USE_HEXAGON": "OFF", "USE_CUSTOM_LOGGING": "OFF", "USE_UMA": "OFF", "USE_FALLBACK_STL_MAP": "OFF", "USE_SORT": "ON", "USE_RTTI": "ON", "GIT_COMMIT_TIME": "2023-04-20 13:52:14 +0800", "USE_HEXAGON_SDK": "/path/to/sdk", "USE_BLAS": "none", "USE_ETHOSN": "OFF", "USE_LIBTORCH": "OFF", "USE_RANDOM": "ON", "USE_CUDA": "OFF", "USE_COREML": "OFF", "USE_AMX": "OFF", "BUILD_STATIC_RUNTIME": "OFF", "USE_CMSISNN": "OFF", "USE_KHRONOS_SPIRV": "OFF", "USE_CLML_GRAPH_EXECUTOR": "OFF", "USE_TFLITE": "OFF", "USE_HEXAGON_GTEST": "/path/to/hexagon/gtest", "PICOJSON_PATH": "3rdparty/picojson", "USE_OPENCL_ENABLE_HOST_PTR": "OFF", "INSTALL_DEV": "OFF", "USE_PROFILER": "ON", "USE_NNPACK": "OFF", "LLVM_VERSION": "14.0.6", "USE_OPENCL": "ON", "COMPILER_RT_PATH": "3rdparty/compiler-rt", "RANG_PATH": "3rdparty/rang/include", "USE_SPIRV_KHR_INTEGER_DOT_PRODUCT": "OFF", "USE_OPENMP": "none", "USE_BNNS": "OFF", "USE_CUBLAS": "OFF", "USE_METAL": "OFF", "USE_MICRO_STANDALONE_RUNTIME": "OFF", "USE_HEXAGON_EXTERNAL_LIBS": "OFF", "USE_ALTERNATIVE_LINKER": "AUTO", "USE_BYODT_POSIT": "OFF", "USE_HEXAGON_RPC": "OFF", "USE_MICRO": "OFF", "DMLC_PATH": "3rdparty/dmlc-core/include", "INDEX_DEFAULT_I64": "ON", "USE_RELAY_DEBUG": "OFF", "USE_RPC": "ON", "USE_TENSORFLOW_PATH": "none", "USE_MIOPEN": "OFF", "USE_ROCM": "OFF", "USE_PAPI": "OFF", "USE_CURAND": "OFF", "TVM_CXX_COMPILER_PATH": "/usr/bin/c++", "HIDE_PRIVATE_SYMBOLS": "OFF"}

Steps to reproduce

Import tvm, and then import torch. See logs above.

Triage

Please refer to the list of label tags here to find the relevant tags and add them below in a bullet format (example below).

masahi commented 1 year ago

See https://github.com/apache/tvm/issues/9362. The workaround is documented in https://tvm.apache.org/docs/install/from_source.html#build-the-shared-library