search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
8.55k stars 969 forks source link

`import tensorrt_llm` prints out `[TensorRT-LLM][INFO] Initializing MPI with thread mode 3` and gets stuck there #2405

Open mrakgr opened 3 hours ago

mrakgr commented 3 hours ago

System Info

System: DGX H100 OS: Ubuntu 22.04 Driver Version: 560.35.03
CUDA Version: 12.6 conda export:

name: trt
channels:
  - defaults
  - https://repo.anaconda.com/pkgs/main
  - https://repo.anaconda.com/pkgs/r
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - bzip2=1.0.8=h5eee18b_6
  - ca-certificates=2024.9.24=h06a4308_0
  - ld_impl_linux-64=2.40=h12ee557_0
  - libffi=3.4.4=h6a678d5_1
  - libgcc-ng=11.2.0=h1234567_1
  - libgfortran-ng=7.5.0=ha8ba4b0_17
  - libgfortran4=7.5.0=ha8ba4b0_17
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - libuuid=1.41.5=h5eee18b_0
  - mpi=1.0=mpich
  - mpi4py=3.1.4=py310hfc96bbd_0
  - mpich=3.3.2=hc856adb_0
  - ncurses=6.4=h6a678d5_0
  - openssl=3.0.15=h5eee18b_0
  - pip=24.2=py310h06a4308_0
  - python=3.10.15=he870216_1
  - readline=8.2=h5eee18b_0
  - setuptools=75.1.0=py310h06a4308_0
  - sqlite=3.45.3=h5eee18b_0
  - tk=8.6.14=h39e8969_0
  - wheel=0.44.0=py310h06a4308_0
  - xz=5.4.6=h5eee18b_1
  - zlib=1.2.13=h5eee18b_1
  - pip:
      - accelerate==1.0.1
      - aenum==3.1.15
      - aiohappyeyeballs==2.4.3
      - aiohttp==3.11.0b0
      - aiosignal==1.3.1
      - async-timeout==4.0.3
      - attrs==24.2.0
      - build==1.2.2.post1
      - certifi==2024.8.30
      - click-option-group==0.5.6
      - cloudpickle==3.1.0
      - colored==2.2.4
      - coloredlogs==15.0.1
      - cuda-python==12.6.0
      - datasets==3.1.0
      - diffusers==0.31.0
      - dill==0.3.8
      - distro==1.9.0
      - evaluate==0.4.3
      - frozenlist==1.5.0
      - fsspec==2024.9.0
      - h5py==3.10.0
      - humanfriendly==10.0
      - idna==3.10
      - importlib-metadata==8.5.0
      - lark==1.2.2
      - mpmath==1.3.0
      - multidict==6.1.0
      - multiprocess==0.70.16
      - networkx==3.4.2
      - ninja==1.11.1.1
      - numpy==1.26.4
      - nvidia-cublas-cu12==12.1.3.1
      - nvidia-cuda-cupti-cu12==12.1.105
      - nvidia-cuda-nvrtc-cu12==12.1.105
      - nvidia-cuda-runtime-cu12==12.1.105
      - nvidia-cudnn-cu12==9.1.0.70
      - nvidia-cufft-cu12==11.0.2.54
      - nvidia-curand-cu12==10.3.2.106
      - nvidia-cusolver-cu12==11.4.5.107
      - nvidia-cusparse-cu12==12.1.0.106
      - nvidia-modelopt==0.17.0
      - nvidia-nccl-cu12==2.20.5
      - nvidia-nvjitlink-cu12==12.6.77
      - nvidia-nvtx-cu12==12.1.105
      - onnx==1.17.0
      - onnx-graphsurgeon==0.5.2
      - openai==1.39.0
      - optimum==1.23.3
      - pandas==2.2.3
      - pillow==10.3.0
      - polygraphy==0.49.9
      - propcache==0.2.0
      - protobuf==5.29.0rc2
      - psutil==6.1.0
      - pulp==2.9.0
      - pyarrow==18.0.0
      - pynvml==11.5.3
      - pyproject-hooks==1.2.0
      - python-dateutil==2.9.0.post0
      - pytz==2024.2
      - pyyaml==6.0.2
      - safetensors==0.4.5
      - scipy==1.14.1
      - sentencepiece==0.2.0
      - six==1.16.0
      - strenum==0.4.15
      - sympy==1.13.3
      - tensorrt==10.5.0
      - tensorrt-cu12==10.5.0
      - tensorrt-cu12-bindings==10.5.0
      - tensorrt-cu12-libs==10.5.0
      - tensorrt-llm==0.15.0.dev2024102900
      - tokenizers==0.20.1
      - tomli==2.0.2
      - torch==2.4.1
      - torchprofile==0.0.4
      - torchvision==0.19.1
      - transformers==4.45.1
      - triton==3.0.0
      - tzdata==2024.2
      - urllib3==2.2.3
      - xxhash==3.5.0
      - yarl==1.17.1
      - zipp==3.20.2
prefix: /home/ceti/miniconda3/envs/trt

Who can help?

No response

Information

Tasks

Reproduction

Execute python3 -c "import tensorrt_llm" in the terminal.

Expected behavior

The program terminates.

actual behavior

[TensorRT-LLM][INFO] Initializing MPI with thread mode 3

The above gets printed and the terminal gets stuck on that.

additional notes

I wonder if I made a mistake installing mpi4py via conda instead of pip? Originally, when I installed tensorrt-llm I ran into an error installing the MPI package.

mrakgr commented 3 hours ago

When I try installing it via pip, I get the following humongous error:

(trt) ceti@ceti16:~$ pip install mpi4py
Collecting mpi4py
  Using cached mpi4py-4.0.1.tar.gz (466 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mpi4py
  Building wheel for mpi4py (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for mpi4py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [316 lines of output]
      running bdist_wheel
      running build
      running build_src
      using Cython 3.0.11
      cythonizing 'src/mpi4py/MPI.pyx' -> 'src/mpi4py/MPI.c'
      running build_py
      creating build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/__init__.py -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/bench.py -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/typing.py -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/run.py -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/__main__.py -> build/lib.linux-x86_64-cpython-310/mpi4py
      creating build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/__init__.py -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/server.py -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/util.py -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/_base.py -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/pool.py -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/__main__.py -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/aplus.py -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/_core.py -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      creating build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/util/__init__.py -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/util/sync.py -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/util/pkl5.py -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/util/pool.py -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/util/dtlib.py -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/bench.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/run.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/__init__.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/MPI.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/__main__.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/typing.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/py.typed -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/__init__.pxd -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/MPI.pxd -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/libmpi.pxd -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/MPI.h -> build/lib.linux-x86_64-cpython-310/mpi4py
      copying src/mpi4py/MPI_api.h -> build/lib.linux-x86_64-cpython-310/mpi4py
      creating build/lib.linux-x86_64-cpython-310/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/pycapi.h -> build/lib.linux-x86_64-cpython-310/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-cpython-310/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-cpython-310/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-cpython-310/mpi4py/include/mpi4py
      copying src/mpi4py/util/sync.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/util/pkl5.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/util/pool.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/util/__init__.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/util/dtlib.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/util
      copying src/mpi4py/futures/_core.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/_base.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/pool.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/aplus.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/__init__.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/server.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/__main__.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      copying src/mpi4py/futures/util.pyi -> build/lib.linux-x86_64-cpython-310/mpi4py/futures
      running build_ext
      MPI configuration: [mpi] from 'mpi.cfg'
      MPI C compiler:    /usr/bin/mpicc
      MPI C++ compiler:  /usr/bin/mpicxx
      checking for MPI compile and link ...
      /usr/bin/mpicc -pthread -B /home/ceti/miniconda3/envs/trt/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/ceti/miniconda3/envs/trt/include -fPIC -O2 -isystem /home/ceti/miniconda3/envs/trt/include -fPIC -I/home/ceti/miniconda3/envs/trt/include/python3.10 -c _configtest.c -o _configtest.o
      success!
      removing: _configtest.c _configtest.o
      /usr/bin/mpicc -pthread -B /home/ceti/miniconda3/envs/trt/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/ceti/miniconda3/envs/trt/include -fPIC -O2 -isystem /home/ceti/miniconda3/envs/trt/include -fPIC -I/home/ceti/miniconda3/envs/trt/include/python3.10 -c _configtest.c -o _configtest.o
      /usr/bin/mpicc -pthread -B /home/ceti/miniconda3/envs/trt/compiler_compat _configtest.o -o _configtest
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: warning: libopen-rte.so.40, needed by /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so, not found (try using -rpath or -rpath-link)
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: warning: libopen-pal.so.40, needed by /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so, not found (try using -rpath or -rpath-link)
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: warning: libm.so.6, needed by /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so, not found (try using -rpath or -rpath-link)
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: warning: libhwloc.so.15, needed by /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so, not found (try using -rpath or -rpath-link)
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_framework_components_close'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_list_sort'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_get_element_count'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convertor_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_namelist_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_dump'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_install_dirs'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_set_value_ptr'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_rand'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `hwloc_bitmap_alloc'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `hwloc_get_obj_by_depth'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_delay_abort'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convertor_raw'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `hwloc_bitmap_free'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_get'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_local_arch'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_list_item_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_backtrace_print'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pmix_collect_all_data'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_component_close'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_get_value_ptr'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_data_server_uri'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_str_to_bool'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_register'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_register_framework_params'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_is_monotonic'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `hwloc_bitmap_iszero'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hwloc_base_cset2mapstr'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_proc_local_set'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_unsetenv'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `hwloc_bitmap_isincluded'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_dump_data_flags'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hwloc_topology'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convertor_pack'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_output_stream_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_out'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pmix_base_exchange'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_get_valuelen'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_timer_base_get_cycles'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_allocator_base_framework'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_argv_join'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pmix_base_async_modex'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_set_element_count'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_reset'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_argv_free'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_init'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_process_info'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_path_nfs'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_get'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convertor_clone'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_backtrace_buffer'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hwloc_compute_relative_locality'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_btl_base_framework'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pmix_app_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_progress_set_event_poll_rate'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pointer_array_test_and_set_item'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_thread_get_self'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pointer_array_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_get_nthkey'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_copy_content_same_ddt'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pmix_verbose_output'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `log@GLIBC_2.29'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_components_close'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pointer_array_init'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_infosubscribe_subscribe'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_info_show_orte_version'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_init'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_select'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_argv_append_unique_nosize'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_read_value'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_timer_base_get_freq'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_finalize'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_session_dir_finalize'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_process_name_print'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_alloc'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `hwloc_get_type_depth'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_buffer_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convert_process_name_to_string'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_free_list_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_output'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `sqrt@GLIBC_2.2.5'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_find_by_name'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_get'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_progress'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hwloc_base_single_cpu'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_compare_proc'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `ompi_sync_wait_mt'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_free_list_item_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_contain_basic_datatypes'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_framework_is_open'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_cr_output'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convertor_prepare_for_recv'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `hwloc_bitmap_compare'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_dss'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_value_unload'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_get_value_enum'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_built_with_cuda_support'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_set_value'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_progress_register'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_register_project_frameworks'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_get_bool'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_get_next_key_uint32'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pointer_array_add'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_util_compare_name_fields'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_argv_count'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_finalize_util'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_output_close'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_progress_event_users_increment'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pointer_array_set_item'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_in_parallel_debugger'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convertor_prepare_for_send'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_remove_value_uint32'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_mpool_base_framework'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_ess'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_value_load'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_enum_create_flag'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_bitmap_set_bit'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_compute_ptypes'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_close_components'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_output_verbose'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_dup'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_process_info'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_argv_append_nosize'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `hwloc_get_cpubind'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_class_init_epoch'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_register'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pmix_pdata_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_cr_reg_coord_callback'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_free'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_info_close_components'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_proc_applied_binding'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_proc_local_get'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_component_var_register'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_proc_is_bound'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_name_wildcard'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_get_value'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_rcache_base_framework'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_find'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_output_set_verbosity'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_write_value'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hwloc_base_cset2str'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convertor_compute_remote_size'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_dup_mpistandard'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_crs_base_state_str'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_mpool_base_alloc'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_init_util'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convertor_create'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_output_open'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_thread_self_compare'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_free_list_grow_st'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_condition_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_delete'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_show_help'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_predefined_elem_desc'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_cuda_support'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_clone'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_odls'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_mpool_base_free'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_progress_set_event_flag'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convertor_unpack'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_register_synonym'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_get_value_uint32'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_group_find_by_name'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_argv_split'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_strerror'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_group_get_stamp'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_session_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_pmix'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_srand'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_initialized'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_arch_set_fortran_logical_size'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_object_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_util_convert_process_name_to_string'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_get_count'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_mpool_base_tree_print'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_class_initialize'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_progress_set_yield_when_idle'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_bitmap_is_set_bit'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_get_count'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_find_by_name'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_proc_for_name'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_framework_close'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_bitmap_init'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_framework_components_open'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_standalone_operation'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_uses_threads'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_util_convert_string_to_process_name'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_proc_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_set'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_argv_append'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_mutex_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_component_to_string'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_util_print_name_args'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_allocator_component_lookup'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_stop'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_init'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_set_value_uint32'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_abort_print_stack'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_get_value_uint64'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_group_get'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_convert_string_to_process_name'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_commit'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_bitmap_find_and_set_first_unset_bit'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_show_help'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_getcwd'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_errmgr'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_dump_data_desc'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_enum_create'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_component_repository_release'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_value_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_free_list_init'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_bitmap_set_max_size'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_make_version_str'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_progress_unregister'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_pvar_handle_start'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_infosubscriber_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_create_desc'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_setenv'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_progress_event_users_decrement'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_remove_value_ptr'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_finalize'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_bitmap_clear_bit'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_bitmap_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_set_value_enum'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_get_first_key_uint32'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_component_list_item_t_class'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_infosubscribe_change_info'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_info_register_framework_params'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_framework_open'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `orte_session_dir_cleanup'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_info_show_opal_version'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_add'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_class_finalize'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `mca_base_var_group_get_count'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_datatype_resize'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_hash_table_set_value_uint64'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `MPIR_being_debugged'
      /home/ceti/miniconda3/envs/trt/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to `opal_list_t_class'
      collect2: error: ld returned 1 exit status
      failure.
      removing: _configtest.c _configtest.o
      error: Cannot link MPI programs. Check your configuration!!!
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mpi4py
Failed to build mpi4py
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (mpi4py)
mrakgr commented 3 hours ago

It seems like getting it from conda-forge works. Previously, when I did conda install mpi4py it got an older version instead.

(trt) ceti@ceti16:~$ conda install -c conda-forge mpi4py
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/ceti/miniconda3/envs/trt

  added / updated specs:
    - mpi4py

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    libevent-2.1.12            |       hf998b51_1         417 KB  conda-forge
    libhwloc-2.11.2            |default_he43201b_1000         2.3 MB  conda-forge
    libiconv-1.17              |       hd590300_2         689 KB  conda-forge
    libnl-3.10.0               |       h4bc722e_0         717 KB  conda-forge
    libsqlite-3.45.2           |       h2797004_0         837 KB  conda-forge
    libxml2-2.13.4             |       h064dc61_2         673 KB  conda-forge
    mpi-1.0                    |          openmpi           4 KB  conda-forge
    mpi4py-4.0.1               |  py310h58152c7_0         823 KB  conda-forge
    openmpi-5.0.5              |     h6ae21d5_103        13.3 MB  conda-forge
    python-3.10.13             |hd12c33a_1_cpython        24.5 MB  conda-forge
    python_abi-3.10            |          5_cp310           6 KB  conda-forge
    sqlite-3.45.2              |       h2c6b66d_0         829 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        45.0 MB

The following NEW packages will be INSTALLED:

  libevent           conda-forge/linux-64::libevent-2.1.12-hf998b51_1 
  libgcc             conda-forge/linux-64::libgcc-14.2.0-h77fa898_1 
  libgfortran        conda-forge/linux-64::libgfortran-14.2.0-h69a702a_1 
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-14.2.0-h69a702a_1 
  libgfortran5       conda-forge/linux-64::libgfortran5-14.2.0-hd5240d6_1 
  libhwloc           conda-forge/linux-64::libhwloc-2.11.2-default_he43201b_1000 
  libiconv           conda-forge/linux-64::libiconv-1.17-hd590300_2 
  libnl              conda-forge/linux-64::libnl-3.10.0-h4bc722e_0 
  libnsl             conda-forge/linux-64::libnsl-2.0.1-hd590300_0 
  libsqlite          conda-forge/linux-64::libsqlite-3.45.2-h2797004_0 
  libstdcxx          conda-forge/linux-64::libstdcxx-14.2.0-hc0a3c3a_1 
  libxcrypt          conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1 
  libxml2            conda-forge/linux-64::libxml2-2.13.4-h064dc61_2 
  libzlib            conda-forge/linux-64::libzlib-1.3.1-hb9d3cd8_2 
  mpi                conda-forge/linux-64::mpi-1.0-openmpi 
  mpi4py             conda-forge/linux-64::mpi4py-4.0.1-py310h58152c7_0 
  openmpi            conda-forge/linux-64::openmpi-5.0.5-h6ae21d5_103 
  python_abi         conda-forge/linux-64::python_abi-3.10-5_cp310 

The following packages will be UPDATED:

  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-14.2.0-h69a702a_1 
  libgomp              pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-14.2.0-h77fa898_1 
  libstdcxx-ng       pkgs/main::libstdcxx-ng-11.2.0-h12345~ --> conda-forge::libstdcxx-ng-14.2.0-h4852527_1 
  libuuid              pkgs/main::libuuid-1.41.5-h5eee18b_0 --> conda-forge::libuuid-2.38.1-h0b41bf4_0 
  openssl              pkgs/main::openssl-3.0.15-h5eee18b_0 --> conda-forge::openssl-3.3.2-hb9d3cd8_0 
  zlib                    pkgs/main::zlib-1.2.13-h5eee18b_1 --> conda-forge::zlib-1.3.1-hb9d3cd8_2 

The following packages will be SUPERSEDED by a higher-priority channel:

  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu 
  python               pkgs/main::python-3.10.15-he870216_1 --> conda-forge::python-3.10.13-hd12c33a_1_cpython 
  sqlite                pkgs/main::sqlite-3.45.3-h5eee18b_0 --> conda-forge::sqlite-3.45.2-h2c6b66d_0 
  tk                        pkgs/main::tk-8.6.14-h39e8969_0 --> conda-forge::tk-8.6.13-noxft_h4845f30_101 

Proceed ([y]/n)? 

Downloading and Extracting Packages:

Preparing transaction: done                                                                                                                                              
Verifying transaction: done                                                                                                                                              
Executing transaction: \                                                                                                                                                 
On Linux, Open MPI is built with UCX support but it is disabled by default.                                                                                              
To enable it, first install UCX (conda install -c conda-forge ucx).                                                                                                      
Afterwards, set the environment variables                                                                                                                                
OMPI_MCA_pml=ucx OMPI_MCA_osc=ucx                                                                                                                                        
before launching your MPI processes.                                                                                                                                     
Equivalently, you can set the MCA parameters in the command line:                                                                                                        
mpiexec --mca pml ucx --mca osc ucx ...                                                                                                                                  

On Linux, Open MPI is built with UCC support but it is disabled by default.
To enable it, first install UCC (conda install -c conda-forge ucc).
Afterwards, set the environment variables
OMPI_MCA_coll_ucc_enable=1
before launching your MPI processes.
Equivalently, you can set the MCA parameters in the command line:
mpiexec --mca coll_ucc_enable 1 ...

On Linux, Open MPI is built with CUDA awareness but it is disabled by default.
To enable it, please set the environment variable
OMPI_MCA_opal_cuda_support=true
before launching your MPI processes.
Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via
UCX. Please consult UCX documentation for further details.

done
mrakgr commented 3 hours ago

Now I am getting 4.0 instead of 3.1. Should I be setting the Cuda awareness variables and also installing the ucc and ucx packages?

mrakgr commented 3 hours ago

Now I get the following error:

(trt) ceti@ceti16:~$ python3 -c "import tensorrt_llm"
[ceti16:1300619] shmem: mmap: an error occurred while determining whether or not /tmp/ompi.ceti16.1000/jf.0/3453681664/shared_mem_cuda_pool.ceti16 could be created.
[ceti16:1300619] create_and_attach: unable to create shared memory BTL coordinating structure :: size 134217728 
[TensorRT-LLM] TensorRT-LLM version: 0.15.0.dev2024102900

Maybe I really should be doing this from a container?

mrakgr commented 3 hours ago

Even after installing ucc, ucx and doing conda install cuda-cudart cuda-version=12 I get the above shmem error. I'll try all of this out from the container instead.

mrakgr commented 3 hours ago 
root@858b7ca53f5c:/# python3 -c "import tensorrt_llm"
[TensorRT-LLM] TensorRT-LLM version: 0.15.0.dev2024102900
root@858b7ca53f5c:/#

From the container it works flawlessly. You guys decide whether you want to bother looking into this further or close it here. For me this is a good opportunity to get familiar with VS Code remote container dev capabilities.