search

triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
BSD 3-Clause "New" or "Revised" License
8.24k stars 1.47k forks source link

The Jetson Nano(JetPack4.6.1) cannot use GPU in the Triton inference container #6702

Open blackhu opened 10 months ago

blackhu commented 10 months ago

Description A clear and concise description of what the bug is.

The Jetson Nano(JetPack4.6.1) cannot use GPU in the Triton inference container

Triton Information What version of Triton are you using?

  Docker container:
  - nvcr.io/nvidia/tritonserver:23.11-py3-igpu-sdk
  - nvcr.io/nvidia/tritonserver:23.11-py3-sdk

Are you using the Triton container or did you build it yourself?

Triton container

To Reproduce Steps to reproduce the behavior.

Shell Command:
- docker run -it --rm --gpus all --net=host nvcr.io/nvidia/tritonserver:23.11-py3-igpu-sdk
- docker run -it --rm --gpus all --net=host nvcr.io/nvidia/tritonserver:23.11-py3-sdk

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

 Jetson Nano System Image:  [jetson-nano-jp461-sd-card-image.zip](https://developer.nvidia.com/embedded/learn/get-started-jetson-nano-devkit#write)

Expected behavior A clear and concise description of what you expected to happen.

  =================================
  == Triton Inference Server SDK ==
  =================================

  NVIDIA Release  (build 74929371)

  Copyright (c) 2018-2023, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

  Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

  This container image and its contents are governed by the NVIDIA Deep Learning Container License.
  By pulling and using the container, you accept the terms and conditions of this license:
  https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

  WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
     Use the NVIDIA Container Toolkit to start this container with GPU support; see
     https://docs.nvidia.com/datacenter/cloud-native/ .

  root@jetson_nano:/workspace#
oandreeva-nv commented 10 months ago

CC'ing @kthui and @nv-kmcgill53 , by any chance, have you seen this issue before?

oandreeva-nv commented 10 months ago

Internal ticket: [5946]

kthui commented 10 months ago

Hi @blackhu,

docker run -it --rm --gpus all --net=host nvcr.io/nvidia/tritonserver:23.11-py3-igpu-sdk docker run -it --rm --gpus all --net=host nvcr.io/nvidia/tritonserver:23.11-py3-sdk

I am wondering if you meant to download nvcr.io/nvidia/tritonserver:23.11-py3-igpu container when you downloaded nvcr.io/nvidia/tritonserver:23.11-py3-sdk?

The containers published for 23.11 for iGPU are:

on NGC https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags

liujiachang commented 6 months ago

I also encountered this issue, but it can run normally on Jetson 5.1.2.