search

dlstreamer / pipeline-server

Home of Intel(R) Deep Learning Streamer Pipeline Server (formerly Video Analytics Serving)
BSD 3-Clause "New" or "Revised" License
123 stars 50 forks source link

Pipeline failure in some multi-GPU systems #98

Closed whbruce closed 1 year ago

whbruce commented 2 years ago

Pipelines can sometimes fail in multi GPU systems due to video decode incompatibility. Pipeline will return an error when starting and service log will contain an entry like this

{"levelname": "INFO", "asctime": "2021-11-30 22:48:40,620", "message": "Creating Instance of Pipeline object_detection/person_vehicle_bike", "module": "pipeline_manager"}
{"levelname": "ERROR", "asctime": "2021-11-30 22:48:47,369", "message": "Error on Pipeline 1: gst-stream-error-quark: No valid frames decoded before end of stream (7): ../gst-libs/gst/video/gstvideodecoder.c(1246): gst_video_decoder_sink_event_default (): /GstPipeline:pipeline11/GstURIDecodeBin:source/GstDecodeBin:decodebin2/GstVaapiDecodeBin:vaapidecodebin2/GstVaapiDecode:vaapidecode2:\nno valid frames found", "module": "gstreamer_pipeline"}

Workaround is to disable GPU decode by clearing environment variable LIBVA_DRIVER_NAME e.g. if using run script to start service:

./run.sh -v /tmp:/tmp -e LIBVA_DRIVER_NAME=
whbruce commented 1 year ago

Fixed in 2022.2.0 (v1.0). See https://github.com/dlstreamer/pipeline-server/blob/main/docs/running_pipeline_server.md#gpu.