search

voidmainvoid95 / gst-nvmaxine

Gstreamer plugin that allows use of NVIDIA Maxine SDK in a generic pipeline.
GNU Lesser General Public License v2.1
32 stars 1 forks source link

no element "nvmaxinevideofx" on Ubuntu 20 through WSL #1

Closed valadhi closed 1 year ago

valadhi commented 1 year ago

First of all, thank you for writing the plugin. It would be of great help if I can get it running.

I keep getting this error and I'm wondering if there is anything fundamentally wrong with my approach or if I'm just making a mistake somewhere. I am running on Ubuntu 20 through wsl. I managed to build the files successfully, but no matter how I add the plugin path to the build directory it's not getting picked up by either gst-launch or gst-inspect. I tried both adding to the GST_PLUGIN_PATH variable and as an argument to gst-launch with --gst-plugin-path. I also have maxine installed at /usr/local/VideoFX.

I have also managed to build the gst-template plugin and run it successfully with: gst-launch-1.0 -v -m --gst-plugin-path=/mnt/d/projects/stream/gst-template/build/gst-plugin/ fakesrc ! my_filter ! fakesink silent=TRUE

But when I try to run the same thing with the appropriate parameters for this plugin it returns no element "nvmaxinevideofx": gst-launch-1.0 -v -m --gst-plugin-path=/mnt/d/projects/stream/gst-nvmaxine/build/ fakesrc ! nvmaxinevideofx ! fakesink silent=TRUE

I am extremely new to gstreamer and C programming, any help would be appreciated.

voidmainvoid95 commented 1 year ago

You’re welcome! Actually I haven't a RTX card for fully testing the plugin under windows WSL. However I tried to reproduce your environment to get gst-inspect-1.0 nvmaxinevideofx working on a fresh Ubuntu WSL installation without exporting GST_PLUGIN_PATH. The setup I used, on branch master, is:

All other requirements are the same listed in Dockerfile. LD_LIBRARY_PATH needs to be exported. Can you give me more details on your setup?

valadhi commented 1 year ago

With your help I managed to make some progress and at least make the plugin available. On a fresh install and with versions close to what you provided I got inspect to find it. However, on trying the sample pipeline from the readme I am getting this error:

gst-launch-1.0 filesrc location="example_input.mp4" ! \ qtdemux ! avdec_h264 ! queue ! \ nvmaxinevideofx effect=SuperRes modeldir="/usr/local/VideoFX/lib/models" strength=1 upscalefactor=2 ! queue ! \ x264enc ! qtmux ! filesink location="example_output.mp4"

Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Redistribute latency... Redistribute latency... ERROR: from element /GstPipeline:pipeline0/GstQTDemux:qtdemux0: Internal data stream error. Additional debug info: qtdemux.c(6619): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstQTDemux:qtdemux0: streaming stopped, reason not-negotiated (-4) ERROR: pipeline doesn't want to preroll. Setting pipeline to NULL ... Freeing pipeline ...

I got it on a system with :

Do you have any suggestions on how to approach debugging the above error? As I understand it, I have followed all the required setup including the requirements from the dockerfile and the command should just work...

Also, what I am actually trying to do is implement the same thing for Windows. I suppose the modifications wouldn't need to be consistent, but I don't actually know what they are. What would you say would be needed to modify the code to run on Windows? I am also considering the fact that wsl could be the culprit here.

voidmainvoid95 commented 1 year ago

Great news! I managed to get all filters working on my Linux server with Tesla V100, and tested gst-inspect-1.0 nvmaxinevideofx on WSL. \ I have found two major problems reading maxine docs: 1) Cuda, cudnn and TensorRT versions listed here https://catalog.ngc.nvidia.com/orgs/nvidia/teams/maxine/collections/maxine are wrong. \ I have found the correct versions reading the docs shipped with the library:

2) Nvidia have changed the parameters of some filters.

Checkout master branch for the latest update and have a look to README first. I have also started a windows porting of the plugin, so stay tuned for the next release.

valadhi commented 1 year ago

Thank you for your effort. I did manage to get the Upscale filter working in a pipeline, but unfortunately all the others return an error similar to this:

gst-launch-1.0 filesrc location="example_input.mp4" ! \ qtdemux ! avdec_h264 ! queue ! \ nvmaxinevideofx effect=SuperRes modeldir="/usr/local/VideoFX/lib/models" mode=1 upscalefactor=2 ! queue ! \ x264enc ! qtmux ! filesink location="example_output.mp4"

Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Redistribute latency... ERROR: Loading TRTModel failed in Super Res ERROR: Loading TRTModel failed in Super Res Redistribute latency... ERROR: Loading TRTModel failed in Super Res ERROR: Loading TRTModel failed in Super Res ERROR: Loading TRTModel failed in Super Res ERROR: Loading TRTModel failed in Super Res ERROR: from element /GstPipeline:pipeline0/GstQTDemux:qtdemux0: Internal data stream error. Additional debug info: qtdemux.c(6619): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstQTDemux:qtdemux0: streaming stopped, reason not-negotiated (-4) ERROR: pipeline doesn't want to preroll. Setting pipeline to NULL ... Freeing pipeline ...

Do you have any clue as to what might be causing this? I have adapted the versions to pretty much what you have. cuda 11.6, cudnn 8.6.0, tensort 8.2.1.5 If you think this might be causing it I can try to match exactly.

Also, I am trying to understand what would be needed for a port of the plugin on Windows. Am I correct in assuming it is just a matter of adapting the makefiles or otherwise making the nvidia libraries available to the code? Can you more or less use the existing code as is?

voidmainvoid95 commented 1 year ago

Hi, windows port is now complete and it's available on master branch. You can find the dependencies in README file. I've tested only gst-inspect-1.0 nvmaxinevideofx as always, so I would be grateful if you fully test the plugin under Windows too.

If I remember correctly TRTModel errors came up when the SDK can't find the appropriate model for your video. I have used a 1280x720 video for testing and all pipelines from README work.\ The are some combination of resolution and filters that don't work: for example using an upscalefactor of 4/3 or 4 with the video I used. You can find more details on models limitations in Maxine SDK docs.

Can you confirm me that a 1280x720 video works on Ubuntu WSL and Windows?

valadhi commented 1 year ago

Great news! I can indeed confirm that it works on windows although I still can only use the Upscale model, but that might have to do with a faulty installation of maxine effects perhaps.

All the other models claim to not be able to find the .engine file such as "Cannot load model "SR_2x_agg_86.engine" even though I have file "SR_2x_agg_86.engine.trtpkg" at path: "C:\Program Files\NVIDIA Corporation\NVIDIA Video Effects\models"

I tested gst-launch on a 1280x720 video and it worked. Thank you for your help!