[QUESTION] Debugging erroneous pipeline: no element "kvssink"

[crmaykish]

I have cross-compiled the Kinesis producer SDK for armhf. I have installed the built libraries, but I'm still receiving the no element "kvssink" error.

$ gst-launch-1.0 v4l2src do-timestamp=TRUE device=/dev/video0 ! image/jpeg ! jpegdec ! x264enc  bframes=0 key-int-max=45 bitrate=500 ! video/x-h264,stream-format=avc,alignment=au,profile=baseline ! kvssink stream-name="xxx" storage-size=512 access-key="xxx" secret-key="xxx" aws-region="xxx"

WARNING: erroneous pipeline: no element "kvssink"

$ ls /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libcproducer.so 
/usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libcproducer.so

$ ls /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libKinesisVideoProducer.so
/usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libKinesisVideoProducer.so

$ ls /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstkvssink.so 
/usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstkvssink.so

$ echo $GST_PLUGIN_PATH 
/usr/lib/arm-linux-gnueabihf/gstreamer-1.0:

echo $LD_LIBRARY_PATH 
/usr/lib/arm-linux-gnueabihf/gstreamer-1.0:

Target device is an armhf device running Debian Jessie.

BUILD_GSTREAMER_PLUGIN=ON BUILD_DEPENDENCIES=OFF

I've installed the dependencies manually from the Debian repository.

Not sure what I'm missing here. There are no other errors. CMake builds without error and the gstreamer sample applications run successfully on the target. Thanks.

[hassanctech]

Can you please report the output of gst-inspect-1.0 libgstkvssink.so

[crmaykish]

$ gst-inspect-1.0 libgstkvssink.so
Plugin Details:
  Name                     libgstkvssink.so
  Description              Plugin for blacklisted file
  Filename                 /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstkvssink.so
  Version                  0.0.0
  License                  BLACKLIST
  Source module            BLACKLIST
  Binary package           BLACKLIST
  Origin URL               BLACKLIST

  0 features:

$ gst-inspect-1.0 /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstkvssink.so
Could not load plugin file: File "/usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstkvssink.so" appears to be a GStreamer plugin, but it failed to initialize

[hassanctech]

Ah ok looks like it's blacklisted so to do that you need to remove it from the blacklist in your registry, you just need to clear your registry cache. Just to make sure GStreamer is actually set up properly on the machine, does running gst-inspect-1.0 output what you expect or does it say everything is blacklisted?

Here's an example for clearing the cache: https://forums.developer.nvidia.com/t/deepstream-plugins-blacklisted/77858

[crmaykish]

Thanks for the followup. I've cleared my cache and set GST_DEBUG=5 to see what's going on.

When I run gst-inspect-1.0 libgstkvssink.so, I'm getting this in the debug log:

GST_PLUGIN_LOADING gstplugin.c:484:gst_plugin_register_func: plugin "/usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstkvssink.so" has incompatible version, not loading

My understanding from other threads on this issue is that there's a GStreamer version difference between my build environment and the environment of the running system? How can I identify that difference?

[hassanctech]

Oh I see then do you have multple gst-inspect-1.0, and gst-launch-1.0 commands as well, perhaps you're invoking the wrong one? Since you've set GST_PLUGIN_PATH and LD_LIBRARY_PATH we should be OK but the loader will still look in default places if it doesn't find something there /usr/lib or /usr/local/lib. Can you try which gst-inspect-1.0 and which gst-launch-1.0. Then also run gst-inspect-1.0 --version to find the version of the one that is being invoked. Compare to the version of GStreamer that you used to build the kvssink plugin. Finally if you are using the wrong gst-inspect-1.0 you will need to say the install path of the GStreamer that you built in your build environment and see where it put it and make sure to invoke the correct one. Let me know if anything is unclear.

[crmaykish]

There only appears to be one version of GStreamer installed and they are the same between the target and the sysroot used to cross-compile. They are quite old though:

$ gst-inspect-1.0 --version
gst-inspect-1.0 version 1.4.4
GStreamer 1.4.4

It would be great to use the OS-supplied version of GStreamer, but if it's too far out of date, I can try building a new version from source.

[hassanctech]

Yeah but I'm afraid that version might be too old, the oldest one I've successfully tested on is 1.8.3. In the process of building kvssink then you didn't build gstreamer again from scratch you just used the system provided one, is that correct? You can check for that symbol by running something like nm -d $file_where_it_is_supposed_to_be | grep gst_plugin_register_func I'm not sure where it's supposed to be but I imagine in one of the standard gstreamer libs then we can see if it was added after 1.4.4 -- not sure if that's the root issue though. If you can try with a newer version from source that would be nice. I'm actually surprised kvssink built for 1.4.4... if it built then I suppose it should work.

[hassanctech]

Oops looks like I misread that error message it's not complaining about that being missing! It looks like it built kvssink actually for a different version`

[crmaykish]

Is there a way to tell which version libgstkvssink.so was built for? The GStreamer version in the sysroot is the same as the one on the target device. I don't know what other version the build system would even have access to. Maybe some host header files are sneaking into the cross-compile :thinking:

[hassanctech]

Yeah let me see I'm not aware of an easy way to do that, but that logging in gstplugin.c was updated as of 1.5.1 to print the version for the GStreamer install vs the plugin but unfortunately your version is older than that :(

https://github.com/GStreamer/gstreamer/commit/423bd42d44fdf0d8969d38d963aab15f1bb7f016

If you can try upgrading to a newer GStreamer that would probably be the best route, I'm not confident kvssink will even build for 1.4.4.

[hassanctech]

Also make sure to sort out to make sure your build is completely sandboxed from the host headers / libraries so you don't get weird run time behavior!

[crmaykish]

Thanks, the build system should be pretty well isolated, but I'm guessing my GStreamer libs are just way too old. I'll see what I can do to get those updated and try again.

I appreciate all the help!

[hassanctech]

Yeah sure thing! I hope we can get you unblocked here!

[crmaykish]

Making some progress! I built GStreamer 1.19 from source for my target device. Basic functions are working well.

$ gst-inspect-1.0 --version
gst-inspect-1.0 version 1.19.0
GStreamer 1.19.0 (GIT)
Unknown package origin

I've also rebuilt kvssink targeting the new version of GStreamer:

$ gst-inspect-1.0 kvssink
Factory Details:
  Rank                     primary + 10 (266)
  Long-name                KVS Sink
  Klass                    Sink/Video/Network
  Description              GStreamer AWS KVS plugin
  Author                   AWS KVS <kinesis-video-support@amazon.com>

Plugin Details:
Factory Details:
  Rank                     primary + 10 (266)
  Long-name                KVS Sink
  Klass                    Sink/Video/Network
  Description              GStreamer AWS KVS plugin
  Author                   AWS KVS <kinesis-video-support@amazon.com>

Plugin Details:
  Name                     kvssink
  Description              GStreamer AWS KVS plugin
  Filename                 /usr/local/lib/gstreamer-1.0/libgstkvssink.so
  Version                  1.0
  License                  Proprietary
  Source module            kvssinkpackage
  Binary package           GStreamer
  Origin URL               http://gstreamer.net/

Attempting to run the following command:

$ gst-launch-1.0 v4l2src do-timestamp=TRUE device=/dev/video0 ! h264parse ! video/x-h264,stream-format=avc,alignment=au ! kvssink stream-name="___" storage-size=512 access-key="___" secret-key="___" aws-region="___"

and getting the following error:

Setting pipeline to PAUSED ...
ERROR: from element /GstPipeline:pipeline0/GstKvsSink:kvssink0: Could not initialize supporting library.
Additional debug info:
___/src/gstreamer/gstkvssink.cpp(1403): gst_kvs_sink_change_state (): /GstPipeline:pipeline0/GstKvsSink:kvssink0:
Failed to init kvs producer. Error:  Unable to create Kinesis Video client. Error status: 0x4
ERROR: pipeline doesn't want to preroll.
Failed to set pipeline to PAUSED.
 Unable to create Kinesis Video client. Error status: 0x4Setting pipeline to NULL ...
Freeing pipeline ...

So we're at least up and running, GStreamer is recognizing kvssink as a plugin and attempting to use it. It's just hitting a run time error.

[MushMal]

Perhaps you are hitting a out of memory error defined here: https://github.com/awslabs/amazon-kinesis-video-streams-pic/blob/master/src/common/include/com/amazonaws/kinesis/video/common/CommonDefs.h#L467

You should enable the verbose logging by specifying your own log4cplus config: https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp/blob/master/src/gstreamer/gstkvssink.cpp#L541

The default storage size is 128MB while you are setting it to 512

[hassanctech]

@crmaykish That's great progress! I would definitely try out @MushMal's recommendations it's very likely your embedded device is running out of memory if you're setting it to 512MB. I would start out with something smaller like 32MB and adjust based on the memory actually available on your system.

Verbose logging will be very helpful in this case as well.

Lastly just want to make sure you're aware GStreamer 1.{odd-number} releases are unstable dev versions and 1.{even-number} are stable releases. If you continue to hit issues I might recommend staying on a stable release instead, like 1.18 especially in production.

[crmaykish]

That was it, just running out of memory. I set storage-size=32 and got a stream running!

Thanks for the tip on the GStreamer versions. I'll rebuild on a stable version.

Really appreciate all the help, thanks!

[hassanctech]

Yayyy! Glad to hear that! Closing the issue since the original problem has been resolved. If you run into more issues feel free to open a new issue detailing the problem!