search

ros-misc-utilities / ffmpeg_image_transport

ROS2 image transport plugin for encoding/decoding with h264 codec
Apache License 2.0
65 stars 22 forks source link

Error decoding ffmpeg packets. #3

Closed akssri-sony closed 1 year ago

akssri-sony commented 1 year ago

I run into the following ffmpeg decoder error on the sub-side under this setup,

....
[WARN] [1682494335.377477553] [FFMPEGDecoder]: send_packet failed for pts: 202
[h264 @ 0x55b21f7f68c0] Invalid setup for format cuda: missing configuration.
Failed to get HW surface format.
[h264 @ 0x55b21f7f68c0] decode_slice_header error
[h264 @ 0x55b21f7f68c0] no frame!
[WARN] [1682494335.444379691] [FFMPEGDecoder]: send_packet failed for pts: 203
[h264 @ 0x55b21f7f68c0] Invalid setup for format cuda: missing configuration.
Failed to get HW surface format.
[h264 @ 0x55b21f7f68c0] decode_slice_header error
[h264 @ 0x55b21f7f68c0] no frame!
[WARN] [1682494335.510952361] [FFMPEGDecoder]: send_packet failed for pts: 204
^C[INFO] [1682494335.541615526] [rclcpp]: signal_handler(signum=2)
...

Would be grateful if you have some insight into what's going wrong or how this could solved. Thanks!

berndpfrommer commented 1 year ago

Could you please post the output (first few lines) of 'ros2 echo /your_image_topic'? I'd like to see what encoding is being used by the encoder?

The problem is apparently that the decoder tries to use some cuda decoder. Not sure why. Could you please also post the line with which you launch the republisher (decoder)?

The command for starting the republisher in the README almost certainly will have to be modified to avoid using the accelerated decoder. Maybe something like this:

ros2 run image_transport republish ffmpeg in/ffmpeg:=image_raw/ffmpeg raw out:=image_raw/uncompressed --ros-args -p "ffmpeg_image_transport.map.hevc_nvenc:=libx264"

akssri-sony commented 1 year ago

This is what I get when I echo the topic with ros2 topic echo /camera/color/image_rect_raw/ffmpeg

header:
  stamp:
    sec: 1682568513
    nanosec: 971701760
  frame_id: camera_color_optical_frame
width: 848
height: 480
encoding: libx264
pts: 1394
flags: 0
is_bigendian: false
data:
- 0
- 0
- 0
- 1
- 65
- 155
- 211
- 120
- 67
- 252
- 153
- 76
- 0
- 130
- 31
- 130
- 15
- 106
- 70
- 132
- 92
- 54
- 11
- 5
- 182
- 224
- 235
- 183
- 147
- 56
- 48
- 149
- 103
-

The republisher was initially run with ros2 run image_transport republish ffmpeg in/ffmpeg:=/camera/color/image_rect_raw/ffmpeg raw out:=/camera/color/image_rect_raw/uncompressed.

I tried setting all the mappings with

ros2 run image_transport republish ffmpeg in/ffmpeg:=/camera/color/image_rect_raw/ffmpeg raw out:=/camera/color/image_rect_raw/uncompressed --ros-args -p "ffmpeg_image_transport.map.h264_nvmpi:=libx264"  --ros-args -p "ffmpeg_image_transport.map.h264_nvenc:=libx264"  --ros-args -p "ffmpeg_image_transport.map.hevc_nvenc:=libx264" --ros-args -p "ffmpeg_image_transport.map.libx264:=libx264"

Only the last one had any effect (since the stream is encoded with libx264). Now I get,

root@garuda:/home/ros/humble# ros2 run image_transport republish ffmpeg in/ffmpeg:=/camera/color/image_rect_raw/ffmpeg raw out:=/camera/color/image_rect_raw/uncompressed --ros-args -p "ffmpeg_image_transport.map.h264_nvmpi:=libx264"  --ros-args -p "ffmpeg_image_transport.map.h264_nvenc:=libx264"  --ros-args -p "ffmpeg_image_transport.map.hevc_nvenc:=libx264" --ros-args -p "ffmpeg_image_transport.map.libx264:=libx264"
[WARN] [1682571049.574895085] [rcl]: Found remap rule 'in/ffmpeg:=/camera/color/image_rect_raw/ffmpeg'. This syntax is deprecated. Use '--ros-args --remap in/ffmpeg:=/camera/color/image_rect_raw/ffmpeg' instead.
[WARN] [1682571049.574909802] [rcl]: Found remap rule 'out:=/camera/color/image_rect_raw/uncompressed'. This syntax is deprecated. Use '--ros-args --remap out:=/camera/color/image_rect_raw/uncompressed' instead.
[ERROR] [1682571049.637535289] [image_republisher]: SubscriberPlugin::subscribeImpl with five arguments has not been overridden
[ERROR] [1682571050.100268191] [FFMPEGDecoder]: cannot open decoder libx264
[ERROR] [1682571050.100311730] [FFMPEGDecoder]: cannot open decoder libx264
[ERROR] [1682571050.100349746] [FFMPEGSubscriber]: cannot initialize decoder!
[ERROR] [1682571051.100484505] [FFMPEGDecoder]: cannot open decoder libx264
[ERROR] [1682571051.100518457] [FFMPEGDecoder]: cannot open decoder libx264
[ERROR] [1682571051.100523935] [FFMPEGSubscriber]: cannot initialize decoder!

Some digging on the internet led me to something that says that ffmpeg needs to be recompiled with spec. flags; not sure if this is the issue.

akssri-sony commented 1 year ago 
root@garuda:/home/ros/humble# ffmpeg -codecs | grep -i 264
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
 DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_v4l2m2m h264_qsv h264_cuvid ) (encoders: libx264 libx264rgb h264_nvenc h264_omx h264_qsv h264_v4l2m2m h264_vaapi nvenc nvenc_h264 )

All the decoders for h264 have some CUDA error associated with them.

akssri-sony commented 1 year ago

Looks like get_hw_type in ffmpeg_decoder.cpp always returns the AV_HWDEVICE_TYPE_CUDA (=2), regardless of whether there's a cuda device or not. Hacking this to take the non-Cuda code path leads to other errors: the decoded image is all messed up. Thanks for the help!

Input: image

Output: image

berndpfrommer commented 1 year ago

What happens if you run the republisher like this:

ros2 run image_transport republish ffmpeg in/ffmpeg:=image_raw/ffmpeg raw out:=image_raw/uncompressed --ros-args -p "ffmpeg_image_transport.map.libx264:=h264" If you use straight ffmpeg (the executable) from the command line, can you encode/decode with libx264? If that works, then the ROS2 transport plugins should also work.

berndpfrommer commented 1 year ago

Oh, and your image resolution is really odd. The line size of 848 is not even a multiple of 32. Other users have reported weirdly decoded images in such situations. I'm surprised you didn't get a warning on the encoder side:

if ((width % 32) != 0) {
      RCLCPP_WARN(logger_, "horiz res must be multiple of 32!");
    }
akssri-sony commented 1 year ago

If I run the republisher with,

ros2 run image_transport republish ffmpeg in/ffmpeg:=image_raw/ffmpeg raw out:=image_raw/uncompressed --ros-args -p "ffmpeg_image_transport.map.libx264:=h264"

I end up with the same error as in the first comment.

On the resolution side, I did get the warning, but didn't realize this would be the effect. Is there an easy fix for this. I guess padding would work, but is there a better way on the libav side ?

berndpfrommer commented 1 year ago

I really don't know how to fix this on the libav side. But does standalone ffmpeg work correctly for decoding h264 packets?

On Thu, Apr 27, 2023, 9:55 PM akssri-sony @.***> wrote:

If I run the republisher with,

ros2 run image_transport republish ffmpeg in/ffmpeg:=image_raw/ffmpeg raw out:=image_raw/uncompressed --ros-args -p "ffmpeg_image_transport.map.libx264:=h264"

I end up with the same error as in the first comment.

On the resolution side, I did get the warning, but didn't realize this would be the effect. Is there an easy fix for this. I guess padding would work, but is there a better way on the libav side ?

— Reply to this email directly, view it on GitHub https://github.com/berndpfrommer/ffmpeg_image_transport/issues/3#issuecomment-1526868969, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPLK2S7XONDDSV7K3CRRUTXDMPQPANCNFSM6AAAAAAXMBDPHM . You are receiving this because you commented.Message ID: @.***>

berndpfrommer commented 1 year ago

Fixed by PR #4