search

FDH2 / UxPlay

AirPlay Unix mirroring server
GNU General Public License v3.0
1.65k stars 83 forks source link

HEVC/h265 support for GStreamer on Raspbery Pi 4B/400/5 #358

Open fduncanh opened 3 weeks ago

fduncanh commented 3 weeks ago

Now that UxPlay 1.70 supports HEVC/h265/4K video, this issue thread is for discussion of how to get the stateless Video4Linux2 HEVC driver(s) on Raspberry Pi 4B/400/5 working with the stateless V4l2 plugin in gstreamer1.0-plugins-bad

LibreELEC/Kodi have HEVC on these Pi's working using a modified ffmpeg created by R Pi engineers: see https://github.com/jc-kynesim/rpi-ffmpeg/issues/94#issuecomment-2437314698

and https://discourse.gstreamer.org/t/v4l2-stateless-api-support/3339

connorh315 commented 3 weeks ago

Just as a minor bit of information. I've nearly gotten ffmpeg working with uxplay - Currently the frames are all green for some reason but I'll iron that out soon. Main thing is though that the decoder block on the RPi 4 is capable of decoding the AirPlay H265 packets and it seems to run at a relatively good frame rate (hard to tell as the stream looks shocking). The relevant changes just needs to be pushed into gstreamer now.

fduncanh commented 3 weeks ago

I have some RPI 5's and can dedicate one to testing too.

Ideally we would rebuild the RPi ffmpeg from its code so printf statements could be inserted to follow how it connects to the drivers and to which drivers).

Stateless v4l2 justs queues an encoded frame in the "OUTPUT" (from encoder) queue and dequeues if from the "CAPTURE" queue, so interaction with the driver which connects to the hardware is relatively simple. I just dont yet know how the interface to the driver works, and where it is in the ffmpeg code.

fduncanh commented 3 weeks ago

more info from jc-kinesim

The driver is in the PiOS linux tree - you can find it under driver/staging/media/rpivid/*. Hevc decode is a completely independent h/w block - GPU not involved. I'm sure the h/w is proprietary, I'm not completely sure to whom, so you are on your own for figuring out what is going on there (but honestly the h/w setup corresponds quite closely to the info passed in the V4L2 stateless structures so most of it is quite obvious).

fduncanh commented 3 weeks ago

current R Pi OS HEVC driver

https://github.com/raspberrypi/linux/tree/rpi-6.6.y/drivers/staging/media/rpivid

connorh315 commented 3 weeks ago

Ideally we would rebuild the RPi ffmpeg from its code so printf statements could be inserted to follow how it connects to the drivers and to which drivers).

RPi-FFmpeg is quite easy to build, unfortunately it just takes quite a while (~30 minutes on a RPi 4 with a decent SD card). I expect that subsequent rebuilds with minor changes would be faster but I haven't tried. Just something worth mentioning is that the hardware decoder outputs a DRM-frame which doesn't seem to be supported by many graphics libraries. I originally used SDL and tried to copy the hardware frame via the CPU into main memory to then be converted to YUV and eventually output back onto the display however it seemed to make those green frames that I was talking about earlier. Instead I'm using a OpenGL instance now, as I'm relatively sure that it can directly render DRM-Prime frames without any conversion/copying. I just want to get a working prototype at the moment and test that it can decode relatively well (it seemed relatively stable with an iPhone 15 Pro, it was decent with an iPad pro and i'll test my macbook later once I've actually got it stably outputting frames (can force the full 3840x2160 output with that) to stress test it.

connorh315 commented 3 weeks ago

Might have to give up with trying to get ffmpeg imported. Non-stop getting green-tinted frames from the decode and I can't seem to figure out what I'm doing wrong. Including ffmpeg into uxplay is relatively easy though (including adding hw-accelerated). Very frustrating

fduncanh commented 3 weeks ago

I am trying to find out why gstreamer is not detecting the rpivid driver on a R Pi 5.

The rpivid_hevc driver is loaded (see it in lsmod), the gstreamer libv4l2codec is correctly compiled and present (installed by gstreamer-1.0-plugins-bad)

"gstreamer-inspect-1.0 v4l2codec" shows the drivers, but lists "0 features" which means that no v4l2 devices was detected.

however, the v4l2 devices can be listed with "v4l2-ctl --list-devices" (v4l2-ctl is in the "v4l-utils" package, which is probably installed, if not, install with apt on R Pi OS)

This shows that the rpivid kernel model (on R Pi OS) is connected to /dev/video19 on my system. (and /dev/media0)

"v4l2-ctl -d19 -D" lists all its details, including that it is enabled.

fduncanh commented 3 weeks ago

posted a query about this on GStreamer discourse board: deleted it

$v4l2-ctl --list-devices
<snip>
rpivid (platform:rpivid):
        /dev/video19
        /dev/media0

$v4l2-ctl -d19 -D
Driver Info:
        Driver name      : rpivid
        Card type        : rpivid
        Bus info         : platform:rpivid
        Driver version   : 6.6.51
        Capabilities     : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : rpivid
        Model            : rpivid
        Serial           : 
        Bus info         : platform:rpivid
        Media version    : 6.6.51
        Hardware revision: 0x00000000 (0)
        Driver version   : 6.6.51
Interface Info:
        ID               : 0x0300000c
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : rpivid-source
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Source
          Link 0x02000008: to remote pad 0x1000004 of entity 'rpivid-proc' (Video Decoder): Data, Enabled, Immutable
fduncanh commented 3 weeks ago

This is where the problem with gstreamer happens

0:00:00.143873339 82483 0x55563a41b400 INFO              v4l2codecs gstv4l2codecdevice.c:391:gst_v4l2_codec_find_devices: Found decoder device rpivid-proc
0:00:00.144000691 82483 0x55563a41b400 INFO              v4l2codecs plugin.c:59:register_video_decoder:<v4l2decoder0> Registering rpivid-proc as H265 Decoder
0:00:00.144019024 82483 0x55563a41b400 WARN      v4l2codecs-h265dec gstv4l2codech265dec.c:1684:gst_v4l2_codec_h265_dec_register: Not registering H265 decoder since it produces no supported format
0:00:00.144033524 82483 0x55563a41b400 INFO      GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/lib/aarch64-linux-gnu/gstreamer-1.0/libgstv4l2codecs.so" loaded

its a place to start to see what is going wrong.

connorh315 commented 3 weeks ago

This is where the problem with gstreamer happens

0:00:00.143873339 82483 0x55563a41b400 INFO              v4l2codecs gstv4l2codecdevice.c:391:gst_v4l2_codec_find_devices: Found decoder device rpivid-proc
0:00:00.144000691 82483 0x55563a41b400 INFO              v4l2codecs plugin.c:59:register_video_decoder:<v4l2decoder0> Registering rpivid-proc as H265 Decoder
0:00:00.144019024 82483 0x55563a41b400 WARN      v4l2codecs-h265dec gstv4l2codech265dec.c:1684:gst_v4l2_codec_h265_dec_register: Not registering H265 decoder since it produces no supported format
0:00:00.144033524 82483 0x55563a41b400 INFO      GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/lib/aarch64-linux-gnu/gstreamer-1.0/libgstv4l2codecs.so" loaded

its a place to start to see what is going wrong.

I was reading somewhere that the output of the decoder is SAND8/SAND30... Maybe that's where the issue lies?

fduncanh commented 3 weeks ago

The problem is much more basic than that: 

   src_caps = gst_v4l2_decoder_enum_src_formats (decoder);

  if (gst_caps_is_empty (src_caps)) {
    GST_WARNING ("Not registering H265 decoder since it produces no "
        "supported format");
    goto done;

GstCaps *
gst_v4l2_decoder_enum_src_formats (GstV4l2Decoder * self)
{
  gint ret;
  struct v4l2_format fmt = {
    .type = self->src_buf_type,
  };
  GstCaps *caps;
  gint i;

  g_return_val_if_fail (self->opened, FALSE);

  ret = ioctl (self->video_fd, VIDIOC_G_FMT, &fmt);
  if (ret < 0) {
    GST_ERROR_OBJECT (self, "VIDIOC_G_FMT failed: %s", g_strerror (errno));
    return FALSE;
  }

  caps =
      gst_v4l2_decoder_probe_caps_for_format (self,
      fmt.fmt.pix_mp.pixelformat, fmt.fmt.pix_mp.width, fmt.fmt.pix_mp.height);

  /* And then enumerate other possible formats and place that as a second
   * structure in the caps */
  for (i = 0; ret >= 0; i++) {
    struct v4l2_fmtdesc fmtdesc = { i, self->src_buf_type, };
    GstCaps *tmp;

    ret = ioctl (self->video_fd, VIDIOC_ENUM_FMT, &fmtdesc);
    if (ret < 0) {
      if (errno != EINVAL)
        GST_ERROR_OBJECT (self, "VIDIOC_ENUM_FMT failed: %s",
            g_strerror (errno));
      continue;
    }

    tmp = gst_v4l2_decoder_probe_caps_for_format (self, fmtdesc.pixelformat,
        fmt.fmt.pix_mp.width, fmt.fmt.pix_mp.height);
    caps = gst_caps_merge (caps, tmp);
  }

  return caps;
}
fduncanh commented 3 weeks ago

HOORAY!!!!

commenting out that test indeed allows the "v4l2slh265dec V4L2 Stateless H.265 Video Decoder" to be registered, and show up with "1 feature" in "gst-inspect-1.0 v4l2codec"

connorh315 commented 3 weeks ago

And it works???

fduncanh commented 3 weeks ago

not yet, sorry GST_DEBUG=2 The caps are returned as "empty"

v4l2slh265dec0 error Unsupported bitdepth/chroma format v4l2slh265dec0 error No support for 2896x2160 8bit chroma IDC1 v4l2slh265dec0 Failed to negotiate with downstream v4l2slh265dec0 subclass does not want accept new sequence v4l2slh265dec0 Failed to process sps queue error internal data stream error queue error streaming stopped, reason not-negotiated (-4)

but it's a start......

fduncanh commented 3 weeks ago

(without UxPlay, just run gst-inspect-1.0)

The rpivid driver is reporting valid formats in response to VIDIOC_G_FMT and VIDIOC_ENUM_FMT ioctl calls

VIDIOC_G_FMT:
width 1920
height 1088
pixelformat   (fourcc code)   32 31 43 4e    NC12

VIDIOC_ENUM_FMT:

index   type   flags    pixelformat      description
0           9     0     32 31 43 4e  NC12   "Y/CbCr 4:2:0 (120b cols)"
1           9     0     32 31 43 4e  NC30   "10-bit Y/CbCr 4:2:0 (120b cols)"
fduncanh commented 3 weeks ago 
static GstCaps *
gst_v4l2_decoder_probe_caps_for_format (GstV4l2Decoder * self,
    guint32 pixelformat, gint unscaled_width, gint unscaled_height)
{
  gint index = 0;
  GstCaps *caps, *tmp, *size_caps;
  GstVideoFormat format;
  guint32 drm_fourcc;

  GST_DEBUG_OBJECT (self, "enumerate size for %" GST_FOURCC_FORMAT,
      GST_FOURCC_ARGS (pixelformat));

  if (!gst_v4l2_format_to_video_format (pixelformat, &format))    <+++++++++++++++++++++this is what fails!!!
    return gst_caps_new_empty ();

  caps = gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING,
      gst_video_format_to_string (format), NULL);

  size_caps = gst_caps_new_empty ();
  while ((tmp = gst_v4l2_decoder_enum_size_for_format (self, pixelformat,
              index++, unscaled_width, unscaled_height))) {
    size_caps = gst_caps_merge (size_caps, tmp);
  }

  if (!gst_caps_is_empty (size_caps)) {
    tmp = caps;
    caps = gst_caps_intersect_full (tmp, size_caps, GST_CAPS_INTERSECT_FIRST);
    gst_caps_unref (tmp);
  }

  /* TODO: Add a V4L2 to DRM fourcc translator for formats that we don't support
   * in software.
   */
  drm_fourcc = gst_video_dma_drm_fourcc_from_format (format);
  if (drm_fourcc /* != DRM_FORMAT_INVALID */ ) {
    GstCaps *drm_caps;

    drm_caps = gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING,
        "DMA_DRM", "drm-format", G_TYPE_STRING,
        gst_video_dma_drm_fourcc_to_string (drm_fourcc, 0), NULL);
    gst_caps_set_features_simple (drm_caps,
        gst_caps_features_new_single_static_str
        (GST_CAPS_FEATURE_MEMORY_DMABUF));

    if (!gst_caps_is_empty (size_caps)) {
      gst_caps_set_features_simple (size_caps,
          gst_caps_features_new_single_static_str
          (GST_CAPS_FEATURE_MEMORY_DMABUF));
      tmp = drm_caps;
      drm_caps =
          gst_caps_intersect_full (tmp, size_caps, GST_CAPS_INTERSECT_FIRST);
      gst_caps_unref (tmp);
    }

    caps = gst_caps_merge (drm_caps, caps);
  }

  gst_caps_unref (size_caps);

  return caps;
}
gboolean
gst_v4l2_format_to_video_format (guint32 pix_fmt, GstVideoFormat * out_format)
{
  struct FormatEntry *entry = lookup_v4l2_fmt (pix_fmt);

  if (!entry)
    return FALSE;

  *out_format = entry->gst_fmt;
  return TRUE;
}

static struct FormatEntry format_map[] = {
  {V4L2_PIX_FMT_NV12, 1, GST_VIDEO_FORMAT_NV12, 8, 420},
  {V4L2_PIX_FMT_YUYV, 1, GST_VIDEO_FORMAT_YUY2, 8, 422},
  {V4L2_PIX_FMT_SUNXI_TILED_NV12, 1, GST_VIDEO_FORMAT_NV12_32L32, 8, 422},
  {V4L2_PIX_FMT_NV12_4L4, 1, GST_VIDEO_FORMAT_NV12_4L4, 8, 420},
  {V4L2_PIX_FMT_MM21, 2, GST_VIDEO_FORMAT_NV12_16L32S, 8, 420},
  {V4L2_PIX_FMT_YUV420M, 3, GST_VIDEO_FORMAT_I420, 8, 420},
  {V4L2_PIX_FMT_P010, 1, GST_VIDEO_FORMAT_P010_10LE, 16, 420},
  {V4L2_PIX_FMT_NV15_4L4, 1, GST_VIDEO_FORMAT_NV12_10LE40_4L4, 10, 420},
  {V4L2_PIX_FMT_MT2110T, 2, GST_VIDEO_FORMAT_MT2110T, 10, 420},
  {V4L2_PIX_FMT_MT2110R, 2, GST_VIDEO_FORMAT_MT2110R, 10, 420},
  {0,}
};


static struct FormatEntry *
lookup_v4l2_fmt (guint v4l2_pix_fmt)
{
  gint i;
  struct FormatEntry *ret = NULL;

  for (i = 0; format_map[i].v4l2_pix_fmt; i++) {
    if (format_map[i].v4l2_pix_fmt == v4l2_pix_fmt) {
      ret = format_map + i;
      break;
    }
  }

  return ret;
}
fduncanh commented 3 weeks ago 

VIDIOC_G_FMT:
width 1920
height 1088
pixelformat   (fourcc code)   32 31 43 4e    NC12

VIDIOC_ENUM_FMT:

index   type   flags    pixelformat      description
0           9     0     32 31 43 4e  NC12   "Y/CbCr 4:2:0 (120b cols)"
1           9     0     32 31 43 4e  NC30   "10-bit Y/CbCr 4:2:0 (120b cols)"
connorh315 commented 3 weeks ago
  • so the issue is the unusual fourcc pixelformat codes NC12 and NC30. I havent yet mange to find them listed anywhere.

VIDIOC_G_FMT:

width 1920

height 1088

pixelformat   (fourcc code)   32 31 43 4e    NC12

VIDIOC_ENUM_FMT:

index   type   flags    pixelformat      description

0           9     0     32 31 43 4e  NC12   "Y/CbCr 4:2:0 (120b cols)"

1           9     0     32 31 43 4e  NC30   "10-bit Y/CbCr 4:2:0 (120b cols)"

Those are the SAND formats I was referencing earlier. (Post 12 of https://forums.raspberrypi.com/viewtopic.php?t=343593)

fduncanh commented 3 weeks ago

OK! I'm guessing "sand" refers to the proprietary hevc decoding block hardware in R PI.

Well I got to understand enough to discover that by myself!

The "unsand" code in jc-kinesim's ffmped rpi-ffmpeg work is

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavfilter/vf_unsand.c

fduncanh commented 3 weeks ago

old code (2008)

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavfilter/vf_unsand.c

new SAND code. from R Pi (committed by jc-kinesim dec 2023)https://github.com/jc-kynesim/rpi-ffmpeg/commit/b6b137b1d039b42b15325f87f55cb7c38e2270b0

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavutil/rpi_sand_fn_pw.h

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavutil/rpi_sand_fns.c

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavutil/rpi_sand_fns.h

fduncanh commented 3 weeks ago

I have asked jc-kinesim whether this R Pi Foundation code (copyright date 2018) replaces or works with the older non-RPi vf_unsand.c https://github.com/jc-kynesim/rpi-ffmpeg/issues/94#issuecomment-2437314698

Its a LOT of code. Maybe too big to be accepted into gstreamer? I wonder if it could be accessed by checking whether libav locally contains this code or not.

this is the commit in dec 2023 at rpi-ffmpeg.

https://github.com/jc-kynesim/rpi-ffmpeg/commit/b6b137b1d039b42b15325f87f55cb7c38e2270b0

fduncanh commented 3 weeks ago

Its possible that UxPlay could offer a gstreamer patch that UxPlay users could use to rebuild libgstv4l2codecs.c

This is how it supported gstreamer's v4l2 (stateful h264 decode) before I got fixes accepted into gstreamer.

fduncanh commented 3 weeks ago

dependecies of RPI sand code

include "libavutil/frame.h" this might drag in a lot, should check if gstreamer-libav uses it.

include "avassert.h" (presumably replaceable by assert.h)

include "frame.h" same as libavutil/frame.h ???

connorh315 commented 3 weeks ago

I doubt that you would want this as it's just adding unnecessary bloat on the project, however I've just gotten FFmpeg working in UxPlay and correctly decoding H.265 streams. Worst comes to worst that can be an alternative provision?

fduncanh commented 3 weeks ago

well now at least we know what the issues are. 

$v4l2-ctl --list-formats-ext -d 19
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'NC12' (Y/CbCr 4:2:0 (128b cols))
        [1]: 'NC30' (10-bit Y/CbCr 4:2:0 (128b cols))

SAND pixelformats fourcc = NC12, NC30 are not supported in gstreamer-plugins-bad v4l2codecs

fduncanh commented 3 weeks ago

vf_unsand .c was also added Dec 2023 its much older, Dec 2023 is just the date when kernel 6.0.1 was added

https://github.com/jc-kynesim/rpi-ffmpeg/commit/c7d8474ffa5689abf99367c43ec2d39a1957f564

fduncanh commented 3 weeks ago

some more into

A response from @6by9 (R Pi engineer) to my post at https://github.com/jc-kynesim/rpi-ffmpeg/issues/94#issuecomment-2437314698

AIUI Mainline GStreamer will only support formats that are defined in the mainline kernel, and upstreaming the HEVC decoder is still in progress (I need to talk to jc-kynesim about it this coming week).

Once the format and driver are in the mainline kernel, I wouldn't expect there to be any great obstacles to adding the format to GStreamer, but would ask Nicolas Dufresne as the person who knows that area best.

Sand was an internal Broadcom name for the format as it originally came from a company called Sand Video that was bought by them in 2004 - https://www.theregister.com/2004/04/13/broadcom_acquires_sand/

I also posted on Gstreamer forum:

https://discourse.gstreamer.org/t/v4l2codecs-feasibility-of-adding-support-for-raspberry-pi-hevc-sand-formats-nc12-nc30/3522

fduncanh commented 3 weeks ago

some more activity: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7355

maybe some action at R Pi might get stirred up by my queries.

fduncanh commented 1 week ago

@connorh315

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7355 there is interesting R Pi HEVC actvity at the above link today