Problem decoding larger packets

[ConnorRigby]

I'm finding a problem with decoding RTP packets that are larger than what gstreamer chunks. I'm not certain this is the complete issue, i'm just not quite sure how to debug further.

Testing using the code in the rtp demo without any changes, i can stream via gstreamer successfully with something like:

gst-launch-1.0 v4l2src ! video/x-h264, stream-format=byte-stream, alignment=au, width=1920, height=1080, pixel-aspect-ratio=1/1, framerate=30/1 ! rtph264pay pt=96 ! udpsink host=127.0.0.1 port=5000

However using something that i assumed would be functionally equivilent:

gst-launch-1.0 -q v4l2src ! video/x-h264, stream-format=byte-stream, alignment=au, width=1920, height=1080, pixel-aspect-ratio=1/1, framerate=30/1 ! rtph264pay pt=96 ! fdsink | nc -r -u 127.0.0.1 5000

does not work. I assume the udpsink is chunking messages in such a way that always triggers the incomplete portion of this code:

  defp handle_unit_type(:fu_a, {header, data}, buffer, state) do
    %Buffer{metadata: %{rtp: %{sequence_number: seq_num}}} = buffer

    case FU.parse(data, seq_num, map_state_to_fu(state)) do
      {:ok, {data, type}} ->
        IO.inspect(data, label: "parsed data")
        data = NAL.Header.add_header(data, 0, header.nal_ref_idc, type)
        buffer_output(data, buffer, %State{state | parser_acc: nil})

      {:incomplete, fu} ->
        IO.inspect(fu, label: "incomplete")
        {{:ok, []}, %State{state | parser_acc: fu}}

      {:error, _} = error ->
        error
    end
  end

in depayloader.ex.

[ConnorRigby]

For further explanation of what i'm trying to do: i'm building a Nerves application that streams h264 data from a nerves device too a membrane backend. I'm getting h264 data from gstreamer via an Erlang port. Here is my test code if you are interested.

defmodule StreamTest do
  use GenServer

  def start_link(args) do
    GenServer.start_link(__MODULE__, args, name: __MODULE__)
  end

  def init(_) do
    port = open_port()
    socket = open_socket()
    {:ok, %{port: port, socket: socket}}
  end

  def handle_info({port, {:data, data}}, %{port: port} = state) do
    :ok = :gen_udp.send(state.socket, {{127, 0, 0, 1}, 5000}, data)
    {:noreply, state}
  end

  def open_socket do
    {:ok, socket} = :gen_udp.open(0, [:binary, ip: :any, active: true])
    socket
  end

  defp open_port do
    gst = System.find_executable("gst-launch-1.0")

    args = [
      "v4l2src",
      "!",
      "video/x-h264,",
      "stream-format=byte-stream,",
      "alignment=au,",
      "width=1920,",
      "height=1080,",
      "pixel-aspect-ratio=1/1,",
      "framerate=30/1",
      "!",
      "rtph264pay",
      "pt=96",
      "!",
      "fdsink",
      "fd=4"
    ]

    :erlang.open_port({:spawn_executable, gst}, [
      {:args, args},
      :binary,
      :nouse_stdio,
      :exit_status
    ])
  end
end

[bblaszkow06]

Hi @ConnorRigby! I'm glad you got interested in our framework! First, the 2 pipelines you provided only seem functionally equivalent. I'm not sure how familiar you are with network protocols, but for RTP the way you chunk the data does matter - RTP adds additional headers on top of UDP datagram, so 1 UDP Datagram must only contain 1 RTP packet. Netcat (nc) will not know how to split them. This can be observed in Wireshark - here's output using udpsink: And here's the second pipeline: Notice that packets some datagrams are enormous, some get not recognized by Wireshark (Unknown RTP version 1), and you can see some datagrams containing lots of RTP packets - sequence number jumps by 9 between datagrams 23 and 24.

---

Regarding your test: similar thing may happen if the data sent through port get buffered either on write in fdsink or when receiving on the erlang side. I'm not sure where it happens, but it does happen - notice the sequence number jumps:

The bottom line is - you either have to use udpsink or find a way to mark the end of each RTP packet when sending them over port

[ConnorRigby]

Thanks! this is more or less what i thought the issue was. When i read the docs for the udpsink element in gstreamer, they lead me to believe that it was just blasting the packets from whatever the previous element was, no chunking etc. I think i have some code that buffers RTP packets from a different project, so i think i know how to deal with it. Thanks for the help!

Closing. Feel free to reopen if you feel necessary.