Trying to pass rtc media packet to ffmpeg

[generalomosco]

I've be trying to pass rtc media packet to ffmpeg, so i decide using unix socket but seems the frame is not valid! I'm passing the frame bytes s.vCallback(s.width,s.height,videoKeyframe, int64(t), sample.Data); via unix socket then use ffmpeg to convert it to mp4 by connecting to the socket but unfortunately the mp4 is always corrupted without displaying anything.

The command line ffmpeg -vsync cfr -f rawvideo -c:v rawvideo -fflags nobuffer -s 320x180 -pix_fmt yuv420p -i unix:./video.sock -f s16le -ar 48k -ac 1 -fflags nobuffer -i unix:./audio.sock -flags +global_header -acodec libfdk_aac -vsync 1 -vcodec libx264 -r 25 -b:v 4000k -pix_fmt yuv420p -preset slow -qp 0 vid.mp4

Here is the code below which i strap scope from save-to-webm

package streamBuf

import (
    "errors"
    "fmt"
    "github.com/pion/rtp"
    "github.com/pion/rtp/codecs"
    "github.com/pion/webrtc/v2"
    "github.com/pion/webrtc/v2/pkg/media/samplebuilder"
)

var (
    // ErrCodecNotSupported is returned when a rtp packed it pushed with an unsupported codec
    ErrCodecNotSupported = errors.New("codec not supported")
)

type MediaBuf struct {
    id                             string
    typ                             string
    audioBuilder, videoBuilder     *samplebuilder.SampleBuilder
    audioTimestamp, videoTimestamp uint32
  audCallback AudCallback
  vidCallback VidCallback
  width int
  height int
  initVid bool
}
type AudCallback func(bool, int64, []byte)
type VidCallback func(int, int, bool, int64, []byte)
func NewMediaBuf(id, typ string, audCB AudCallback, vidCB VidCallback) *MediaBuf {
    return &MediaBuf{
        id:           id,
        typ:           typ,
        audCallback:           audCB,
        vidCallback:           vidCB,
        audioBuilder: samplebuilder.New(10, &codecs.OpusPacket{}),
        videoBuilder: samplebuilder.New(10, &codecs.VP8Packet{}),
    }
}

func (s *MediaBuf) ID() string {
    return s.id
}
func (s *MediaBuf) PushRTP(pkt *rtp.Packet) error {
    if s.typ == "video" && pkt.PayloadType == webrtc.DefaultPayloadTypeVP8 {
        s.PushVP8(pkt)
    } else if s.typ == "audio" && pkt.PayloadType == webrtc.DefaultPayloadTypeOpus {
        s.PushOpus(pkt)
    }
    return ErrCodecNotSupported
}

func (s *MediaBuf) Stop() {
    s.Close()
}

func (s *MediaBuf) Close() {
    fmt.Printf("Finalizing media...\n")
}
func (s *MediaBuf) PushOpus(rtpPacket *rtp.Packet) {
    s.audioBuilder.Push(rtpPacket)

    for {
        sample, timestamp := s.audioBuilder.PopWithTimestamp()
        if sample == nil {
            return
        }
            if s.audioTimestamp == 0 {
                s.audioTimestamp = timestamp
            }
            t := (timestamp - s.audioTimestamp) / 48
             s.aCallback(true, int64(t), sample.Data)

    }
}
func (s *MediaBuf) PushVP8(rtpPacket *rtp.Packet) {
    s.videoBuilder.Push(rtpPacket)

    for {
        sample, timestamp := s.videoBuilder.PopWithTimestamp()
        if sample == nil {
            return
        }
        // Read VP8 header.
        videoKeyframe := (sample.Data[0]&0x1 == 0)
        if videoKeyframe {
            // Keyframe has frame information.
            raw := uint(sample.Data[6]) | uint(sample.Data[7])<<8 | uint(sample.Data[8])<<16 | uint(sample.Data[9])<<24
            width := int(raw & 0x3FFF)
            height := int((raw >> 16) & 0x3FFF)

            if !s.initVid {
                // Initialize WebM saver using received frame size.
                s.width = width
        s.height = height
        s.initVid = true
            }
        }
        if s.initVid {
            if s.videoTimestamp == 0 {
                s.videoTimestamp = timestamp
            }
            t := (timestamp - s.videoTimestamp) / 90
            s.vCallback(s.width,s.height,videoKeyframe, int64(t), sample.Data);
        }
    }
}
func (s *MediaBuf) vCallback(width, height int, keyframe bool, timestamp int64, b []byte) {
  s.vidCallback(width, height, keyframe, timestamp, b)
}
func (s *MediaBuf) aCallback(keyframe bool, timestamp int64, b []byte) {
  s.audCallback(keyframe, timestamp, b)
}

[at-wat]

In your code, vCallback and aCallback receive VP8 frames ~with RTP payload descriptor~ and OPUS frames.

ffmpeg \
  -vsync cfr \
  -f rawvideo \ # (I'm not very sure ffmpeg can read raw VP8 stream as rawvideo *1)
  -c:v rawvideo \ # it specifies input codec as raw, but actually VP8 stream is given
  -fflags nobuffer 
  -s 320x180 \
  -pix_fmt yuv420p \
  -i unix:./video.sock \
  -f s16le \ # it specifies input format as raw signed 16^bit little-endian, but actually OPUS stream is given
  -ar 48k \
  -ac 1 \
  -fflags nobuffer \
  -i unix:./audio.sock \
  -flags +global_header -acodec libfdk_aac -vsync 1 -vcodec libx264 -r 25 -b:v 4000k -pix_fmt yuv420p \
  -preset slow -qp 0 vid.mp4

VP8 RTP descriptor https://tools.ietf.org/html/rfc7741#section-4.1 ~Some heading bytes in sample.Data are VP8 RTP descriptor, which is not a part of raw VP8 stream.~ Payload descriptor is parsed by codecs.VP8Packet depacketizer and not in sample.Data.

*1: In general, raw stream of encoded video is difficult to be handled since it usually doesn't have information of frame boundary. Using RTP is easier.

[generalomosco]

Thanks for the info, i will follow the process

[at-wat]

@Generalomosco sorry, payload descriptor is already parsed by codecs.VP8Packet depacketizer and not in sample.Data. Fixed above comment.

[Sean-Der]

Hey @Generalomosco it would be really amazing if you were able to share this with others!

If you are able to open a PR to this repo me and @at-wat would be able to help also :) no rush though.

The PR doesn't have to be perfect. I can refactor the code to make it pass our tests, as long as you can get something basic working. Thanks for using Pion, I love seeing cool stuff like this.

[generalomosco]

Thanks i will by tomorrow