Server to server video sending question

[crisboarna]

Hi, I am trying to send via WebRTC server-to-server some H264 data.

On the sending side I am reading a .264 file that is playable on VLC. I also extract the data via ffmpeg for comparison (Image 1).

On the receiving side I am writing the data to a .264 file from which when I extract via same command the images from the container, the image is distorted significantly. (Image 2)

Size wise, the input data is 6017 bytes while on the receiving side the file is 6097 bytes.

Here comes my first question, is this expected behaviour for the size to be different?

Secondly, comparing the two with binary editor, I can see there is no similarity between the two bitewise. Is my understanding about this data sending wrong or I am missing out on some steps ?

Why is the image received on other side so significantly distorted ?

The 264 containers I am sending contain single images(project requirement) so it being distored is a big issue.

Image 1 - on sender side Image 2 -on receiver side

#include "RTCClient.hpp"
#include <sys/time.h>
#include <chrono>
#include <fstream>
#include <future>
#include <thread>
#include <vector>

#include <nlohmann/json.hpp>
#include <spdlog/sinks/stdout_color_sinks.h>

using nlohmann::json;

namespace webrtc {
void RTCClient::onSignalMessage(nlohmann::json &message) {
  auto iterator_message = message.find("id");
  if (iterator_message == message.end()) { return; }

  auto message_id = iterator_message->get<std::string>();

  iterator_message = message.find("type");
  if (iterator_message == message.end()) { return; }

  auto type = iterator_message->get<std::string>();

  std::shared_ptr<webrtc::Client> target_client;
  if (auto iterator_clients = clients.find(message_id); iterator_clients != clients.end()) {
    target_client = iterator_clients->second;
  } else if (type == "offer") {
    logger->trace("Answering to: {}", message_id);
    target_client = createPeerConnection(message_id);
    clients.emplace(message_id, target_client);
  } else {
    return;
  }

  if (type == "offer" || type == "answer") {
    auto sdp = message["sdp"].get<std::string>();
    target_client->peer_connection->setRemoteDescription(rtc::Description(sdp, type));
  }
}

std::shared_ptr<ClientTrackData> RTCClient::addVideo(const std::shared_ptr<rtc::PeerConnection> &peer_connection,
  const uint8_t payloadType,
  const uint32_t ssrc,
  const std::string &cname,
  const std::string &msid,
  const std::function<void(void)> &onOpen) {
  auto video = rtc::Description::Video(cname);
  video.setDirection(rtc::Description::Direction::SendRecv);
  video.addH264Codec(payloadType);
  video.addSSRC(ssrc, cname, msid, cname);
  std::shared_ptr<rtc::Track> track = peer_connection->addTrack(video);

  // create RTP configuration
  auto rtpConfig =
    make_shared<rtc::RtpPacketizationConfig>(ssrc, cname, payloadType, rtc::H264RtpPacketizer::defaultClockRate);
  // create packetizer
  auto packetizer = make_shared<rtc::H264RtpPacketizer>(rtc::H264RtpPacketizer::Separator::LongStartSequence, rtpConfig);
  // create H264 handler
  auto h264Handler = make_shared<rtc::H264PacketizationHandler>(packetizer);
  // add RTCP SR handler
  auto srReporter = make_shared<rtc::RtcpSrReporter>(rtpConfig);
  h264Handler->addToChain(srReporter);
  // add RTCP NACK handler
  auto nackResponder = std::make_shared<rtc::RtcpNackResponder>();
  h264Handler->addToChain(nackResponder);
  // set handler
  track->setMediaHandler(h264Handler);
  track->onOpen(onOpen);
  auto trackData = make_shared<ClientTrackData>(track, srReporter);
  return trackData;
}

void RTCClient::addToStream(std::shared_ptr<Client> &client, bool is_adding_video) {
  if (client->getState() == Client::State::Waiting) {
    client->setState(is_adding_video ? Client::State::WaitingForAudio : Client::State::WaitingForVideo);
  } else if ((client->getState() == Client::State::WaitingForAudio && !is_adding_video)
             || (client->getState() == Client::State::WaitingForVideo && is_adding_video)) {

    // Audio and video tracks are collected now
    // NOLINTNEXTLINE(cppcoreguidelines-pro-bounds-array-to-pointer-decay,hicpp-no-array-decay)
    assert(client->video.has_value() && client->audio.has_value());

    auto video = client->video.value();
    auto audio = client->audio.value();

    auto currentTime_us = double(currentTimeInMicroSeconds());
    constexpr int conversion_factor = 1000;

    auto currentTime_s = currentTime_us / (conversion_factor * conversion_factor);

    // set start time of stream
    video->sender->rtpConfig->setStartTime(currentTime_s, rtc::RtpPacketizationConfig::EpochStart::T1970);
    audio->sender->rtpConfig->setStartTime(currentTime_s, rtc::RtpPacketizationConfig::EpochStart::T1970);

    // start stat recording of RTCP SR
    video->sender->startRecording();
    audio->sender->startRecording();

    client->setState(Client::State::Ready);
  }
  if (client->getState() == Client::State::Ready) { logger->trace("A/V stream ready"); }
}

std::shared_ptr<webrtc::Client> RTCClient::createPeerConnection(std::string &remote_id) {
  auto peer_connection = std::make_shared<rtc::PeerConnection>(config);
  std::shared_ptr<webrtc::Client> peer_client = make_shared<webrtc::Client>(peer_connection);

  peer_connection->onStateChange([this, remote_id](rtc::PeerConnection::State state) {
    logger->trace("State: {}", int(state));
    if (state == rtc::PeerConnection::State::Disconnected || state == rtc::PeerConnection::State::Failed
        || state == rtc::PeerConnection::State::Closed) {
      thread_pool->push_task([this, remote_id]() { clients.erase(remote_id); });
    }
  });

  peer_connection->onGatheringStateChange(
    [this, remote_id, wpc = make_weak_ptr(peer_connection)](rtc::PeerConnection::GatheringState state) {
      logger->trace("Gathering State change: {}", int(state));
      if (state == rtc::PeerConnection::GatheringState::Complete) {
        if (auto peer_con = wpc.lock()) {
          auto description = peer_con->localDescription();
          json message = {
            { "id", remote_id }, { "type", description->typeString() }, { "sdp", std::string(description.value()) }
          };
          // Gathering complete, send answer
          ws_client->send(message.dump());
        }
      }
    });

  peer_connection->onLocalDescription([this, remote_id](const rtc::Description &description) {
    json message = {
      { "id", remote_id }, { "type", description.typeString() }, { "description", std::string(description) }
    };

    ws_client->send(message.dump());
  });

  constexpr int video_payload_type = 102;
  peer_client->video = addVideo(peer_client->peer_connection,
    video_payload_type,
    1,
    "video-stream",
    "stream1",
    [this, remote_id, weak_ptr_client = make_weak_ptr(peer_client)]() {
      thread_pool->submit([this, weak_ptr_client]() {
        if (auto client_ptr = weak_ptr_client.lock()) { addToStream(client_ptr, true); }
      });
      logger->debug("Video from {} opened", remote_id);
    });
  return peer_client;
}

void RTCClient::init() {
  try {

    bool terminate = false;
    std::chrono::seconds timeout(1);
    std::shared_ptr<std::promise<void>> get_remote_id_exit_signal_promise = std::make_shared<std::promise<void>>();
    std::shared_ptr<std::future<void>> exit_signal_future =
      std::make_shared<std::future<void>>(get_remote_id_exit_signal_promise->get_future());

    std::string remote_id;
    auto getRemoteID = [&remote_id, &exit_signal_future, &get_remote_id_exit_signal_promise]() {
      getline(std::cin, remote_id);
      if (exit_signal_future->wait_for(std::chrono::milliseconds(0)) != std::future_status::ready) {
        get_remote_id_exit_signal_promise->set_value();
      }
    };

    while (!terminate && !skip_waiting_init) {
      std::this_thread::sleep_for(std::chrono::milliseconds(RTC_INIT_DELAY));
      logger->info("Enter a remote ID to connect:");

      std::thread th(getRemoteID);

      while (!skip_waiting_init && exit_signal_future->wait_for(timeout) != std::future_status::ready) {}

      if (exit_signal_future->wait_for(std::chrono::milliseconds(0)) != std::future_status::ready) {
        th.detach();
        get_remote_id_exit_signal_promise->set_value();
      } else {
        th.join();
      }

      if (skip_waiting_init) { continue; }

      if (remote_id.empty()) {
        logger->error("Invalid remote ID (No ID provided)");
        remote_id.clear();
        continue;
      }

      if (remote_id == this->ws_client->ws_local_id) {
        logger->error("Invalid remote ID (This is the local ID)");
        remote_id.clear();
        continue;
      }

      logger->info("Attempting connection to: {}", remote_id);

      auto peer_client = createPeerConnection(remote_id);

      // We are the offerer, so create a data channel to initiate the process
      const std::string label = "test";
      logger->debug("Creating DataChannel with label: {}", label);

      auto peer_data_channel = peer_client->peer_connection->createDataChannel(label);

      peer_data_channel->onOpen([this, remote_id]() { logger->trace("DataChannel from {} open", remote_id); });

      peer_data_channel->onClosed([this, remote_id]() { logger->warn("DataChannel from {} closed", remote_id); });

      peer_client->data_channel = peer_data_channel;
      peer_client->peer_connection->setLocalDescription();
      clients.emplace(remote_id, peer_client);
      terminate = true;
      logger->info("Init success to {}", remote_id);
    }
  } catch (const std::exception &e) { logger->error("Error finishing RTC init: {}", e.what()); }
}

bool RTCClient::sendVideo(const std::vector<std::byte> &data) {
  sampleTime_us += sampleDuration_us;
  std::vector<ClientTrack> tracks{};

  for (auto &pair_id_client : clients) {
    auto client_id = pair_id_client.first;
    auto client = pair_id_client.second;
    auto optTrackData = client->video;
    if (client->getState() == Client::State::Ready && optTrackData.has_value()) {
      auto trackData = optTrackData.value();
      tracks.emplace_back(ClientTrack(client_id, trackData));
    }

    if (!tracks.empty()) {
      for (auto &client_track : tracks) {
        auto track_data = client_track.trackData;
        // sample time is in us, we need to convert it to seconds
        auto elapsedSeconds = double(sampleTime_us) / (RTC_TIME_CONVERSION_DEFAULT * RTC_TIME_CONVERSION_DEFAULT);
        auto rtpConfig = track_data->sender->rtpConfig;
        // get elapsed time in clock rate
        uint32_t elapsedTimestamp = rtpConfig->secondsToTimestamp(elapsedSeconds);

        // set new timestamp
        rtpConfig->timestamp = rtpConfig->startTimestamp + elapsedTimestamp;

        // get elapsed time in clock rate from last RTCP sender report
        auto reportElapsedTimestamp = rtpConfig->timestamp - track_data->sender->previousReportedTimestamp;
        // check if last report was at least 1 second ago
        if (rtpConfig->timestampToSeconds(reportElapsedTimestamp) > 1) { track_data->sender->setNeedsToReport(); }

        try {
          logger->trace("Sending size: {} to client: {}", data.size(), client_id);
          bool sent = track_data->track->send(data.data(), data.size());
          logger->trace("RTC message sending status: {} for size: {}", sent, data.size());
        } catch (const std::exception &e) { logger->error("Error sending RTC message: {}", e.what()); }
      }
    }
  }
  return true;
}

[[noreturn]] void RTCClient::onVideoEvent(const std::function<void(std::vector<std::byte>)> &callback) {
  for (const auto &client_entry : clients) {
    auto remote_id = client_entry.first;
    auto client = client_entry.second;
    auto client_video_channel = client->video;
    client_video_channel.value()->track->onMessage([this, remote_id, &callback](rtc::message_variant data) {
      try {
        // data holds either std::string or rtc::binary
        if (std::holds_alternative<std::string>(data)) {
          std::string raw_data = std::get<std::string>(data);
          logger->trace("Message from '{}' received: {}", remote_id, raw_data);
          // TODO: deal with string data ? callback(raw_data);
        } else {
          std::vector<std::byte> raw_data = std::get<rtc::binary>(data);
          logger->trace("Binary message from '{}' received, size: {}", remote_id, raw_data.size());
          callback(raw_data);
        }
      } catch (const std::exception &e) { logger->error("Error processing RTC message: {}", e.what()); }
    });
  }
  while (true) {}
}
}// namespace webrtc

[17:38:53:113803 +03:00] [WebSocketClient<xxxx.com>] [info] [thread 2103359]: ws_local_id: zGArZJY, ws_url: ws://xxxx:80/zGArZJY 
[17:38:53:117194 +03:00] [RTCClient<stun.l.google.com>] [info] [thread 2103359]: Using stun_server: stun:stun.l.google.com:19302, data_size: 500000
2022-07-07 17:38:53.117 DEBUG [2103359] [rtc::impl::TcpTransport::TcpTransport@42] Initializing TCP transport
2022-07-07 17:38:53.117 DEBUG [2103359] [rtc::impl::TcpTransport::connect@125] Connecting to xxxx.com:80
2022-07-07 17:38:53.119 DEBUG [2103359] [rtc::impl::TcpTransport::prepare@208] Trying address 34.245.187.250:80
2022-07-07 17:38:53.211 INFO  [2103404] [rtc::impl::TcpTransport::connect@180] TCP connected
2022-07-07 17:38:53.211 DEBUG [2103404] [rtc::impl::WsTransport::WsTransport@68] Initializing WebSocket transport
2022-07-07 17:38:53.211 DEBUG [2103404] [rtc::impl::WsTransport::sendHttpRequest@183] Sending WebSocket HTTP request
2022-07-07 17:38:53.299 DEBUG [2103404] [rtc::impl::WsHandshake::parseHttpResponse@213] WebSocket response code=101
2022-07-07 17:38:53.299 INFO  [2103404] [rtc::impl::WsTransport::incoming@115] WebSocket client-side open
2022-07-07 17:38:53.299 DEBUG [2103404] [rtc::impl::WebSocket::initWsTransport@344] WebSocket open
[17:38:53:299152 +03:00] [WebSocketClient<xxxx.com>] [info] [thread 2103404]: Connected, signaling ready
[17:38:53:804214 +03:00] [RTCClient<stun.l.google.com>] [info] [thread 2103359]: Enter a remote ID to connect:
2022-07-07 17:39:47.909 DEBUG [2103404] [rtc::impl::WsTransport::recvFrame@291] WebSocket finished message: type=text, length=1241
[17:39:47:909615 +03:00] [WebSocketClient<xxxx.com>] [debug] [thread 2103404]: {"description":"v=0\r\no=rtc 944047320 0 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE video-stream audio-stream 0\r\na=group:LS video-stream audio-stream\r\na=msid-semantic:WMS *\r\na=setup:actpass\r\na=ice-ufrag:C3vH\r\na=ice-pwd:hVew9Hab3783YqIgVdUTxt\r\na=ice-options:ice2,trickle\r\na=fingerprint:sha-256 F9:49:93:F1:26:D7:7A:07:26:E0:16:14:6E:37:B9:70:74:86:16:9C:1E:E5:99:F8:8A:1E:48:86:34:E1:C2:F7\r\nm=video 9 UDP/TLS/RTP/SAVPF 102\r\nc=IN IP4 0.0.0.0\r\na=mid:video-stream\r\na=sendrecv\r\na=ssrc:1 cname:video-stream\r\na=ssrc:1 msid:stream1 video-stream\r\na=rtcp-mux\r\na=rtpmap:102 H264/90000\r\na=fmtp:102 nack\r\na=fmtp:102 nack pli\r\na=fmtp:102 goog-remb\r\na=fmtp:102 profile-level-id=42e01f;packetization-mode=1;level-asymmetry-allowed=1\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111\r\nc=IN IP4 0.0.0.0\r\na=mid:audio-stream\r\na=sendrecv\r\na=ssrc:2 cname:audio-stream\r\na=ssrc:2 msid:stream1 audio-stream\r\na=rtcp-mux\r\na=rtpmap:111 OPUS/48000/2\r\na=fmtp:111 minptime=10;maxaveragebitrate=96000;stereo=1;sprop-stereo=1;useinbandfec=1\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=max-message-size:500000\r\n","id":"YlpGit7","type":"offer"}
[17:39:47:909714 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103404]: Answering to: YlpGit7
2022-07-07 17:39:47.909 DEBUG [2103400] [rtc::impl::Certificate::Generate@223] Generating certificate (OpenSSL)
2022-07-07 17:39:47.910 WARN  [2103404] [rtc::impl::Transport::recv@74] [json.exception.type_error.302] type must be string, but is null
2022-07-07 17:39:48.079 DEBUG [2103404] [rtc::impl::WsTransport::recvFrame@275] WebSocket received frame: opcode=1, length=1444
2022-07-07 17:39:48.079 DEBUG [2103404] [rtc::impl::WsTransport::recvFrame@291] WebSocket finished message: type=text, length=1444
[17:39:48:079620 +03:00] [WebSocketClient<xxxx.com>] [debug] [thread 2103404]: {"id":"YlpGit7","sdp":"v=0\r\no=rtc 944047320 0 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE video-stream audio-stream 0\r\na=group:LS video-stream audio-stream\r\na=msid-semantic:WMS *\r\na=setup:actpass\r\na=ice-ufrag:C3vH\r\na=ice-pwd:hVew9Hab3783YqIgVdUTxt\r\na=ice-options:ice2,trickle\r\na=fingerprint:sha-256 F9:49:93:F1:26:D7:7A:07:26:E0:16:14:6E:37:B9:70:74:86:16:9C:1E:E5:99:F8:8A:1E:48:86:34:E1:C2:F7\r\nm=video 60831 UDP/TLS/RTP/SAVPF 102\r\nc=IN IP4 192.168.173.240\r\na=mid:video-stream\r\na=sendrecv\r\na=ssrc:1 cname:video-stream\r\na=ssrc:1 msid:stream1 video-stream\r\na=rtcp-mux\r\na=rtpmap:102 H264/90000\r\na=fmtp:102 nack\r\na=fmtp:102 nack pli\r\na=fmtp:102 goog-remb\r\na=fmtp:102 profile-level-id=42e01f;packetization-mode=1;level-asymmetry-allowed=1\r\na=candidate:1 1 UDP 2122317823 192.168.173.240 60831 typ host\r\na=candidate:2 1 UDP 1686109951 86.124.113.189 60831 typ srflx raddr 0.0.0.0 rport 0\r\na=end-of-candidates\r\nm=audio 60831 UDP/TLS/RTP/SAVPF 111\r\nc=IN IP4 192.168.173.240\r\na=mid:audio-stream\r\na=sendrecv\r\na=ssrc:2 cname:audio-stream\r\na=ssrc:2 msid:stream1 audio-stream\r\na=rtcp-mux\r\na=rtpmap:111 OPUS/48000/2\r\na=fmtp:111 minptime=10;maxaveragebitrate=96000;stereo=1;sprop-stereo=1;useinbandfec=1\r\nm=application 60831 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 192.168.173.240\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=max-message-size:500000\r\n","type":"offer"}
2022-07-07 17:39:48.079 DEBUG [2103404] [rtc::impl::IceTransport::IceTransport@58] Initializing ICE transport (libjuice)
2022-07-07 17:39:48.079 INFO  [2103404] [rtc::impl::IceTransport::IceTransport@114] Using STUN server "stun.l.google.com:19302"
2022-07-07 17:39:48.080 INFO  [2103404] [rtc::impl::PeerConnection::changeSignalingState@1180] Changed signaling state to have-remote-offer
2022-07-07 17:39:48.080 DEBUG [2103404] [rtc::impl::PeerConnection::processLocalDescription@876] Adding media to local description, mid="video-stream", removed=false
2022-07-07 17:39:48.080 DEBUG [2103404] [rtc::impl::PeerConnection::processLocalDescription@876] Adding media to local description, mid="audio-stream", removed=false
2022-07-07 17:39:48.080 DEBUG [2103404] [rtc::impl::PeerConnection::processLocalDescription@864] Reciprocating application in local description, mid="0"
2022-07-07 17:39:48.080 INFO  [2103404] [rtc::impl::PeerConnection::changeSignalingState@1180] Changed signaling state to stable
2022-07-07 17:39:48.080 INFO  [2103404] [rtc::impl::PeerConnection::changeGatheringState@1167] Changed gathering state to in-progress
2022-07-07 17:39:48.080 DEBUG [2103403] [rtc::impl::WsTransport::sendFrame@338] WebSocket sending frame: opcode=1, length=1236
2022-07-07 17:39:48.080 INFO  [2103404] [rtc::impl::IceTransport::LogCallback@354] juice: agent.c:1002: Changing state to gathering
[17:39:48:080490 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103401]: Gathering State change: 1
2022-07-07 17:39:48.080 INFO  [2103404] [rtc::impl::IceTransport::LogCallback@354] juice: agent.c:1002: Changing state to connecting
2022-07-07 17:39:48.080 INFO  [2103404] [rtc::impl::PeerConnection::changeState@1149] Changed state to connecting
[17:39:48:080572 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103401]: State: 1
2022-07-07 17:39:48.081 INFO  [2103701] [rtc::impl::IceTransport::LogCallback@354] juice: agent.c:394: Using STUN server stun.l.google.com:19302
2022-07-07 17:39:48.080282+0300 nexttv-net[36441:2103404] [si_destination_compare] send failed: Invalid argument
2022-07-07 17:39:48.080294+0300 nexttv-net[36441:2103404] [si_destination_compare] send failed: Undefined error: 0
2022-07-07 17:39:48.080299+0300 nexttv-net[36441:2103404] [si_destination_compare] send failed: Invalid argument
2022-07-07 17:39:48.223 INFO  [2103700] [rtc::impl::IceTransport::LogCallback@354] juice: agent.c:1269: STUN server binding successful
2022-07-07 17:39:48.223 INFO  [2103700] [rtc::impl::IceTransport::LogCallback@354] juice: agent.c:1288: Got STUN mapped address 86.124.113.189:49308 from server
2022-07-07 17:39:48.223 INFO  [2103700] [rtc::impl::IceTransport::LogCallback@354] juice: agent.c:2269: Candidate gathering done
2022-07-07 17:39:48.223 INFO  [2103700] [rtc::impl::PeerConnection::changeGatheringState@1167] Changed gathering state to complete
[17:39:48:223214 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103403]: Gathering State change: 2
2022-07-07 17:39:48.223 DEBUG [2103403] [rtc::impl::WsTransport::sendFrame@338] WebSocket sending frame: opcode=1, length=1439
2022-07-07 17:39:50.243 INFO  [2103700] [rtc::impl::IceTransport::LogCallback@354] juice: agent.c:1002: Changing state to connected
2022-07-07 17:39:50.243 INFO  [2103700] [rtc::impl::PeerConnection::initDtlsTransport@242] This connection requires media support
2022-07-07 17:39:50.243 DEBUG [2103700] [rtc::impl::DtlsTransport::DtlsTransport@383] Initializing DTLS transport (OpenSSL)
2022-07-07 17:39:50.244 DEBUG [2103700] [rtc::impl::DtlsSrtpTransport::DtlsSrtpTransport@67] Initializing DTLS-SRTP transport
2022-07-07 17:39:50.244 DEBUG [2103700] [rtc::impl::DtlsTransport::start@472] Starting DTLS recv thread
2022-07-07 17:39:50.244 INFO  [2103700] [rtc::impl::IceTransport::LogCallback@354] juice: agent.c:1002: Changing state to completed
2022-07-07 17:39:50.246 INFO  [2103739] [rtc::impl::DtlsTransport::runRecvLoop@565] DTLS handshake finished
2022-07-07 17:39:50.246 INFO  [2103739] [rtc::impl::DtlsSrtpTransport::postHandshake@266] Deriving SRTP keying material (OpenSSL)
2022-07-07 17:39:50.246 DEBUG [2103739] [rtc::impl::SctpTransport::SctpTransport@188] Initializing SCTP transport
2022-07-07 17:39:50.246 DEBUG [2103739] [rtc::impl::SctpTransport::connect@370] SCTP connecting (local port=5000, remote port=5000)
[17:39:50:247012 +03:00] [RTCClient<stun.l.google.com>] [debug] [thread 2103401]: Video from YlpGit7 opened
[17:39:50:247036 +03:00] [RTCClient<stun.l.google.com>] [debug] [thread 2103401]: Audio from YlpGit7 opened
[17:39:50:247038 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103373]: A/V stream ready
2022-07-07 17:39:50.247 DEBUG [2103400] [rtc::impl::SctpTransport::processNotification@824] SCTP negotiated streams: incoming=1024, outgoing=1024
2022-07-07 17:39:50.247 INFO  [2103400] [rtc::impl::SctpTransport::processNotification@829] SCTP connected
2022-07-07 17:39:50.247 INFO  [2103400] [rtc::impl::PeerConnection::changeState@1149] Changed state to connected
[17:39:50:247508 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103400]: State: 2
[17:39:50:247797 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103402]: DataChannel from YlpGit7 received with label 'test'
[17:39:50:247818 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103402]: DataChannel from YlpGit7 open
[17:44:03:950332 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 26
[17:44:03:950417 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 16
[17:44:03:951089 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 616
[17:44:03:951384 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 657
[17:44:03:951705 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 658
[17:44:03:951790 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 659
[17:44:03:951870 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 629
[17:44:03:952229 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 807
[17:44:03:952322 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 837
[17:44:03:952414 +03:00] [RTCClient<stun.l.google.com>] [trace] [thread 2103739]: Binary message from 'YlpGit7' received, size: 1192

[crisboarna]

Could it be due to the h264packetizer ? As the output image seems "smudged", the packetization is not configured fully properly ?

I did have to change compared to the streamer example the following line:

  auto packetizer = make_shared<rtc::H264RtpPacketizer>(rtc::H264RtpPacketizer::Separator::Length, rtpConfig);

to

  auto packetizer = make_shared<rtc::H264RtpPacketizer>(rtc::H264RtpPacketizer::Separator::LongStartSequence, rtpConfig);

for it to be able to send.

Would something else need modification ?

[paullouisageneau]

By default, tracks are just RTP transceivers, they expect RTP packets as input and output RTP packets. Media handlers may be plugged in, for instance to the RTP packetizer allows to input h264 frames which will be packetized internally. However, the library lacks a depacketizer to do the reverse, so you still get RTP packets as output.

Therefore, you would need to reassemble incoming RTP packets (the most reliable way to do so is using the fact that they have the same timestamp), the removing the RTP headers and glue the payloads together before passing the data to the decoder. You could also pass the incoming RTP string as-is to gstreamer or ffmpeg. Otherwise, the decoder will just get the beginning of the frame and reconstruct the rest by duplicating data, that's what you observe here. For correct performance over non-local networks the implementation would require a proper jitter buffer to reorder packets and pace frames correctly. I'm opening an issue to add such an handler to the library: https://github.com/paullouisageneau/libdatachannel/issues/676

You were right about changing the separator to LongStartSequence, it's actually the most common separator for NAL units.

Also, you don't need to handle string message from tracks, messages will always be binary. The variant is only to expose the same interface as DataChannel and WebSocket, which may transmit string messages.

However, if your use case is actually sending still images, you might want to consider sending compressed images over a data channel, as it will make sure images are properly delivered. Media transport is for real-time video at typically 30+ FPS and does not offer reliability.