How to write out H264 data in a valid way?

[darkvertex]

Hello, I'm on Linux CentOS 7 x64. I have a Zcam V1 Pro.

I'm not a C++ expert so this may not be the best way, but I've modified the libssp_test.cpp file a little bit to:

• dump h264 data to a file called "output.h264"
• catch ctrl+c and close the file.
• assume zcam IP from the "ZCAM_IP" env variable
• print debug prints to stderr instead of stdout
• only print debug prints when "DEBUG" env variable is set.

Here is my code in full:

#define _GLIBCXX_USE_CXX11_ABI 1
#include <functional>
#include <memory>
#include <thread>
#include <string>

#include <stdlib.h>

#include "imf/net/loop.h"
#include "imf/net/threadloop.h"
#include "imf/ssp/sspclient.h"

using namespace std::placeholders;

#ifdef _DEBUG
#pragma comment (lib, "libsspd.lib")
#else
#pragma comment (lib, "libssp.lib")
#endif

static FILE* output_file;

static void on_264_1(struct imf::SspH264Data * h264)
{
    if (std::getenv("DEBUG"))
        fprintf(stderr, "on 1 264 [frm_no: %i] [pts: %lld] [type: %d] [ntp_timestamp: %lld] [data length: %d]\n", h264->frm_no, h264->pts, h264->type, h264->ntp_timestamp, h264->len);

    fwrite(h264->data, 1, h264->len, output_file);
}

static void on_264_2(struct imf::SspH264Data * h264)
{
    fprintf(stderr, "on 2 264 [frm_no: %i] [pts: %lld] [type: %d] [ntp_timestamp: %lld] [data length: %d]\n", h264->frm_no, h264->pts, h264->type, h264->ntp_timestamp, h264->len);
}

static void on_audio_data_1(struct imf::SspAudioData * audio)
{
    if (std::getenv("DEBUG"))
        fprintf(stderr, "on audio data 1 [pts: %lld] [ntp_timestamp: %lld] [data length: %d]\n", audio->pts, audio->ntp_timestamp, audio->len);
}

static void on_meta_1(struct imf::SspVideoMeta *v, struct imf::SspAudioMeta *a, struct imf::SspMeta * m)
{
    fprintf(stderr, "on meta 1 wall clock %d", m->pts_is_wall_clock);
    fprintf(stderr, "              video %dx%d %d/%d\n", v->width, v->height, v->unit, v->timescale);
    fprintf(stderr, "              audio %d\n", a->sample_rate);
}

static void on_disconnect()
{
    fclose(output_file);
    fprintf(stderr, "on disconnect\n");
}

void signal_handler(int s)
{
    printf("Caught signal %d\n", s);
    fclose(output_file);
    exit(1);
}

static void setup(imf::Loop *loop)
{
    signal(SIGINT, signal_handler);

    output_file = fopen("output.h264", "w+b");
    std::string ip = std::getenv("ZCAM_IP");
    imf::SspClient * client = new imf::SspClient(ip, loop, 0x400000);
    client->init();

    client->setOnH264DataCallback(std::bind(on_264_1, _1));
    client->setOnMetaCallback(std::bind(on_meta_1, _1, _2, _3));
    client->setOnDisconnectedCallback(std::bind(on_disconnect));
    client->setOnAudioDataCallback(std::bind(on_audio_data_1, _1));

    client->start();

}

int main(int argc, char ** argv)
{
    std::unique_ptr<imf::ThreadLoop> threadLooper(new imf::ThreadLoop(std::bind(setup, _1)));
    threadLooper->start();

    while (1) {
        std::this_thread::sleep_for(std::chrono::seconds(1));
    }

    threadLooper->stop();
    return 0;
}

---

I can compile it and it runs and writes my file. I have a few questions though:

1) What sets the encoding settings? Is it using the settings for video mode on the zcam?

2) In your opinion, am I dumping the data to disk in a valid way? (fwrite(h264->data, 1, h264->len, output_file); = OK?)

2) The written file is supposed to be a raw H264 stream, correct?

I tried calling ffmpeg's ffprobe utility on it and it seems to hint that it may be invalid/corrupt:

$ ffprobe output.h264 
ffprobe version 3.1 Copyright © 2007-2016 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-4)
  configuration: --prefix=/Volumes/fpsan1-nas/fps/people/greg/ffmpeg-libs --enable-static --extra-cflags='-I/Volumes/fpsan1-nas/fps/people/greg/ffmpeg-libs/include -fopenmp -static' --extra-ldflags=-fopenmp --pkg-config-flags=--static --prefix=/Volumes/fpsan1-nas/fps/people/greg/ffmpeg-libs --enable-libfdk-aac --enable-gpl --enable-version3 --disable-shared --disable-debug --enable-runtime-cpudetect --enable-libmp3lame --enable-libx264 --enable-libx265 --enable-libwebp --enable-libspeex --enable-libvorbis --enable-libvpx --enable-libfreetype --enable-fontconfig --enable-libxvid --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvo-amrwbenc --enable-gray --enable-libopenjpeg --enable-libopus --enable-libass --enable-gnutls --enable-libvidstab --enable-libsoxr --enable-frei0r --enable-libfribidi --disable-indev=sndio --disable-outdev=sndio --enable-librtmp --cc=gcc --enable-nonfree
  libavutil      55. 27.100 / 55. 27.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 40.101 / 57. 40.101
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 46.102 /  6. 46.102
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
[h264 @ 0x388bac0] Format h264 detected only with low score of 1, misdetection possible!
[h264 @ 0x388cfa0] non-existing PPS 0 referenced
[h264 @ 0x388cfa0] Invalid NAL unit 0, skipping.
    Last message repeated 6 times
[h264 @ 0x388cfa0] slice type 32 too large at -1
[h264 @ 0x388cfa0] decode_slice_header error
[h264 @ 0x388cfa0] no frame!
[h264 @ 0x388cfa0] missing picture in access unit with size 98288
[h264 @ 0x388cfa0] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x388cfa0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[h264 @ 0x388cfa0] no frame!
[h264 @ 0x388cfa0] sps_id 1 out of range
[h264 @ 0x388cfa0] non-existing PPS 0 referenced
[h264 @ 0x388cfa0] Invalid NAL unit 2, skipping.
[h264 @ 0x388cfa0] Invalid NAL unit 0, skipping.
    Last message repeated 4 times
[h264 @ 0x388cfa0] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x388cfa0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[h264 @ 0x388cfa0] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x388cfa0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[h264 @ 0x388cfa0] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x388cfa0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[h264 @ 0x388cfa0] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x388cfa0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[h264 @ 0x388cfa0] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x388cfa0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[h264 @ 0x388cfa0] no frame!
[h264 @ 0x388bac0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[h264 @ 0x388bac0] decoding for stream 0 failed
[h264 @ 0x388bac0] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, h264, from 'output.h264':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264, none, 25 fps, 25 tbr, 1200k tbn, 50 tbc

Likewise, ffmpeg fails to encode me an mp4 from it, so something is wrong... but I don't understand where.

Any advice?? :/

[darkvertex]

Hmmm... if I run strings on the outputted file it's full of this:

$ strings output.h264
Us^d
[m[connector]   Connector ip:192.168.55.129, port:9999
Us^d
[m[connector]   Connector ip:192.168.55.129, port:9999
Us^d
[m[connector]   Connector ip:192.168.55.129, port:9999
Us^d
[m[connector]   Connector ip:192.168.55.129, port:9999
Us^d
[m[connector]   Connector ip:192.168.55.129, port:9999
Us^d
[m[connector]   Connector ip:192.168.55.129, port:9999
Us^d
[m[connector]   Connector ip:192.168.55.129, port:9999
Us^d
[m[connector]   Connector ip:192.168.55.129, port:9999

Should that be in the H264 raw data?? :thinking:

[secit]

I copy/pasted your code and it works perfectly fine here.

On Ubuntu here so I'm on commit 051f1a3. Commits after this doesn't compile for Ubuntu... 🤔

[secit]

I just managed to compile the latest in Ubuntu with g++ 4.8.5. Now I get the same results as you. h264->len seems a bit off to me.

Probably best to use 051f1a3 until this has been fixed.

[secit]

I was able to debug the application and found a temporary solution.

It seems that the sspclient.h is not up to date with the library in the latest commit. The data has been shifted in some of the structs.

In sspclient.h.

Change SspVideoMeta to this:

struct LIBSSP_API SspVideoMeta {
    uint32_t unknowndata[6]; // This is the fix
    uint32_t width;
    uint32_t height;
    uint32_t timescale;
    uint32_t unit;
    uint32_t gop;
    uint32_t encoder;
};

and change SspH264Data to this:

struct LIBSSP_API SspH264Data {
    uint16_t unknowndata; // This is the fix
    uint8_t * data;
    size_t len;
    uint64_t pts;
    uint64_t ntp_timestamp;
    uint32_t frm_no;
    uint32_t type;          // I or P
};

Maybe @ailike can submit a correct version of sspclient.h?

[darkvertex]

@secit Your fix made me get a real stream. Thank you! (It may be broken in other ways, but ffplay can play the frames now so that's cool.)

I tried code at commit 051f1a3 and I straight up cannot compile on my CentOS7 with g++ 4.8.5:

$ make all
g++ ../libssp_test.cpp --std=c++11 -L../../lib/linux_x64/ -lssp -lpthread -I../../include/ -I../../include/libuv/include/ -o libssp_test
../../lib/linux_x64//libssp.a(sspclient.o): In function `imf::SspClientImp::SspClientImp(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, imf::Loop*, unsigned long, unsigned short, unsigned int)':
sspclient.cpp:(.text+0x7f5): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)'
../../lib/linux_x64//libssp.a(connector.o): In function `imf::Connector::Connector(imf::Loop*, char const*, unsigned short)':
connector.cpp:(.text+0x3ea): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)'
/tmp/cc8XM10J.o: In function `setup(imf::Loop*)':
libssp_test.cpp:(.text+0x3b2): undefined reference to `imf::SspClient::SspClient(std::string const&, imf::Loop*, unsigned long, unsigned short, unsigned int)'
collect2: error: ld returned 1 exit status

[secit]

@darkvertex That's great!

I think you can "safely" use the latest commit with the fix. That should compile with g++ 4.8.5. The frames should be safe to use anyway. Not so sure about the other data. I didn't check audio for instance.

With the commit 051f1a3 I actually used g++ 7.4.0, but with other compile options.

[darkvertex]

@secit By the way, do you have any trouble with disconnections after some period of time?

I was testing with an E2 model, writing to a named pipe file (mkfifo to make the file and edited the file open mode in my C sample code to "a" to append) and I could have ffmpeg's ffplay player play the file direct in real-time (which is great) but for some reason I was getting consistently disconnected about 43 seconds in.

Have you experienced any behavior like that? (What model do you test with?)

I'm going to try playing with encoding settings to see if it changes anything.

[secit]

@darkvertex E2 camera with default settings that are connected for several days without any issues.

For realtime video I use libav to decode and present images. I also remux and transport an RTP stream at the same time so I can view it realtime on another computer as well.

You could also write raw H264 to stdout and pipe it to ffplay.

Good luck!

[darkvertex]

So I found out that when using named pipes I get disconnected after some time (under a minute) and with real files I don't.

After some reading, I learnt that sending EOF to a pipe effectively closes it everywhere, so I'm guessing that at some point the camera closes the file and opens a new one, even though it's streaming non-stop it would seem it sends the EOF through the buffer anyway.

By the way, does someone know what's the point of the on_264_2 callback? why "2"?