Writing (kind of) corrupt video

[towolf]

Hi, your project is pretty interesting.

I built it today to check it out and I’m not sure what I’m supposed to see.

Starting it, it prints

Framerate:  1966080 (1e0000); (30.000000)

So, I assume it is supposed to write "live action" 30fps video? When I play the output mkv files (with mpv, ffplay, or totem), I get either a kind of low framerate "timelapse" with lots of errors, or one or two frames and then nothing (with lots of errors).

The output seems to have zeroed PTS timestamps

And it’s complaining about incomplete units frequently.

I re-encoded a video to generate fresh timestamps and suddenly I got "full motion" video:

ffmpeg  -i file:2015-05-11T14\:12\:44.mkv  -preset superfast -vf "setpts=N/(25*TB)" test.mp4

So, there were all the frames in the file, but the "corrupted" muxing and/or the h.264 stream prevented playing them properly. The player skips almost all of them somehow.

I think it would b good if the camera native timestamps were muxed into the file. And muxing to mp4 would be best, so the files can be played natively straight in the browser.

[towolf]

One minor thing

Your filenames are ISO 8601 with the T separator.

So ffmpeg seeing the filename 2015-05-11T14:12:44.mkv thinks everything before the first : is a protocol 2015-05-11T14:

ISO 8601 with space instead of T would prevent this.

[dickontoo]

I'm glad you like it. The timestamp problems are noted in the README.md; I'm not quite sure what to do about them at present.

mplayer plays the output files fine, albeit whilst complaining bitterly about the SPS and PPS duplicates it comes across (if you're not using the live output, you can fix that problem by commenting out line 927). I'm not sure why ffplay doesn't work; I'd've expected that to playback as fast as possible.

You can change the format. Line 300 onwards. I hadn't considered browser playback, and don't most of them handle Matroska anyway?

Use './' as a prefix to avoid ffmpeg issues. I have a loathing of spaces in filenames, and won't be changing that.

[towolf]

How are you muxing the PTS into the files? OMX has these STC or RAW timestamps. Are they used?

And are my files maybe different from yours? I saw that you are generating some kind of PTS in omxmotion.c, but couldn’t understand how. They are not in my output files however.

You can check with ffprobe -show_frames -of json file.mkv | less

My ffmpeg is the one default on Archlinux ARM

# pacman -Qi ffmpeg
Name           : ffmpeg
Version        : 1:2.6.2-1

[towolf]

The streaming doesn’t work either. I get a green jumbled frame that changes once every 3-5 seconds.

[dickontoo]

I assume there's a PTS being set in the buffer that the encoder produces, and I'm (at least in theory) using that. It doesn't seem to be working, though.

My files also do not play with ffplay. They do with mplayer, which is what I've been using to test with -- do yours?

Jumbled up green frames are good indicators of packet loss. I've not had much luck with it on a Pi 1, although it's been mostly fine up to about 2Mb/s on a Pi 2. Note, as I've said, it's hopeless on wifi -- most devices simply can't do it, for some reason.

[towolf]

You can change the format. Line 300 onwards. I hadn't considered browser playback, and don't most of them handle Matroska anyway?

Browsers play HTML5 video in <video> element or with a direct URL only if H.264 in MP4 container or if VP[89] in webm container. Webm is a stripped down MKV, but MKV proper is not supported.

[towolf]

My files also do not play with ffplay. They do with mplayer, which is what I've been using to test with -- do yours?

I didn’t have mplayer installed, only mpv. It’s a fork to pick up the stagnant mplayer project. After installing mplayer, which links to mplayer2 in Ubuntu, it played, albeit at the wrong speed. I had it set to omxmotion -r 30. What does -r do, anyway? Is it mapping input fps to a different output fps?

Ideally the files should be valid, though. Maybe mplayer (but not the others) overrides the broken PTS.

[dickontoo]

It should set the framerate. By default, 25fps.

[towolf]

It should set the framerate. By default, 25fps.

Yes, which one? The framerate of the camera? Or the framerate of the container?

I think the wrong speed in mplayer is due to it overriding file timestamps entirely. In modern video containers the framerate is often discounted, it’s all about per frame/packet timestamps.

Jumbled up green frames are good indicators of packet loss. I've not had much luck with it on a Pi 1, although it's been mostly fine up to about 2Mb/s on a Pi 2. Note, as I've said, it's hopeless on wifi -- most devices simply can't do it, for some reason.

I’ve been streaming a lot of video over UDP with various protocols. I’ve never seen it this bad. This is on Pi2 with 2mbps too. I suspect the mpegts suffers from the same problems.

[dickontoo]

Both.

And yes, it will. The same code is used to output it. If you can see what's up with the timestamps, please, please, please submit a patch :-)

[towolf]

I can’t code. At least not this kind of stuff.

I just gathered genral info about this stuff here and there.

I think the H.264 encoder does not generate PTS. H.264 bytestream does not have timing information?

I think you need to supply something like OMX_TimestampModeRawStc I guess something equivalent to this commit in another project I follow:

https://github.com/thaytan/gst-rpicamsrc/commit/f708a8ecc4110d2ab28a4f2db2ee56fdf0ec44c0

[dickontoo]

Right. Spot on. OMX_TimestampModeRawStc: I mistakenly assumed that behaviour was what the thing did by default in video mode with no clock. I was wrong.

I've also been abusing ffmpeg for some time, what with never calling avcodec_open2() on a new stream: it's a miracle anything's ever worked, but it explains one or two problems I've had with ffmpeg.

I now have code which, with a suitable build, including libx264 support, produces a multicast transport stream that ffplay seems to like (although again, I suspect it's not doing what it should with the timestamps, and is simply decoding and displaying frames as and when they arrive), but mplayer dislikes (takes a while to sync, and errors a fair bit). Recordings seem to work, but they're transport streams. I tried .mp4, but had issues with the MOOV atom nonsense; TS just works.

Try that.

[towolf]

including libx264 support

Just wondering, why is that needed? We have H264 already?

had issues with the MOOV atom nonsense

Maybe you need this with MOOV muxed at the front? I think they call this variant "faststart". Normally MOOV is at the end, which is pretty silly.

But when ffmpeg is writing the files it should finalize them properly?

Going to try it now.

[towolf]

http://stackoverflow.com/questions/8061798/post-processing-in-ffmpeg-to-move-moov-atom-in-mp4-files-qt-faststart

The thing with mp4 is, it’s the only format that could be written to webserver directory to be played from a browser, as regrettable as it is.

With exec command one could notify via email and play the file from anywhere from any device.

[dickontoo]

libx264 support is needed so that the muxer knows what to do with the extradata, and to set some codec-suitable defaults (no idea what). Or something. It's all a bit poorly-documented.

I'd like to stick the MOOV at the front, but I don't know how to do that programatically. And currently by camera is showing a lot of movement as I haven't got it setup correctly, so it's continuously recording.

With exec commd you could repack the file, and notify via emailand play the file from anywhere from any device... Yeah, not ideal. I'll have a look at it again later in the week.

[towolf]

libx264 support is needed so that the muxer knows what to do with the extradata, and to set some codec-suitable defaults (no idea what). Or something. It's all a bit poorly-documented.

I can imagine. I implemented a gstreamer-rtsp-server in Python with "gobject introspection". The API docs are not really helpful. This kind of OOP is not my thing.

I'd like to stick the MOOV at the front, but I don't know how to do that programatically. And currently by camera is showing a lot of movement as I haven't got it setup correctly, so it's continuously recording.

Apparently you need to set a flag on the mov format encoder. Don’t ask me how, but it’s how it looks like here:

http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavformat/movenc.c;h=16107b159656514d4a94b2bdb6d8f2a024049338;hb=a9553bbb336a7186c62b3b14a0499c1b6ba61ebd#l3167

With exec commd you could repack the file, and notify via emailand play the file from anywhere from any device... Yeah, not ideal. I'll have a look at it again later in the week.

Sure the basics are there to play with. Take your time.

I wonder if writing a Gstreamer module to only handle the motion detection would have been easier.

[rpicamsrc]  → [network setup stuff] → [network streaming (easy)]
             ↓
             [omxmotion as frame "valve"] → [movmux] →[multifilesink]

Only one box would have to be implemented. Cool thing about GStreamer is the modularity.

rpicamsrc would have to be extended to pass the vectors through.

[dickontoo]

     MOVMuxContext *mov = s->priv_data;

concerns me. Not sure I want to go prodding at some perfectly innocent multiplexer's privates.

Thing about this project is, it's basically just a bit of fun. Something to play with as a proof of concept to show it can be done. There are a lot of people on the RPi forums talking about combining motion with the camera module and getting 5fps or so content out of it, but I couldn't help thinking that if you abuse the codec properly you can get much, much more out of it. I think I've proved that.

I'm not sure how in gstreamer it would be possible to implement the pre-trigger ringbuffer, which is rather fundamental to the usefulness of a motion detector.

[towolf]

MOVMuxContext *mov = s->priv_data;

concerns me. Not sure I want to go prodding at some perfectly innocent multiplexer's privates.

Don’t know if that means "no touching" or whatever. There is a stackoverflow answer to suggest just that

Thing about this project is, it's basically just a bit of fun. Something to play with as a proof of concept to show it can be done.

I see. It’s quite promising precisely for the reasons you laid out. Tempting to use on a daily basis. The hacks on the forum are often underwhelming while your method is genius. It achieves the scenario: give me a snippet of full live video whenever something moves and give me the option to stream on demand.

I’ll have to see if I will drop my Gstreamer RTSP server in favour of it.

[spikedrba]

guys, been following along and pretty interested in the developments. Just to share based off of the second to last comment @dickontoo made about the forum being full of ppl struggling with 5fps motion detection - I've been riding along raspimjpeg and we're getting 30fps motion detection. That's in mmal which I understand some ppl don't like and want to use something more standard like omx, but that's where that project is at. They are getting 30fps motion pretty easily, all implemented on the GPU, and the newest frontend also has a nice vector preview showing motion. The next challenge right now is to fix some bugs and improve the detection algo.

Here's a couple links: -https://github.com/roberttidey/userland/blob/master/host_applications/linux/apps/raspicam/RaspiMJPEG.c

• https://github.com/silvanmelchior/RPi_Cam_Web_Interface

long term I'm trying to build a professional sec cam based on FLOSS and the pi so that's where I'm personally headed, I need it because spending $100s for something that only does a fraction of what the pi does with no api or much integration possible isn't right. Happy to chat more about that if there's interest.

[dickontoo]

On Mon, May 11, 2015 at 16:06:51 -0700, Tobias Wolf wrote:

MOVMuxContext *mov = s->priv_data;

concerns me. Not sure I want to go prodding at some perfectly innocent multiplexer's privates.

Don???t know if that means "no touching" or whatever. There a stackoverflow answer to suggest just that

Interesting. I'll have a look at doing it tomorrow. I'm always wary of fiddling with private structures, as they have a tendency to change.

Thing about this project is, it's basically just a bit of fun. Something to play with as a proof of concept to show it can be done.

I see. It???s quite promising precisely for the reasons you laid out. Tempting to use on a daily basis. The hacks on the forum are often underwhelming while your method is genius. It achieves the scenario: give me a snippet of full live video whenever something moves and give me the option to stream on demand.

I intend to get it to a useful state (ie, recordings work, streaming works, get real-time timestamps working with camera ID tags (probably in a subtitle stream); that sort of thing), and see if anyone's interested in it. Fancy camera controls I'll leave to others.

Dickon Hood

Due to unavoidable staff shortages, my .signature is temporarily unavailable. Normal service will be resumed as soon as possible. We apologise for the inconvenience in the meantime.

This email was sent from a colocated server, and needs no excuses.

[dickontoo]

On Mon, May 11, 2015 at 16:12:53 -0700, spikedrba wrote:

guys, been following along and pretty interested in the developments. Just to share based off of the second to last comment @dickontoo made about the forum being full of ppl struggling with 5fps motion detection - I've been riding along raspimjpeg and we're getting 30fps motion detection. That's in mmal which I understand some ppl don't like and want to use something more standard like omx, but that's where that project is at.

My complaint about MMAL is that it's a Broadcom-specific shim atop the (admittedly dreadful) industry standard OpenMAX API. If there was another, portable, API shim around, I'd probably use that. OpenMAX isn't exactly pleasant.

They are getting 30fps motion pretty easily, all implemented on the GPU, and the newest frontend also has a nice vector preview showing motion. The next challenge right now is to fix some bugs and improve the detection algo.

Things have clearly improved since the last time I visited that thread. But still, MJPEG. Not a fantastic format, bit-for-bit: 16Mb/s H.264 is going to look a lot better than 16Mb/s MJPEG, and a format that Crimewatch can take and broadcast has got to be advantageous. You're better off using H.264.

Here's a couple links: -https://github.com/roberttidey/userland/blob/master/host_applications/linux/apps/raspicam/RaspiMJPEG.c

• https://github.com/silvanmelchior/RPi_Cam_Web_Interface

Thanks.

long term I'm trying to build a professional sec cam based on FLOSS and the pi so that's where I'm personally headed, I need it because spending $100s for something that only does a fraction of what the pi does with no api or much integration possible isn't right. Happy to chat more about that if there's interest.

That was partly my initial aim: a Pi-powered open security camera, but I don't think I actually need it any more, so it's more of a plaything than anything else. Fun, though.

Dickon Hood

Due to unavoidable staff shortages, my .signature is temporarily unavailable. Normal service will be resumed as soon as possible. We apologise for the inconvenience in the meantime.

This email was sent from a colocated server, and needs no excuses.

[dickontoo]

OK, so github weren't lying when they said 'Reply to this email directly'.

[spikedrba]

complaint about MMAL noted, but I'm not really good at this stuff and figuring openmax was way over my head and to make the changes I'm interested into it didn't take too long to figure out MMAL (still in the process tho).

regarding MJPEG, I hear you, however there's really nothing that's going to read it on the other side of the connection that's easy on people (ie a browser), cavia maybe getting h264 incapsulated into rtsp, but even then you're going to need a dedicated app. The alternative would be maybe webrtc, it seems to work rather naticely on pi2:

http://pnetherwood.blogspot.com/2015/05/video-streaming-speeds-as-compared-to.html

raspimjpeg will stream in mjpeg but record in h264 and to me that's a good compromise, people ain't going to be watching the stream all the time anyway, it's mostly to check on things if something beeps and mjpeg is ok for that. On the other hand for recordings of motion you do want full resolution and the compact size of h264.

Of course if rtsp of h264 streaming worked I wouldn't say no to it, people are most certainly looking for it in the cam specs as a way to get the feed off of the camera, but the transcoding just seemed to be prohibitive, it happens on the ARM side and that's just too slow. But maybe you've managed better with omx.

[spikedrba]

actually as I was biking home and thinking about what I wrote - for the sec cam I don't think the stream is a must have, but I do have a pi based streaming project that I'd love this for. I don't know if @towolf , who seem very well versed on gstreamer, already has a solution, but as it stands I haven't seen/found a good way to stream from the pi to say youtube. Some ppl have done it with custom builds of gstreamer but they seem to be laggy and can't quite mix in an audio feed from a pi critrus audio card (or ex wolfson). Anyway, don't want to sidetrack the conversation, just wanted to restate what I said and clarify that h264 streaming from the pi with low latency and high fps would be a huge win.

[spikedrba]

and I had forgotten about this: https://github.com/iizukanao/picam

that's actually h264 streaming at 30fps, altho the author claims 1.3s delay at 3MP, but that's still really good, and it does the audio mixing feed, so maybe this problem has been indeed solved already

[spikedrba]

and now that I've paid further attention I realized that's actually using OpenMAXIL as well and not MMAL, so should be to your liking @dickontoo . What love yo hear what you think about that project.

[towolf]

I'm not sure how in gstreamer it would be possible to implement the pre-trigger ringbuffer, which is rather fundamental to the usefulness of a motion detector.

Just for the sake of "FYI" I asked over at gst-rpicamsrc about this and it seems to be entirely possible in the context of GStreamer: https://github.com/thaytan/gst-rpicamsrc/issues/30

Perhaps at some point vector analysis and motion detection will be added to that project, thereby enabling a lego-style module for file writing.

[spikedrba]

@towolf I was sure I saw http://cgit.freedesktop.org/gstreamer/gst-omx/ having some kind of motion vectors stuff, but I can't find it again.

Also I saw in the other thread you mentioned h264 providing the vectors as part of the buffer, but that's not what I'm seeing in MMAL, the "buffer", seems to just be vectors, no picture, ie a different buffer than the one containing the image/frame. Am I missing something obvious? thanks

[towolf]

No, there was a question by @thaytan whether the vectors are baked into the picture and I said that they come separately. In raspivid they are written to file.

In early demos they showed pictures with the arrows overlaid, which is where I think the assumption comes from.

[towolf]

BTW, do you expect the motion detection to work with very low framerates?

When the camera is set to 0 FPS it switches to a variable frame rate mode. In combination with NIGHT exposure mode this can give frame rates from 1.3 FPS up to 30 FPS depending on lighting.

Of course at low FPS any motion is very blurred, but one could use the signal to turn on an additional flood light.

[spikedrba]

@towolf could you elaborate on why/how the vectors come separatedly? if I look at the code all I see is the same enc and ctx objetcts being used to write to the output file and passed to initmotion that has the vector flag enabled: https://github.com/dickontoo/omxmotion/blob/master/omxmotion.c#L952

or @dickontoo maybe you can comment since you wrote it :)

[dickontoo]

Yes, sorry; I've been a bit busy.

You're right in that the code is the same: it is. The trick is at https://github.com/dickontoo/omxmotion/blob/master/omxmotion.c#L1049 where there's a check for the OMX_BUFFERFLAG_CODECSIDEINFO -- that's the flag to say it's a vector buffer rather than coded picture data.

[spikedrba]

@dickontoo thanks for confirming that, very helpful. Out of curiosity, is there a rationale behind when you get back vectors and when you get video data? is that a 1:1 ratio (it would make sense to)? are there differences if the frame associated with the motion is an I-frame or a P-frame?

Also as I'm trying to figure out smarter ways to do motion I'm trying to understand what happens if the analyse takes too long - is that going to result in skipped frames?

@towolf regarding the question on low fps. I'm not sure why you say that low FPS motion is very blurred, supposedly after all the adjustments you get good exposure on those couple frames, meaning it's actually not blurred. Of course with a smart algorithm that compares frame on a temporal scale that's going to be harder as you have fewer information to deal with, but there's only so much we will ever be able to do. Also in my setup (PiNoir and IR light) I found that with 96LEDs I can get a reasonable good shot at 15fps is I adjust brightness and contrast. It's grainy and noisy, but it works well enough to tell if someobdy is there altho I've yet to check out what the motion vectors look like and how much the noise is taken for motion.

[dickontoo]

I haven't checked exactly, but I think I was seeing one vector buffer per P-frame.

Because I wasn't sure how long the analysis was going to take, I stuck the analysis function in a thread of its own. It turns out it's quite quick (at least my current function is; it's only doing around 8K of 16-bit comparisons), and the actual threadedness needs to be in the frame output function. That's next on the list.

I suspect the motion vectors are going to look hideous in low-light conditions. I've seen some encoders treat every speckle of noise as an individual object which needs tracking -- the low-light scenes in most encodings of the film Cloverfield that I've seen do this, for example -- which causes havoc.

[spikedrba]

yeah, I'm not sure how to tackle noise. I've played around with different resolution that supposedly use binning, reducing noise, but I don't really see much of a difference. I think ultimately it goes back to a point I've made elsewhere: we need to look for baselines and clusters of vectors. Noise is going to be all over the place and as a matter of fact this I suspect isn't much different than another problem I'm looking at: rain (or the wind blowing a cloud of dust). Based on my observations while staring into a bunch of matrixes of motion vectors, if a person walks across or something they will cause a connected set of macroblocks to move (MBs). It seems to me that scanning a column of MBs when a person (or other largish mammal if that matters) is moving should result in an obvious pattern over 3 frames. So basically look at each MB until you find one that's above a given threshold T, then start looping on the same column +/- 2 rows looking for another one that's >T. Based on the value of the vectors look at f-1 and see if the pattern holds. Do this for a number of frames and if that number is greater than a frame threshold Tf call it movement. For rain and other things you could calculate every I-frame a global movement coefficient or something like that.

what do you think? That might be slow enough to actually justify having a separated thread like you do now.

[towolf]

Just FYI, slow framerates and slow shutter speeds drastically reduce noise. In night mode it pegs ISO at 100 and extends shutter speed. In normal exposure mode it ramps up ISO first to maintain frame rate at 30 FPS.

That was the origin of my question, whether it could work with low frame rates.

[spikedrba]

@towolf indeed, tested last night and it made a world of a difference. I seem to have noticed however that on auto, even if you lower the fps, it won't make full use of the possible shutter speed, ie it's lower than it could be. For my project I'm thinking of adding a light sensor and just go with manual settings to the best I can and see what happens. I will try to test what happens to motion vectors in low light one of these nights, but even with a clearer image the motion would be so dramatic that they might not maange to pick it up correctly if the object moved very far.