search

JuliaIO / VideoIO.jl

Reading and writing of video files in Julia via ffmpeg
https://juliaio.github.io/VideoIO.jl/stable
Other
128 stars 53 forks source link

WIP: Adding video encoding to file #172

Closed IanButterworth closed 5 years ago

IanButterworth commented 5 years ago

I've nearly got the encode_video.c example converted into examples/encodevideo.jl but get through most of the setting up to line 106 and then:

[libx264 @ 0x7f97d61ac000] The encoder timebase is not set.

signal (11): Segmentation fault: 11
in expression starting at /Users/ian/Documents/GitHub/VideoIO.jl/examples/encodevideo.jl:106
staticfloat commented 5 years ago

What happens if you run it inside of gdb?

IanButterworth commented 5 years ago

@staticfloat I actually just pushed a commit that gets it beyond the frame encoding to the very final freeing functions, but it hangs then crashes julia without error messages

staticfloat commented 5 years ago

Would be good to run that inside of gdb, then when it crashes, get a stack trace to see where it's crashing. If it's within the final freeing functions though, that usually means that there's an out of bounds memory access somewhere; e.g. a structure that you pass to a C routine was actually too small, resulting in the C routing writing past the end of the allocated memory and clobbering something that free() checks on.

IanButterworth commented 5 years ago

Ok. I'm not familiar with gdb but will give it a go.

By the way, the change to mutable structs is breaking the tests. I haven't looked into how to prevent that yet

staticfloat commented 5 years ago

Ok. I'm not familiar with gdb but will give it a go.

Use gdb (or lldb, doesn't matter which) to run your program, usually through something like gdb --args julia script.jl, then when gdb comes up use r to run the program. It will run as normal, then when it crashes, it will dump you to a gdb REPL, and you can use bt to print out a backtrace. Here's some small commands for navigating the callstack within gdb, which may help you to inspect the state of local variables to figure out why you've crashed.

If it's truly a memory corruption issue, I suggest first double-checking everything that gets sent to C to see if there's an easily-discovered struct that's too small. If nothing jumps out at you, perhaps you will need to run it inside of valgrind or a similar tool to determine where the illegitimate memory access actually happens.

kmsquire commented 5 years ago

@ianshmean thanks for taking this on! :heart:

Some thoughts on the current error (without actually looking at the code--I'll try to take a look later):

IanButterworth commented 5 years ago

Thanks for the tips! The timebase error actually was fixed by unsafe_store!-ing the loaded struct pointer back to its source. (I missed that unsafe_load doesn't point to the original memory)

As of the current branch, it gets through writing the frames, and crashes on the free functions. However, even though it gets through writing the frames and closes the video file, the video file is only ~100 bytes.

I've not yet run it with gdb. Hope to this evening.

IanButterworth commented 5 years ago

I've been struggling to get gdb up and running, so not tried that yet. If anyone is able to test this, I just pushed more of a direct copy of the example

IanButterworth commented 5 years ago

I don't know why, but the encode loop seems to be running out of order.. I guess there's some odd memory accessing timing going on

[libx264 @ 0x7ff4e90f3000] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7ff4e90f3000] profile High, level 1.3, 4:2:0, 8-bit
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Write packet 0 (size=809)
21
Write packet 2 (size=17)
22
Write packet 1 (size=14)
23
Write packet 4 (size=21)
24
Write packet 3 (size=14)
Write packet 6 (size=21)
Write packet 5 (size=14)
Write packet 8 (size=20)
Write packet 7 (size=14)
Write packet 9 (size=22)
Write packet 10 (size=109)
Write packet 12 (size=18)
Write packet 11 (size=16)
Write packet 14 (size=22)
Write packet 13 (size=16)
Write packet 16 (size=23)
Write packet 15 (size=16)
Write packet 18 (size=23)
Write packet 17 (size=16)
Write packet 19 (size=24)
Write packet 20 (size=112)
Write packet 22 (size=18)
Write packet 21 (size=16)
Write packet 24 (size=22)
Write packet 23 (size=16)

Julia has exited. Press Enter to start a new session.
IanButterworth commented 5 years ago

The file is now 1KB, which is reassuring, but cannot be run, which makes sense if the write order is messed up

jpsamaroo commented 5 years ago

Does the C variant also write out of order? Maybe it's some weird optimization that ffmpeg does?

IanButterworth commented 5 years ago

I haven't actually tested the C variant, but I just changed the test video to be based on rand(UInt8) and the encode loop order now runs in the right order.

Also, the new rand video file does actually play in VLC, it just needs to be a .h264 file as per this post https://stackoverflow.com/a/53331847/1364192

image

Still crashing on the free functions though

IanButterworth commented 5 years ago

The h.264 stream can be muxed into a mp4 via the ffmpeg executable: run(`$(VideoIO.ffmpeg) -framerate 24 -i video.h264 -c copy output.mp4`) which runs in quicktime (typically a pretty picky media player so seems robust..)

It's tempting to just use that at the end given it's a single line, and the low-level muxing example is rather long.. https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/muxing.c

IanButterworth commented 5 years ago

After some tips from @jpsamaroo and handling an assertion bug with PaddedViews over at https://github.com/JuliaLang/julia/issues/29594 I've now got lldb running the script and have a backtrace.

Although, the issue isn't immediately clear to me from the backtrace..

...
Write packet 239 (size=945)
Write packet 240 (size=981)
Process 62222 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001345e0a9a libavutil.56.dylib`avpriv_slicethread_free + 26
libavutil.56.dylib`avpriv_slicethread_free:
->  0x1345e0a9a <+26>: movq   (%r13), %r15
    0x1345e0a9e <+30>: testq  %r15, %r15
    0x1345e0aa1 <+33>: je     0x1345e0b91               ; <+273>
    0x1345e0aa7 <+39>: movl   0x8(%r15), %eax
Target 0: (julia-debug) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00000001345e0a9a libavutil.56.dylib`avpriv_slicethread_free + 26
    frame #1: 0x0000000131552fbe libavcodec.58.dylib`___lldb_unnamed_symbol4842$$libavcodec.58.dylib + 30
    frame #2: 0x000000013166c163 libavcodec.58.dylib`avcodec_close + 81
    frame #3: 0x000000013150f597 libavcodec.58.dylib`avcodec_free_context + 23
    frame #4: 0x00000001191e873d
    frame #5: 0x000000010011c1c3 libjulia-debug.dylib`jl_apply_generic(F=0x000000010d4edd58, args=0x00007ffeefbfd438, nargs=1) at gf.c:2213:12
    frame #6: 0x00000001001396d0 libjulia-debug.dylib`jl_apply(args=0x00007ffeefbfd430, nargs=2) at julia.h:1624:12
    frame #7: 0x000000010053e5e8 libjulia-debug.dylib`jl_capture_interp_frame(data=0x0000000000000002, sp=140732920748512, fp=140732920747056, space_remaining=2) at interpreter-stacktrace.c:394
    frame #8: 0x000000010053cb36 libjulia-debug.dylib`jl_gc_wb(parent=0x000000010e204610, ptr=0x000000000016e459) at julia.h:761:9
    frame #9: 0x000000010053d67b libjulia-debug.dylib`eval_stmt_value + 43
    frame #10: 0x000000010053bf7f libjulia-debug.dylib`eval_primitivetype(ex=0x00007ffeefbfd850, s=0xb76358a9cb690049) at interpreter.c:173
    frame #11: 0x000000010053c4b6 libjulia-debug.dylib`jl_capture_interp_frame(data=0x00007ffeefbfda48, sp=140732920748616, fp=4516682544, space_remaining=140732920748336) at interpreter-stacktrace.c:404:13
    frame #12: 0x00000001001377bc libjulia-debug.dylib`enter_interpreter_frame + 28
    frame #13: 0x000000010053c561 libjulia-debug.dylib`jl_is_interpreter_frame(ip=4531963408) at interpreter-stacktrace.c:384
    frame #14: 0x0000000100163dab libjulia-debug.dylib`jl_toplevel_eval_flex(m=0x0000000105b98bf0, e=0x000000010e0b9510, fast=1, expanded=1) at toplevel.c:814:18
    frame #15: 0x0000000100125e37 libjulia-debug.dylib`jl_parse_eval_all(fname="/Users/ian/Documents/Leuko/GitHub/VideoIO.jl/examples/encodevideo.jl", content=0x0000000000000000, contentlen=0, inmodule=0x0000000105b98bf0) at ast.c:873:26
    frame #16: 0x0000000100165abc libjulia-debug.dylib`jl_load(module=0x0000000105b98bf0, fname="/Users/ian/Documents/Leuko/GitHub/VideoIO.jl/examples/encodevideo.jl") at toplevel.c:878:12
    frame #17: 0x0000000100165b32 libjulia-debug.dylib`jl_load_(module=0x0000000105b98bf0, str=0x000000010d503af0) at toplevel.c:885:12
    frame #18: 0x0000000105391188 sys-debug.dylib`japi1_include_relative_3501 [inlined] include at boot.jl:328 [opt]
    frame #19: 0x0000000105391178 sys-debug.dylib`japi1_include_relative_3501(#self#=<unavailable>, mod=Module @ 0x0000000105b98bf0, _path=0x0000000000000044) at loading.jl:1094 [opt]
    frame #20: 0x000000010539d107 sys-debug.dylib`japi1_include_3693(#self#=<unavailable>, mod=Module @ 0x0000000105b98bf0, path=0x0000000000000044) at Base.jl:31 [opt]
    frame #21: 0x00000001001136ea libjulia-debug.dylib`jl_fptr_args(f=0x000000010876b890, args=0x00007ffeefbff240, nargs=2, m=0x000000010876b430) at gf.c:1809:12
    frame #22: 0x000000010011ba8c libjulia-debug.dylib`_jl_invoke(F=0x000000010876b890, args=0x00007ffeefbff240, nargs=2, mfunc=0x0000000105db5730, world=26229) at gf.c:2043:31
    frame #23: 0x000000010011c1c3 libjulia-debug.dylib`jl_apply_generic(F=0x000000010876b890, args=0x00007ffeefbff240, nargs=2) at gf.c:2213:12
    frame #24: 0x000000010539c1dc sys-debug.dylib`julia_exec_options_2250(#self#=<unavailable>, opts=<unavailable>) at client.jl:295 [opt]
    frame #25: 0x000000010539ca3e sys-debug.dylib`japi1__start_2214(#self#=<unavailable>) at client.jl:464 [opt]
    frame #26: 0x00000001001136ea libjulia-debug.dylib`jl_fptr_args(f=0x00000001067dc270, args=0x00007ffeefbff748, nargs=0, m=0x000000010755ba20) at gf.c:1809:12
    frame #27: 0x000000010011ba8c libjulia-debug.dylib`_jl_invoke(F=0x00000001067dc270, args=0x00007ffeefbff748, nargs=0, mfunc=0x00000001067dbf40, world=26229) at gf.c:2043:31
    frame #28: 0x000000010011c1c3 libjulia-debug.dylib`jl_apply_generic(F=0x00000001067dc270, args=0x00007ffeefbff748, nargs=0) at gf.c:2213:12
    frame #29: 0x00000001000017b0 julia-debug`jl_apply(args=0x00007ffeefbff740, nargs=1) at julia.h:1624:12
    frame #30: 0x0000000100001343 julia-debug`true_main(argc=1, argv=0x00007ffeefbff990) at repl.c:96:13
    frame #31: 0x00000001000011d2 julia-debug`main(argc=1, argv=0x00007ffeefbff990) at repl.c:217:15
    frame #32: 0x00007fff6035d3d5 libdyld.dylib`start + 1
    frame #33: 0x00007fff6035d3d5 libdyld.dylib`start + 1
(lldb)
staticfloat commented 5 years ago

EXC_BAD_ACCESS means you're reaching into a page that you shouldn't be (often signifies a use-after-free, or a pointer's value is corrupt (e.g. gibberish or NULL or something like that).

As best I can tell, this is crashing because *pctx is pointing to a non-NULL but invalid memory location. Reading through the source code for this function, it's pretty straightforward, but of course everything would be easier if we had a debug build of the libraries available (so as to have more useful source listings).

kmsquire commented 5 years ago

By the way, the change to mutable structs is breaking the tests. I haven't looked into how to prevent that yet

Some context:

When I started wrapping ffmpeg, Julia types were not layout compatible with C types, but immutable types were (I don't know why).

The problem with immutable types was that they were, well, immutable. But if they were in memory, then modifying them was possible, in theory.

Now, you really aren't supposed to do this, because since they were immutable, the compiler might do funky things with them (like leave them in registers, and not put them in memory at all).

That said, most of the types being wrapped in FFMpeg were large, so there was little chance that they would not be in memory.

So, everything was made immutable, so it could be loaded into Julia types easily, and if I needed to modify anything, I got a pointer to the original struct in memory and modified it with av_setfield (defined in utils.jl).

Tada! (Yes, it was a really ugly hack... yuck!)

None of these shenanigans should be necessary now (Julia structs should be mostly layout compatible with C structs), but I've never taken the time to go back and fix things (sorry!).

This is most likely why the tests are failing.

So, to go forward, we need to either

  1. change the types back to immutable and use av_setfield to modify the values in memory, or
  2. change most/all wrapped C structs to mutable structs, and remove all other uses of av_setfield

1 is easier in the short run, 2 is the right thing to do.

I can help with 2, but I'm not sure when I'll get to it.

kmsquire commented 5 years ago

Note that it also should be possible to use Ref everywhere a Vector is used to store single pointer references.

(Again, when this was first written, Ref didn't exist, and I never went back and changed things.)

IanButterworth commented 5 years ago

@kmsquire After your suggestions this now works! thanks!

I do have to say though,

  1. It outputs a .h264 file, which still needs to be mux-ed into a video container. We could just do run(`$(VideoIO.ffmpeg) -framerate 24 -i video.h264 -c copy output.mp4`) for now.
  2. Looking at the .h264 file in vlc, the output video quality is very poor and glitchy.. and playback time immediately jumps to 75% through, but I'm guessing that's more of a non-muxed stream and ffmpeg encode settings thing, than code implementation thing.

As for the struct refactoring you mentioned, I was also looking at the Clang wrap script and wondering whether that should also be updated to the new Clang functionality? Perhaps if we're doing 2) we should also refresh the clang wrapper?

kmsquire commented 5 years ago

As for the struct refactoring you mentioned, I was also looking at the Clang wrap script and wondering whether that should also be updated to the new Clang functionality? Perhaps if we're doing 2) we should also refresh the clang wrapper?

Yes, that would be good. (In a separate PR, of course.)

kmsquire commented 5 years ago
  1. It outputs a .h264 file, which still needs to be mux-ed into a video container. We could just do run(`$(VideoIO.ffmpeg) -framerate 24 -i video.h264 -c copy output.mp4`) for now.

I think that would be fine for this pass.

  1. Looking at the .h264 file in vlc, the output video quality is very poor and glitchy.. and playback time immediately jumps to 75% through, but I'm guessing that's more of a non-muxed stream and ffmpeg encode settings thing, than code implementation thing.

I agree, it's probably due to the settings. It would be nice to find some nicer settings, but if you don't have time to explore, even getting this in as is would be useful.

IanButterworth commented 5 years ago

The encoding is now added as functionality, available both as a high-level encoder for an image stack, and lower-level functions for incorporating into custom loops

imgstack = map(x->rand(UInt8,2048,1536),1:100)
100-element Array{Array{UInt8,2},1}

Encode entire imgstack in one go

using VideoIO
props = [:priv_data => ("crf"=>"22","preset"=>"medium")]
encodevideo("video.mp4",imgstack,framerate=30,AVCodecContextProperties=props)

[ Info: Video file saved: /Users/ian/Documents/video.mp4
[ Info: frame=  100 fps=0.0 q=-1.0 Lsize=  129867kB time=00:00:03.23 bitrate=329035.1kbits/s speed=8.17x    
[ Info: video:129865kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.001692%

Encode by appending within a loop

using VideoIO, ProgressMeter
filename = "manual.mp4"
props = [:priv_data => ("crf"=>"22","preset"=>"medium")]
codec_name = "libx264"
framerate = 24

encoder = prepareencoder(imgstack[1],codec_name,framerate,props)

io = Base.open("temp.stream","w")
p = Progress(length(imgstack), 1)
index = 1
for img in imgstack
    global index
    appendencode!(encoder, io, img, index)
    next!(p)
    index += 1
end

finishencode!(encoder, io)
close(io)

mux("temp.stream",filename,framerate) #Multiplexes the stream into a video container

[ Info: Video file saved: /Users/ian/Documents/manual.mp4
[ Info: frame=  100 fps=0.0 q=-1.0 Lsize=  134451kB time=00:00:04.04 bitrate=272513.6kbits/s speed=9.88x    
[ Info: video:134448kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.001634%

The AVCodecContextProperties object allows control of the majority of settings required.

A few helpful presets for h264:

TODO

Handle RGB conversion given H264 can't store full RGB frames (and doesn't support AV_PIX_FMT_RGB24). Options:

IanButterworth commented 5 years ago

To fix tests and get this merged, I think I'm going to change mutable structs back to immutable, and use av_setfield within the encoder functions. The alternative seems like a lot more work (updating the clang wrapper, making structs mutable, changing the rest of the code) and I think you're right @kmsquire about the av_setfield method.. while it's not ideal with modern julia functionality, it's low risk here

kmsquire commented 5 years ago

Great job getting all of this working! The package name finally won't be a lie anymore! 🙃 Thank you!

timholy commented 5 years ago

Seconded. It will be huge to have such improvements (this PR and the rest) to the video support in Julia. Thanks to you and @kmsquire for seeing this through!

IanButterworth commented 5 years ago

Happy to help! 🎥🎉

I've added RGB{N0f8} support. Same example as above, just pass a RGB{N0f8} image in.

If lossless RGB is wanted, true RGB lossless requires using libx264rgb, to avoid the lossy YUV420 conversion. That's achieved with codec_name="libx264rgb" and "crf" => "0" in the above example, but that should only really be used for data storage purposes... Not even VLC can handle playing back libx264rgb smoothly. I also added warnings for cases where the crf & codec_name settings seem illogical for RGB lossless or lossy outputs

IanButterworth commented 5 years ago

@kmsquire I tried converting to the av_pointer_to_field approach but got stuck here. How would you handle writing to the immutable frame struct data arrays? https://github.com/JuliaIO/VideoIO.jl/blob/41063bfaca92e8c69da4dc84fc7cfe760ae5cc0a/src/encoding.jl#L148-L178

IanButterworth commented 5 years ago

NEARLY there..

IanButterworth commented 5 years ago

Also, ffmpeg is throwing very infrequent non-fatal error messages out, like: [h264 @ 0x7f8a5fcc7c00] Invalid NAL unit 8, skipping. It would be good to either fix it, or silence it

IanButterworth commented 5 years ago

I've been trying and failing to build a linux ffmpeg binary that provides the libx264rgb codec. If anyone has any ideas, this is the latest build script https://github.com/ianshmean/FFMPEGBuilder/blob/ad00f98ecc016c209390da7b93107f689106ac14/build_tarballs.jl#L16-L45

IanButterworth commented 5 years ago

I think I've run out of ideas for getting ffmpeg built on linux with -libx264rgb. This thread suggested some good ideas but didn't fix it.

Any ideas @SimonDanisch @jpsamaroo ?

https://github.com/ianshmean/FFMPEGBuilder/blob/master/build_tarballs.jl

kmsquire commented 5 years ago

I curious why you need the libx264rgb codec? If vlc doesn't support it by default, then it's obviously pretty rare, and most cameras only generate data in yuv format. If you're getting data in rgb, it's most likely because it was converted from yuv before it was presented to you.

IanButterworth commented 5 years ago

I have a particular interest in being able to store lossless grayscale and true RGB with h264's high performance compression, given I use scientific cameras that provide full range gray and rgb. Unfortunately from what I can tell from testing, GRAY8 clamps the 0-255 color range to 16-235, and there's no standard h264 RGB full plane mode.

libx264rgb is actually provided in the windows and MacOS builds we use from https://ffmpeg.zeranoe.com/builds/ and it's not clear why that happens from their make config.

I guess I could relax the testing, and just test for YUV420 encoding losslessness, which should pass on all.

IanButterworth commented 5 years ago

All tests are passing!! 🎉That's previous tests plus encode tests for gray & RGB.

Thanks to @staticfloat for catching my error in the linux ffmpeg build config.. relative paths for cflags when absolute paths were needed! linux now supports libx246rgb

@kmsquire shall I merge? I'm thinking of squash-merging.. a single commit is probably sufficient here

timholy commented 5 years ago

I have a particular interest in being able to store lossless grayscale and true RGB with h264's high performance compression, given I use scientific cameras that provide full range gray and rgb.

We do that with grayscale. What's your experience here, is it truly lossless and what kind of compression ratio do you expect? I frequently wonder about the merits of compression vs. the ease of mmap when accessing the data.

IanButterworth commented 5 years ago

@timholy I've written up a lossless grayscale set of tests and added it to #174 (where I've fixed gray8 lossless encoding, but gray8 is oddly missing from h264 on the linux build..)

Each setting is tested 3 times (to provide mean and std).

Note that all videos passed the identical check (lossless w.r.t. source imgstack). Also, odd (but understandably odd) results with the random noise example. No variation in file size across repeats, as to be expected.

ffmpeg test video, 150 frames of 1280x720 (testsrc2 example)

│ Row │ preset    │ identical │ fps_mean │ fps_std │ filesize_perc_mean │ filesize_perc_std │
│     │ Any       │ Bool      │ Float64  │ Float64 │ Float64            │ Float64           │
├─────┼───────────┼───────────┼──────────┼─────────┼────────────────────┼───────────────────┤
│ 1   │ ultrafast │ true      │ 304.728  │ 9.58154 │ 100.0              │ 0.0               │
│ 2   │ superfast │ true      │ 255.114  │ 10.716  │ 75.3965            │ 0.0               │
│ 3   │ veryfast  │ true      │ 199.211  │ 52.4159 │ 74.9182            │ 0.0               │
│ 4   │ faster    │ true      │ 230.773  │ 1.19191 │ 74.9975            │ 0.0               │
│ 5   │ fast      │ true      │ 243.758  │ 4.43879 │ 71.6298            │ 0.0               │
│ 6   │ medium    │ true      │ 214.879  │ 7.62118 │ 70.9487            │ 0.0               │
│ 7   │ slow      │ true      │ 166.16   │ 2.05397 │ 69.5094            │ 0.0               │
│ 8   │ slower    │ true      │ 106.76   │ 1.50004 │ 67.8143            │ 0.0               │
│ 9   │ veryslow  │ true      │ 61.9854  │ 3.38506 │ 67.0681            │ 0.0               │

ladybird.mp4, 396 frames of 1080x1920 (from test videos)

│ Row │ preset    │ identical │ fps_mean │ fps_std  │ filesize_perc_mean │ filesize_perc_std │
│     │ Any       │ Bool      │ Float64  │ Float64  │ Float64            │ Float64           │
├─────┼───────────┼───────────┼──────────┼──────────┼────────────────────┼───────────────────┤
│ 1   │ ultrafast │ true      │ 184.186  │ 38.182   │ 100.0              │ 0.0               │
│ 2   │ superfast │ true      │ 137.663  │ 4.6292   │ 84.6588            │ 0.0               │
│ 3   │ veryfast  │ true      │ 110.791  │ 7.99061  │ 83.3687            │ 0.0               │
│ 4   │ faster    │ true      │ 100.058  │ 0.535927 │ 80.5937            │ 0.0               │
│ 5   │ fast      │ true      │ 70.0545  │ 1.19685  │ 78.7228            │ 0.0               │
│ 6   │ medium    │ true      │ 55.1113  │ 4.30227  │ 77.7667            │ 0.0               │
│ 7   │ slow      │ true      │ 37.3717  │ 0.901799 │ 76.3431            │ 0.0               │
│ 8   │ slower    │ true      │ 19.7098  │ 0.165271 │ 75.6813            │ 0.0               │
│ 9   │ veryslow  │ true      │ 9.98514  │ 0.211713 │ 75.638             │ 0.0               │

random noise, 1000 frames of 1280x720

│ Row │ preset    │ identical │ fps_mean │ fps_std  │ filesize_perc_mean │ filesize_perc_std │
│     │ Any       │ Bool      │ Float64  │ Float64  │ Float64            │ Float64           │
├─────┼───────────┼───────────┼──────────┼──────────┼────────────────────┼───────────────────┤
│ 1   │ ultrafast │ true      │ 97.9357  │ 6.64778  │ 100.0              │ 0.0               │
│ 2   │ superfast │ true      │ 58.8997  │ 1.98085  │ 92.0584            │ 0.0               │
│ 3   │ veryfast  │ true      │ 62.149   │ 0.257865 │ 92.0584            │ 0.0               │
│ 4   │ faster    │ true      │ 61.4085  │ 1.86261  │ 92.0584            │ 0.0               │
│ 5   │ fast      │ true      │ 150.466  │ 0.465406 │ 64.4206            │ 0.0               │
│ 6   │ medium    │ true      │ 144.832  │ 3.72393  │ 64.4206            │ 0.0               │
│ 7   │ slow      │ true      │ 137.637  │ 5.64154  │ 64.4206            │ 0.0               │
│ 8   │ slower    │ true      │ 142.924  │ 0.432807 │ 64.4206            │ 0.0               │
│ 9   │ veryslow  │ true      │ 137.805  │ 2.87539  │ 64.4206            │ 0.0               │
timholy commented 5 years ago

Just to make sure I understand, the file size is typically 2/3 that of the raw frames?

IanButterworth commented 5 years ago

Actually, I was just comparing the ultrafast file size onwards.. what you ask is a more appropriate metric.. comparing to the raw data in the image stack. Here filesize_perc = filesize("encodedvideo.mp4")/(img_width*img_height*imgstack_length)

It makes a lot of sense for each of the videos, and reassuring that noise is ~100% when it tries hard.

ffmpeg test video, 150 frames of 1280x720 (testsrc2 example)

│ Row │ preset    │ identical │ fps_mean │ fps_std │ filesize_perc_mean │ filesize_perc_std │
│     │ Any       │ Bool      │ Float64  │ Float64 │ Float64            │ Float64           │
├─────┼───────────┼───────────┼──────────┼─────────┼────────────────────┼───────────────────┤
│ 1   │ ultrafast │ true      │ 228.166  │ 75.1439 │ 4.80392            │ 0.0               │
│ 2   │ superfast │ true      │ 239.73   │ 54.2033 │ 3.62199            │ 0.0               │
│ 3   │ veryfast  │ true      │ 197.506  │ 13.1121 │ 3.59901            │ 0.0               │
│ 4   │ faster    │ true      │ 174.174  │ 18.0316 │ 3.60282            │ 0.0               │
│ 5   │ fast      │ true      │ 235.181  │ 7.40358 │ 3.44104            │ 0.0               │
│ 6   │ medium    │ true      │ 219.654  │ 3.27445 │ 3.40832            │ 0.0               │
│ 7   │ slow      │ true      │ 171.337  │ 3.92415 │ 3.33917            │ 0.0               │
│ 8   │ slower    │ true      │ 105.24   │ 6.59151 │ 3.25774            │ 5.43896e-16       │
│ 9   │ veryslow  │ true      │ 63.1136  │ 2.47291 │ 3.2219             │ 0.0               │

ladybird.mp4, 396 frames of 1080x1920 (from test videos)

│ Row │ preset    │ identical │ fps_mean │ fps_std  │ filesize_perc_mean │ filesize_perc_std │
│     │ Any       │ Bool      │ Float64  │ Float64  │ Float64            │ Float64           │
├─────┼───────────┼───────────┼──────────┼──────────┼────────────────────┼───────────────────┤
│ 1   │ ultrafast │ true      │ 176.787  │ 36.5227  │ 12.2293            │ 0.0               │
│ 2   │ superfast │ true      │ 135.925  │ 7.04431  │ 10.3532            │ 0.0               │
│ 3   │ veryfast  │ true      │ 117.115  │ 1.28102  │ 10.1954            │ 0.0               │
│ 4   │ faster    │ true      │ 94.39    │ 3.48494  │ 9.85604            │ 0.0               │
│ 5   │ fast      │ true      │ 69.657   │ 1.61004  │ 9.62724            │ 0.0               │
│ 6   │ medium    │ true      │ 54.9621  │ 0.568074 │ 9.51032            │ 0.0               │
│ 7   │ slow      │ true      │ 37.8888  │ 1.27484  │ 9.33622            │ 0.0               │
│ 8   │ slower    │ true      │ 20.1112  │ 1.04282  │ 9.25529            │ 0.0               │
│ 9   │ veryslow  │ true      │ 10.0016  │ 0.473213 │ 9.24999            │ 0.0               │

random noise, 1000 frames of 1280x720

│ Row │ preset    │ identical │ fps_mean │ fps_std │ filesize_perc_mean │ filesize_perc_std │
│     │ Any       │ Bool      │ Float64  │ Float64 │ Float64            │ Float64           │
├─────┼───────────┼───────────┼──────────┼─────────┼────────────────────┼───────────────────┤
│ 1   │ ultrafast │ true      │ 92.5769  │ 8.40224 │ 156.444            │ 0.0               │
│ 2   │ superfast │ true      │ 62.3509  │ 1.19652 │ 144.019            │ 0.0               │
│ 3   │ veryfast  │ true      │ 59.9182  │ 1.77294 │ 144.019            │ 0.0               │
│ 4   │ faster    │ true      │ 60.3482  │ 2.32679 │ 144.02             │ 0.0               │
│ 5   │ fast      │ true      │ 149.169  │ 1.56068 │ 100.784            │ 0.0               │
│ 6   │ medium    │ true      │ 146.141  │ 3.41282 │ 100.784            │ 0.0               │
│ 7   │ slow      │ true      │ 147.214  │ 1.23929 │ 100.784            │ 0.0               │
│ 8   │ slower    │ true      │ 138.808  │ 2.553   │ 100.784            │ 0.0               │
│ 9   │ veryslow  │ true      │ 132.505  │ 3.28558 │ 100.784            │ 0.0               │