AVFoundation export in QtApp

[ilia3101]

Pretty easy to implement the library, which is completed now (except audio)

The files that need to be compiled/linked are:

avf_lib.m

And headers included:

../cocoa/avf_lib/avf_lib.h

Here's how it's implemented in Cocoa App:

    AVEncoder_t * encoder = initAVEncoder( getMlvWidth(App->videoMLV),
                                           getMlvHeight(App->videoMLV),
                                           AVF_CODEC_PRORES_422,
                                           AVF_COLOURSPACE_SRGB,
                                           getMlvFramerate(App->videoMLV) );

    beginWritingVideoFile(encoder, exportPath);

    for (uint64_t f = 0; f < getMlvFrames(App->videoMLV); ++f)
    {
        getMlvProcessedFrame16(App->videoMLV, f, App->rawImage);
        addFrameToVideoFile(encoder, App->rawImage);
    }

    endWritingVideoFile(encoder);
    freeAVEncoder(encoder);

Notes:

• only use AVF_COLOURSPACE_SRGB - giving an option to the user right now would make no sense, as the MLV App library currently only outputs in one space, and we currently treat is as sRGB all round.
• There is a few macros like AVF_CODEC_PRORES_422, I've just put it in the sample to avoid foreign variable names.

[masc4ii]

Thank you! I'll give it a try asap! :-)

[bouncyball-git]

@ilia3101: Congrats on new release!!! Great progress! Perhaps you will have more time for processing stuff after all ;)

[masc4ii]

Will there be a way to chose between Prores 422 Proxy, LT, ST & HQ?

[masc4ii]

I get it compiled, output works. But I have some problems - for the cocoa and qt version:

• 422: fine, all is okay
• 4444: there is a file, but nearly no program can open it, also Apple Apps can't (all is black)
• H264: I get an error for each frame, file is 0KB in the end

Speed is okay, more or less the same as ffmpeg anatolyi, but much faster than ffmpeg kostya. Anyway, great job Ilia, it was really easy to add it to the Qt version.

Edit @bouncyball-git : could you please check, if it compiles on non-OSX?

[bouncyball-git]

This error stops compilation process:

:-1: error: error trying to exec 'cc1obj': execvp: No such file or directory

[masc4ii]

Okay. Please try again.

[bouncyball-git]

Now it's ok :)

[ilia3101]

H264: I get an error for each frame, file is 0KB in the end

Hmmmmmm, I think this could be down to it using 16 bit buffers (I made that change and didn't check if h264 still worked). I'll add conversion to 8 bit for h.264.

[ilia3101]

@ilia3101: Congrats on new release!!! Great progress! Perhaps you will have more time for 
processing stuff after all ;)

The release is mainly so no one has to use the previous bugged out version... a surprising amount of people seemed to download it despite the nicely working Qt version available. Maybe I'll make the download page be mixed so there's not so much emphasis on the Cocoa App.

I really hope to do more processing stuff, it's way more interesting.

[ilia3101]

Will there be a way to chose between Prores 422 Proxy, LT, ST & HQ?

It doesn't seem to give a choice, seems Apple keeps to themselves how to access those options, and only offer 422 and 4444 to peasants.

[bouncyball-git]

I really hope to do more processing stuff, it's way more interesting.

I always thought you started this project because of this :)

[masc4ii]

It doesn't seem to give a choice, seems Apple keeps to themselves how to access those options, and only offer 422 and 4444 to peasants.

Hm, I added the "AVFoundation" option to 422ST, 4444 and H264.

[ilia3101]

What's 422ST? Is it different from just 422?

[masc4ii]

There is 422 Proxy, LT, Standard (ST) and HQ. I think and hope that "" is Standard.

[ilia3101]

I assume it probably is standard. Just never heard standard being called 422 ST before

[masc4ii]

@ilia3101 : did you ever see an output for 4444? Is it possible that we need to write ARGB here instead of RGB? (with A=0 or A=65535)

[masc4ii]

Ha... got it. Try it out :-) Alpha was set to transparent, that was why I got only black frames.

[masc4ii]

@ilia3101 : I got the macros for making AVFoundation compiled as best as the compiler can. And I can now export H264 too. As you wrote, it was a 16bit -> 8bit issue. I am not really happy with my solution: there are 2 functions for writing a frame now. Maybe you get it in one function. I don't know why, today I am too stupid to do on my own... I did not get it to work with just one function :-(

[masc4ii]

@ilia3101 : i have a question... you remember maybe, I changed the alpha channel to 2x 255 to make ProRes 4444 visible. Now that seems to make problems - I write over the buffer. The code is this:

/* Copy it to 'encode' buffer in ARGB64 format */
    uint8_t * original = (uint8_t *)frame;
    uint8_t * newframe = (uint8_t *)(encoder->encode_buffer+1);
    int pixels = encoder->width * encoder->height;

    for (int i = 0; i < pixels; ++i, newframe+=8, original+=6)
    { /* Byteswap needed too!!!! WTF */
        /* Red */
        newframe[0] = original[1];
        newframe[1] = original[0];
        /* Green */
        newframe[2] = original[3];
        newframe[3] = original[2];
        /* Blue */
        newframe[4] = original[5];
        newframe[5] = original[4];
        /* Alpha */
        newframe[6] = (uint8_t)0xFF;
        newframe[7] = (uint8_t)0xFF;
    }

Can you tell me, why you wrote in uint8_t * newframe = (uint8_t *)(encoder->encode_buffer+1); the +1 ? This should be the problem, because for newframe[6] of [7] the crash occurs (sometimes). But without the +1 it does not work. I now changed the buffer creation to: encoder->encode_buffer = calloc(width * height * 4 + 1, sizeof(uint16_t)); /* RGBA64 */ in init and the problem seems to be away... but I don't understand why this is the right way (maybe it is not?).

[ilia3101]

That plus one is really weird... not sure why it's there. Get rid of it instead of allocating more. Really not sure what I was doing there :/

[ilia3101]

I will try and make frame writing all be in one function when I get back soon. And sorry for the crashes I caused with that weird code :(

[masc4ii]

Oh no no, you did not cause the crash. It was me by setting the alpha channel. But by doing this the buffer is by one too small. ;-)

[bouncyball-git]

width height 4 + 1, sizeof(uint16_t) <- this has everything for RGBA even 2 bytes more ;) why it crashes? Edit: ah yes in this case it is not crashing :)

[bouncyball-git]

Are you sure it does not work without +1?

uint8_t * newframe = (uint8_t *)(encoder->encode_buffer+1);

What happens without +1?

[masc4ii]

width * height * 4 + 1, sizeof(uint16_t) with that +1 it does not crash. Without it crashes. This was my quick fix without knowing what I do.

uint8_t * newframe = (uint8_t *)(encoder->encode_buffer+1); without this +1 we get pinkish pictures.

[bouncyball-git]

If 1st byte of encoder->encode_buffer is unused everything works fine? ;) hehe interesting.

[masc4ii]

Yes, but this was already before. This is what Ilia and I don't understand. But hey... if we get nice 12bit ProRes4444... why not paying with one single byte extra :-D

[bouncyball-git]

Yes correct but still intriguing :)

[masc4ii]

Hej @ilia3101 , did you have this example from apple? Here they encode Video+Audio. Unfortunately I don't understand this Obj-C code really. Whatever I try it fails... but I don't know what I do there :-D

https://developer.apple.com/library/content/samplecode/ReaderWriter/Introduction/Intro.html

I thought I get ProRes+Audio to work... but hm... :-(

[ilia3101]

Helllo again, I've seen the apple sample code for audio (and a few other versions). I'll try and get it to work with audio probably really soon (been wanting to do a lot in the last few days). I'll be doing loads more MLV App stuff again very soon I promise.

[masc4ii]

AVFoundation export with audio is working since this commit. We used a simple trick proposed by Danne: Copy the wave via ffmpeg onto the silent AVFoundation export file (works without another encoding).