Questions about export

[bouncyball-git]

I was thinking lately and I've got something to ask:

1. Why we are dragging the PNG lib with this project if QT has QImageWriter class and supports PNG natively? I guess AVFoundation or else also supports it on a Mac.

2. I've dug a bit to the 'RenderPngTask' stuff and saw that my future suggestion is not that trivial to implement in present working code. However... I want to share the very simple approach (perhaps already thought by you) to the encoder part.

For sure the best way to implement the native encoder/muxer is to use on win/linux libavformat and libavcodec, but as we all know it is complex and time consuming (on a mac there is mac flavors like avfoundation etc).

So, because we already use FFMPEG I thought why can't we make it swallow the PNGs (whatever) trough the named pipe? Run ffmpeg with pipe open and wright to thet pipe from the MLV App. This is gonna eliminate annoying intermediate files and their handling with less effort.

What do you think guys?

[masc4ii]

1. Qt supports only 8bit png, for 10bit ProRes we need 16bit png, this is only possible with libpng. If you check out a early version, you can see the 8bit Qt solution. AVFoundation is not availlable for Qt (as I know, maybe I am wrong). So Ilia will try to create a library which does that. Edit: This is the first commit with libpng. Before it was done with Qt.

2. If you know how to export using libavformat and libavcodec this is a much better solution. I already lost a lot of time with that without luck. :-( There should be a very early commit where I tried to prepare that. But later I deleted that again, because the intermediate png solution was easier and working. If you get it without png to work, the export speed should be the double from now. Edit: This this was my try to prepare for libavcodec.

Do you have an example with that pipe? Sounds interesting - never heard about that.

[bouncyball-git]

Ah ok I got it (QT 8 bit, bummer)

Yeah I saw your libav* attempts :) And what about pipe the data to ffmpeg? for example like this: https://stackoverflow.com/questions/43853630/c-program-to-pipeout-audio-file-to-ffmpeg-and-generate-video-file

[masc4ii]

This piping is really interesting. There is only one problem: if we want it to have fast, I think we need multithreaded frame calculation as I did in the Qt part now, but then the frames will maybe not come out in the right order. So we have to sort them and give it after that to the pipe. Or what do you think?

[bouncyball-git]

Yes I was thinking about it as now it implemented as a thread task and this is cool. But right order may be a problem with the pipe until we could restrict thread tasks to work only with, say, 4 consecutive frames at a time and so on. Should be some caching mechanism to order those processed 4 frames in memory.

Btw does QT support 16bit tiffs?

[bouncyball-git]

I think it would be fantastic to use the one caching method for both cases: playing and exporting.

[masc4ii]

When I understood right, Qt supports only 8bit, no matter which format.

I am not sure if caching is the solution for the program. Because the only benefit is to have Amaze debayered frames in the speed of the basic debayered frames in playback (and maybe export (when waiting before)). The bottleneck is still all the processing (not threaded). And for laptops, caching will cause battery drain, because you'll always have near to 100% CPU. Maybe we need a completely different caching method... Edit: but caching processed frames is also no solution, because changing only 1 parameter deletes all and everything must be cached again.

Yes, if we use threads for export we have to sort the pictures. If we have just one or four pictures per thread does not change that. The algorithm should be nearly identical.

[bouncyball-git]

Yes right. Sorting needed no matter what.

[ilia3101]

The cocoa app will lose ffmpeg once AVFoundation is sorted. But this is definitely an very important issue 👍 And of course, same cache will be used for export and preview, as getProcessedFrame function(s) is used in both cases, and that function/s does use the cache. When the cache is working, export should become much quicker.

[bouncyball-git]

I really need to try your latest caching code :)

[ilia3101]

@bouncyball-git Will commit today to the caching branch

[ilia3101]

@masc4ii maybe reduce amount of threads caching uses? caching will cause battery drain, because you'll always have near to 100% CPU In the end, it still ends up using less overall power on frames if the user is going to spend a while on the clip.

Could also have a tick box for enabling/disabling it.

[masc4ii]

Great news! Yes, sure! I'll make that switchable via checkbox! I think I already had that in the past... :-)

[ilia3101]

Little caching update:

• There's going to be a caching function which can have multiple instances on multiple threads for one clip. This bit is done. (Uses single threaded amaze, but multiple frames on different threads so no little split lines in frames) < this one is done
• There's also functions to add a cache thread and remove cache thread to a clip, which will be controlled by those setting number of cores functions (previously used) < this bit is mostly done. But not going to be done this evening, tho I made much progress. ~Will most likely be finished tommorow.~ will be (really not much left)

it's gonna be majorly thread safe this time.

It doesn't work yet, so won't commit anything yet.

[masc4ii]

Proof of concept - topic ffmpeg pipe

I got ffmpeg pipe to work with this simple code. TODO: audio, maybe via wav.

    FILE *pPipe;
    long lSize;
    uint16_t * imgdata;

    lSize = getMlvWidth( m_pMlvObject ) * getMlvHeight( m_pMlvObject ) * 3 * 2;

    imgdata = (uint16_t*)malloc( lSize );

    if ( !(pPipe = popen("\"<path to the app>/MLV App.app/Contents/MacOS/ffmpeg\" -r 25 -y -f rawvideo -s 1856x1044 -pix_fmt rgb48 -i - -c:v prores_ks -profile:v 2 <a path>/test.mov", "w")) )
    {
        qDebug() << "popen error";
        return;
    }

    // write to pipe
    for( int i = 0; i < getMlvFrames( m_pMlvObject ); i++ )
    {
        getMlvProcessedFrame16( m_pMlvObject, i, imgdata );
        fwrite(imgdata, 1, lSize, pPipe);
        fflush(pPipe);
    }
    fclose(pPipe);

    free( imgdata );

So intermediate png will be history soon! :-)

[bouncyball-git]

I'm glad it worked :). Are you sure passing imgdata is sufficient? I thought whole png with header should be passed through the pipe.

[masc4ii]

No, only the RawData. I got a first test video with that (without sound for now).

[ilia3101]

oo0h

Also I'll have a fix for noisy sound parts soon.

[bouncyball-git]

@ilia3101: What do you mean? You've implemented audio noise filter?

[bouncyball-git]

@masc4ii: Ah now I understand: "-f rawvideo -s 1856x1044 -pix_fmt rgb48" needs to specify all image info by cli. Cool! I'm so glad You did it. I had no time to experiment with it.

[bouncyball-git]

It works on Linux quickly and nicely! No crash on 2nd export. Progress bar is cool. It's just resolution of the video is not implemented dynamically in ffmpeg's cli string yet ;)

[masc4ii]

Now size is dynamic and I found why it crashed on 2nd export on Mac. Unforunately it does not work on Windows. I got ffmpeg to work via pipe - ffmpeg starts and creates a file, but there is only noise inside. I have no idea why. And on Windows popen(...) creates a terminal window :-P what da f*** :-( On Mac it works fine and much faster than before.

[bouncyball-git]

Yup! On Linux it's working great!

Can be ffmpeg console output made silent and dependent only on 'DEFINES += STDOUT_SILENT'?

[bouncyball-git]

For window we could try #ifdef win32 and _popen

[bouncyball-git]

We can also try the 'ffmpeg [commands] >/dev/null' when STDOUT_SILENT and without it when #STDOUT_SILENT

[masc4ii]

On windows it tells #define popen() _popen(), so I think nothing will change. The only way we could chose is a completely different one, for example doing it with QProcess or something similar, but I had absolutly no success. :-( There is no flush() and all written data is in RAM until I want close() the pipe... then it starts writing... currupted files :-P yeah.

[masc4ii]

I tried:

#ifdef Q_OS_UNIX
#ifdef STDOUT_SILENT
    program.append( " >/dev/null" );
#endif
#endif

before calling popen(), but that does not work. The log is full with ffmpeg output.

[ilia3101]

@ilia3101: What do you mean? You've implemented audio noise filter? No I meant, sometimes the MLV audio just has a random second of really loud hiss, which is really painful to the ears, me and masc discussed it when audio was first implemented, just recently realised it was the collision between reading the same file for audio and frames - this is now less likely but still can happen. Maybe it was a little weird to mention such offtopic so randomly :[

[bouncyball-git]

@ilia3101

collision between reading the same file for audio and frames

Hmm... very strange, can you explain it in more details?

[ilia3101]

Well, I fixed this isssue for caching by having separate files for all threads. Basically when different threads try to read same FILE * in different locations, issues like pink noise frames happen, or noisy chunks of audio.

[bouncyball-git]

Thank you, understood.

[bouncyball-git]

Now pipe export is working under windows too. Try this static binary.

[masc4ii]

What have you changed to make it work?

Edit: does not work for me. There is no output file. Edit2: my fault... ffmpeg.exe is not included and needs to be there as usual... :)

Edit3: oh my god... "w -> wb" did it??? LOL

[masc4ii]

Coooool, thanks so much... do we want to close the issue? Export does not need PNGs anymore on all three platforms. Or do we want to keep it until someone has a "console-less" idea for Windows?

[bouncyball-git]

I don't think the console pop up is curable. So we can close this issue.

The funny thing is if "wb" specified, 'popen' does not return success on Linux (I think on mac too). The distorted image gave me the hint that pipe is working right but the 48bit RGB data was somehow altered. I could not find any info on the net and tortured the command line string of ffmpeg almost 2 hours in a row. And finally, when I've almost given up, just tried binary mode for writing and it worked ;).

[bouncyball-git]

One more thing... I think if audio all the time in the memory it also can be implemented through the pipe. Use 2 pipes something like this:

ffmpeg -pix_fmt uyvy422 -s 720×576 -f rawvideo -i output_pipe -target pal-dvd -aspect 4:3 -y myDVD.mpg -f s16le -ar 48000 -ac 2 -i pcm0_pipe -acodec mp2 -ab 224k -newaudio

ffmpeg -r 24 -pix_fmt rgba -s 1280x720 -f rawvideo -y -i \\.\pipe\videopipe -f s16le -ac 1 -ar 44100 -i \\.\pipe\audiopipe -acodec pcm_s16le -ac 1 -b:a 320k -ar 44100 -vf vflip -vcodec mpeg1video -qscale 4 -bufsize 500KB -maxrate 5000KB OUTPUT_FILE

There should be controlled audio length according to the frames and FPS.

[masc4ii]

Hm, but this uses "Named pipe"... this seems to be more complex than our solution. This command I also found, but no code how to write data to more than one pipe in C/C++ (best using Qt modules). popen() will return your pipe to write to... one pipe only. So using 2 pipes must work different I think.

[bouncyball-git]

Yes the command is mkfifo("pipe_out", 0766) and then pipeoutfd = open("pipe_out", O_WRONLY) then make write or read and you can make as many fifos as you need.

But I think our popen which writes to stdout is a cool and simple solution so you are right - let's leave the wav file alone :) it just works.

[masc4ii]

This solution is really not bad, but if I understood right, Unix only. Or does that work also on Windows? Edit: Wiki says this. If anyone searchs for tasks, this can be implemented and this issue will be reopened. =) But we need two different versions: Unix & Win -> more complex for crossplatform. For now, I close to clean up a bit.