search

olive-editor / olive

Free open-source non-linear video editor
https://olivevideoeditor.org/
GNU General Public License v3.0
8.22k stars 556 forks source link

[Feature Request/ Discussion] Editing and Exporting Videos in HDR (High Dynamic Range) #237

Closed Corruptinator closed 5 years ago

Corruptinator commented 5 years ago

This might be challenging to ask but would it be possible to implement a High Dynamic Range video editing workflow to Olive?

I was doing a bit of research into videos that are compiled/rendered in HDR. They're mostly HEVC/x.265 video files encoded with an HDR metadata embedded information. FFMPEG is capable of rendering videos in HDR using this command:

ffmpeg -i <infile> -c:v libx265 -tag:v hvc1 -crf 22 -pix_fmt yuv420p10le -x265-params "colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,10):max-cll=1000,400" <outfile>.mkv

Just for a useful rundown:

I was thinking that when exporting in HDR, the Export Window could be utilized like this:

hdr export example (Its a photo-editing example just to give an idea nonetheless...)

Assuming that Olive uses FFMPEG 3.4 and above, rendering and exporting videos in HDR sounds simple to do, that is in using the command line above.

Another problem to tackle would be that the video viewer would need to utilize a Colorspace adjuster to be able to view HDR videos in near similar colors in SDR. One way that can be done is using color-grading lut files to quickly modify HDR videos to be able to display similar HDR colors in SDR without spending money on an HDR reference monitor. Just a little useful information that could serve purpose.

As I already said earlier, this is challenging to ask and you probably are very busy, but I just wanted to post this feature request so then the idea of implementing the HDR Video Editing workflow idea isn't faded away. If this doesn't work for some reason, well at least it was worth asking.

P.S. Olive would need to have the x.265 encoder (FFMPEG) to save the videos in HEVC for HDR.

Corruptinator commented 5 years ago

Nevermind, I think I may have figured it out. I'll let you know if the compile works.

Corruptinator commented 5 years ago

Stuck at this error now:

$ make
make -f Makefile.Debug
make[1]: Entering directory '/c/Github/olive'
g++ -Wl,-subsystem,console -mthreads -o debug/olive-editor.exe @object_script.olive-editor.Debug  -lavutil -lavformat -lavcodec -lavfilter -lswscale -lswresample -lopengl32 -luser32 -lOpenColorIO C:/msys64/mingw64/lib/libQt5Multimediad.dll.a C:/msys64/mingw64/lib/libQt5OpenGLd.dll.a C:/msys64/mingw64/lib/libQt5Svgd.dll.a C:/msys64/mingw64/lib/libQt5Widgetsd.dll.a C:/msys64/mingw64/lib/libQt5Guid.dll.a C:/msys64/mingw64/lib/libQt5Networkd.dll.a C:/msys64/mingw64/lib/libQt5Cored.dll.a  C:/msys64/mingw64/x86_64-w64-mingw32/lib/libglu32.a C:/msys64/mingw64/x86_64-w64-mingw32/lib/libopengl32.a C:/msys64/mingw64/x86_64-w64-mingw32/lib/libgdi32.a C:/msys64/mingw64/x86_64-w64-mingw32/lib/libuser32.a
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lOpenColorIO
collect2.exe: error: ld returned 1 exit status
make[1]: *** [Makefile.Debug:663: debug/olive-editor.exe] Error 1
make[1]: Leaving directory '/c/Github/olive'
make: *** [Makefile:38: debug] Error 2
itsmattkc commented 5 years ago

@Corruptinator You're on MSYS2 yeah? Did you compile the OCIO source code I linked above? If not, compiling it on MSYS2 is unfortunately non-trivial, but this zip should provide the files you need (extract into C:\msys64\mingw64

ocio-1.0.9-msys2-win64.zip

Corruptinator commented 5 years ago

Just did that, Still the same error:

$ make
make -f Makefile.Debug
make[1]: Entering directory '/c/Github/olive'
g++ -Wl,-subsystem,console -mthreads -o debug/olive-editor.exe @object_script.olive-editor.Debug  -lavutil -lavformat -lavcodec -lavfilter -lswscale -lswresample -lopengl32 -luser32 -lOpenColorIO C:/msys64/mingw64/lib/libQt5Multimediad.dll.a C:/msys64/mingw64/lib/libQt5OpenGLd.dll.a C:/msys64/mingw64/lib/libQt5Svgd.dll.a C:/msys64/mingw64/lib/libQt5Widgetsd.dll.a C:/msys64/mingw64/lib/libQt5Guid.dll.a C:/msys64/mingw64/lib/libQt5Networkd.dll.a C:/msys64/mingw64/lib/libQt5Cored.dll.a  C:/msys64/mingw64/x86_64-w64-mingw32/lib/libglu32.a C:/msys64/mingw64/x86_64-w64-mingw32/lib/libopengl32.a C:/msys64/mingw64/x86_64-w64-mingw32/lib/libgdi32.a C:/msys64/mingw64/x86_64-w64-mingw32/lib/libuser32.a
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lOpenColorIO
collect2.exe: error: ld returned 1 exit status
make[1]: *** [Makefile.Debug:663: debug/olive-editor.exe] Error 1
make[1]: Leaving directory '/c/Github/olive'
make: *** [Makefile:38: debug] Error 2

I noticed where it says C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lOpenColorIO Is there a typo in the source branch?

DaniSeeh commented 5 years ago

I was finally able to compile and run it, but after downloading the Open color config files I still couldn't replicate the output @itsmattkc showed. I used Nuke-Default but couldn't make it work for some reason

Corruptinator commented 5 years ago

Which OS did you use?

DaniSeeh commented 5 years ago

Manjaro. I got it working. Had to build everything myself.

DGMurdockIII commented 5 years ago

This might help https://github.com/ampas/aces-dev/blob/master/README.md

Corruptinator commented 5 years ago

I know at this point that this has slowly evolved into Color Management, but I found this interesting information when it comes to Master Displays in relation to HDR encoding: https://github.com/SK-Hardwired/nv_hevc_hdr_patcher/issues/9#issuecomment-392572348

sobotka commented 5 years ago

No idea why they need to take identified standard primaries and mangle them up. Seems like broken software using custom broken code paths.

Corruptinator commented 5 years ago

Should Master Displays not be used then?

sobotka commented 5 years ago

If I knew what that was, I would be able to comment.

I wouldn’t go anywhere near something that makes you specify chromaticities, but then mangle them. That seems like meatheadware.

Corruptinator commented 5 years ago

Understood, I was mostly curious in experimenting FFMPEG to try to export to HDR, using this command:

ffmpeg -i -c:v libx265 -tag:v hvc1 -crf 22 -pix_fmt yuv420p10le -x265-params "colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,10):max-cll=1000,400" .mkv

You could also export/mix to "HDR" without using master display:

ffmpeg -i -c:v libx265 -tag:v hvc1 -crf 22 -pix_fmt yuv420p10le -x265-params "colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc" .mkv

Granted that its definitely not going to work well, I'm always down to test and experiment.

sobotka commented 5 years ago

You can’t simply convert SDR material to HDR. Last time I tested extensively, FFMPEG’s colour handling was quite a mess. Not sure if it has improved since then.

At any rate, HDR describes the display’s capability. You would likely want to control how the scene referred data is transformed to the display referred HDR output encoding.

Corruptinator commented 5 years ago

I'm assuming I would a monitor of some sort that handles HDR to examine and adjust the HDR output?

Corruptinator commented 5 years ago

Also FFMPEG isn't good at colour handling? hmm... I don't know if there is anything else that could encode videos in HDR.

Corruptinator commented 5 years ago

There is also LumaHDRV: http://lumahdrv.org/ But I'm assuming that isn't going to be worth it right?

Corruptinator commented 5 years ago

Not that I'm trying to hold hopes onto using HDR but I rather hear critiques/information that could otherwise correct my thoughts about it otherwise.

sobotka commented 5 years ago

I'm assuming I would a monitor of some sort that handles HDR to examine and adjust the HDR output?

You would, ideally. You can also lean on a particular set of view transforms.

Corruptinator commented 5 years ago

Got it. What about 4K HDR Television sets? (Do happen to have one by the way) Most of them have the capability to connect to computers as a secondary monitor, but I'm assuming that would pale in comparison.

sobotka commented 5 years ago

Got it. What about 4K HDR Television sets? (Do happen to have one by the way) Most of them have the capability to connect to computers as a secondary monitor, but I'm assuming that would pale in comparison.

That would work just fine, vendor and quality of the display depending of course.

There are several issues however:

  1. You need to trigger the HDR mode on the display. Only Windows "supports" this at this point, with macOS soon I suspect, but the Windows version is so horribly mangled up with stupid that it is essentially a dead end at this point.
  2. The application would need to support proper encoding at the lower level to feed it to the operating system.
  3. The application would need to be managed if you wanted to work properly. This includes the user interface transforms, so that your window icons and "white" aren't blowing up bright and the colours are correctly encoded.

Currently, using a Vertex type device would also work to trigger HDR mode, but then you are still saddled with item (3) above, as well as low depth buffers at the operating system level, unless the application is using deeper buffers.

Other option is to properly handle the encoding to a standard HDR encoding (ST.2084 with REC.2020 primaries) and then take the master to an encoder to wrap up the metadata and bitstream as required.

Corruptinator commented 5 years ago

Noted. Thanks for the information. I'm probably going to have to investigate this.

I actually wonder if Olive can be able to utilize the "show fullscreen", (with Item # 3 in mind):

image

But with a built in calibration to be compatable to "correctly" output/display the preview in HDR on the external monitor/tv? Regardless of the Windows and MacOS issue you pointed out?

That is sending an HDR Metadata signal to trigger the monitor to display in HDR.

sobotka commented 5 years ago

But with a built in calibration to be compatable to "correctly" output/display the preview in HDR on the external monitor/tv? Regardless of the Windows and MacOS issue you pointed out?

Just let OCIO format the encoding based on the Display / View setting on that particular viewer, and then hand it down the lower level API and it should be fine. If the buffer is minimum 10 bit that hits the display, you could pass that along using a Vertex and trigger it on an OLED or QLED television and get a reasonable output I'd think. Would need to test with a Vertex and suitably finished software.

The bigger issue is the darn UI, which needs to be 100% managed. See those buttons? They'd be 2000 nits if unmanaged on the Vizio Quantum, or peak 750 nits on an LG OLED. The colours would be all mangled too, even on passive icons. The grey surround? That too wouldn't be the appropriate grey as it appears on your sRGB output. You probably get the idea; Digital Content Creation software needs to be properly designed to handle these issues, and a majority are about a decade behind the curve.

Corruptinator commented 5 years ago

Yikes, guess some things needs to get done.

My tv model is: (Samsung) 40" CLASS MU6290 4K UHD TV Got it on good bargaining sale at Best Buy.

sobotka commented 5 years ago

Sadly not terribly HDR. https://www.rtings.com/tv/reviews/samsung/mu6290#comparison_1675

A minimum baseline for HDR as a rule of thumb:

  1. 650+ nit OLED.
  2. 1000+ nit QLED or otherwise.
Corruptinator commented 5 years ago

Understood.

musaire commented 5 years ago

You guys are geniuses

DaniSeeh commented 5 years ago

While this thread was extremely influential, I believe that the implementation of OCIO has mostly solved this Feature Request, yes @sobotka? I know that there may still be more to be done (especially as regards to bugs), so if this is issue is still a productive thread we can just reopen it.

Corruptinator commented 5 years ago

@DaniSeeh, I do agree with your decision. This Feature Request has been largely influential, especially with the implementation of OCIO. Problem related to this request, which prior to finishing color management is Decoding and Encoding of HDR video files.

Say I recorded a video using a DSLR capable of recording in HDR the video would be encoded with HDR metadata. Therefore I think what should come next is a way to implement a library that can decode the HDR footage into Olive and Encode the final cut on export.

@sobotka pointed out that FFMPEG isn't the ideal choice due to a lot of color encoding issues or something

However, maybe olive might not even use FFMPEG for encoding into HDR...

I stumbled upon this webpage that showcases an open source encoder and decoder for HDR videos:

http://lumahdrv.org/

Its a C++ library under the BSD license that allows HDR video files to be read and exported, on the main page it goes into detail on how the encoding and decoding works.

Im not sure what @sobotka's opinion or thoughts on this is going to be but making HDR videos will require a specialized metadata encoding.

sobotka commented 5 years ago

My opinions aren’t worth much.

In terms of “correctness of encoding”, someone pushing pixels around can only handle their encoding as far as they can, which in most cases is REC.2020 with ST.2084, after a decent camera rendering transform.

From there, it should be up to the encoder to permit you to tell the encoder what format your encode is in, and let the encoder assign the appropriate metadata into the bitstream (MaxFALL, MaxCLL, and other such etc.)

Corruptinator commented 5 years ago

Got it.

Hopefullyidontgetbanned commented 2 months ago

While this thread was extremely influential, I believe that the implementation of OCIO has mostly solved this Feature Request, yes @sobotka? I know that there may still be more to be done (especially as regards to bugs), so if this is issue is still a productive thread we can just reopen it.

I don't really get it, we have importing HDR content working fabulously in Olive 5 years later, but there is not any export options into any known HDR formats that isn't raw image sequences? Isn't the point to work in HDR so that you can export to HDR?