Feature: 4K Video Support

peterrussell commented 6 years ago

Looks like we'll finally get high resolution videos as part of the Apple TV 4K. https://9to5mac.com/2017/09/11/tvos-11-4k-apple-tv-screensaver/

Can't wait to get them on my iMac

huebs commented 6 years ago

My AppleTV 4K arrived today. First thing I did was fire up Aerial and... it's gorgeous.

As I said in my last post, I've got a TCL 55P607. It supports HDR and Dolby Vision. When I plugged in the Apple TV, it defaulted to 4K HDR. It then offered to try switching to Dolby Vision and then asked me to confirm it was working.

In the Video Fomats menu, it lists:

4K Dolby Vision 60Hz
4K HDR 60Hz
4K SDR 60Hz
1080p Dolby Vision 60Hz
1080p HDR 60Hz
1080p SDR 60Hz

Then there's an option for "Other Formats", where I find:

4K Dolby Vision 50Hz
4K HDR 50Hz
4K SDR 50Hz
4K Dolby Vision 30Hz
4K HDR 30Hz
4K SDR 30Hz
4K Dolby Vision 25Hz
4K HDR 25Hz
4K SDR 25Hz
4K Dolby Vision 24Hz
4K HDR 24Hz
4K SDR 24Hz
1080p Dolby Vision 50Hz
1080p HDR 50Hz
1080p SDR 50Hz
...and a handful of sub-1080p SDR 60Hz

If I plug in my 2016 15" MacBook Pro with the Radeon Pro 460 4GB graphics card, I'm presented with this:

Unless I'm missing something, it looks like there's there currently no way to drive an HDR display using a 2016 MacBook Pro.

Note that I tried using both a USB-C to HDMI adapter hooked up with the same HDMI cable I used for the Apple TV and a USB-C to HDMI cable.

The cable explicitly says it's only capable of 4K @ 30Hz. The adapter says it supports 4K @ 60Hz and has been, "updated to support 2017 MacBook Pro & iMac with Thunderbolt 3." Has anyone with either of those newer models tried hooking it up to a 4K HDR display?

My next step is to try this 4K @ 60Hz cable and see if that gets the OS to present different resolution options.

huebs commented 6 years ago

A helpful hint for anyone trying the above: to get my MBP to display an image on the 4K TV, I had to manually disable automatic graphics switching before plugging in either the cable or adapter.

benwiggins commented 6 years ago

@sebastienbarre The LG C7 will actually play the .mov files natively, once you patch the dvh1 codec identifier back to hevc.

That said, using the latest ffmpeg snapshot build you can remux the Quicktime MPEG-4 into an ISO standard .mp4 and patch the codec identifier in one go with ffmpeg -i input.mov -f hevc -vcodec copy output.mp4

sebastienbarre commented 6 years ago

@benwiggins thanks for the tip. Didn't quite work, unfortunately, either with the latest stable ffmpeg 3.3.4, or with the ffmpeg-87630-ge9f9175 snapshot (the included x265 library is a multi-bit version (8-bit, 10-bit, 12-bit)).

ffmpeg version N-87630-ge9f9175-tessus Copyright (c) 2000-2017 the FFmpeg developers
  built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth
 --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype 
--enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb 
--enable-libopencore-amrwb --enable-libopus --enable-libsnappy --enable-libsoxr 
--enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc 
--enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 
--enable-libx265 --enable-libxavs --enable-libxvid --enable-libzmq --enable-libzvbi 
--enable-version3 --disable-ffplay
  libavutil      55. 77.101 / 55. 77.101
  libavcodec     57.106.104 / 57.106.104
  libavformat    57. 82.102 / 57. 82.102
  libavdevice    57.  9.101 / 57.  9.101
  libavfilter     6.106.100 /  6.106.100
  libswscale      4.  7.103 /  4.  7.103
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
[dvvideo @ 0x7fca42010c00] could not find dv frame profile
[...]
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Dubai 1.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 0
    compatible_brands: qt
    creation_time   : 2017-07-14T23:32:18.000000Z
  Duration: 00:04:00.24, start: 0.000000, bitrate: 8261 kb/s
    Stream #0:0(und): Video: dvvideo (dvh1 / 0x31687664), yuv420p, 720x576, 8258 kb/s, SAR 1:1 DAR 5:4, 29.97 fps, 29.97 tbr, 30k tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2017-07-15T04:28:28.000000Z
      handler_name    : Core Media Data Handler
      encoder         : H.265
Output #0, hevc, to 'Dubai 1.mp4':
  Metadata:
    major_brand     : qt
    minor_version   : 0
    compatible_brands: qt
    encoder         : Lavf57.82.102
    Stream #0:0(und): Video: dvvideo (dvh1 / 0x31687664), yuv420p, 720x576 [SAR 1:1 DAR 5:4], q=2-31, 8258 kb/s, 29.97 fps, 29.97 tbr, 25 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2017-07-15T04:28:28.000000Z
      handler_name    : Core Media Data Handler
      encoder         : H.265
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[AVBSFContext @ 0x7fca44101fa0] Codec 'dvvideo' (25) is not supported by the bitstream filter
 'hevc_mp4toannexb'. Supported codecs are: hevc (174)
av_interleaved_write_frame(): Invalid argument

benwiggins commented 6 years ago

Whoops. I think I might have been running ffmpeg on a file I'd already patched. 😳

You could do vi -b -c "%s/dvh1/hvc1/g" -c "wq" input.mov && ffmpeg -i input.mov -vcodec copy output.mp4 if you still want an ISO MP4 container.

sebastienbarre commented 6 years ago

@benwiggins you sir, are a hero. Works great, looks beautiful on the C7. Now to find a way to override the default screensaver...

sabaatworld commented 6 years ago

Does the new Apple TV display only these 9~10 HDR videos meintioned in the JSON feed? The last one had more than 20 videos.

habovh commented 6 years ago

That's pretty much off-topic @sabaatworld, the issue already has a lot of comments, let's try to only post useful information related to Aerial and its 4K support. There are plenty of other places on the internet for your question (to which I don't know the answer) 😌

chrisjaber commented 6 years ago

AddoSolutions commented 6 years ago

Is there anything I can help with on this? Maybe we can get a V2 Beta 1 released that simply uses the new JSON url and downloads the SDR versions, when on High Sierra? Refactoring to a new version of swift, and updating the UI seems like something to do in another issue/feature. As aforementioned, this "feature" is already getting bloated enough :)

In my personal case, I see the Arial UI twice, once when I install it and the second when I check the box in screensavers that says "Arial". I may be speaking for myself, but I would much rather just use the SDR versions for my High Sierra box, just as it is.

Specifically:

https://github.com/JohnCoates/Aerial/blob/master/Aerial/Source/Models/ManifestLoader.swift#L76

let apiURL = "http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json"
if #available(OSX 10.13, *) {
         apiURL = "https://sylvan.apple.com/Aerials/2x/entries.json"
}

Then here: https://github.com/JohnCoates/Aerial/blob/master/Aerial/Source/Models/ManifestLoader.swift#L105

Move the guts of the loop into a separate method, check if "batches" is an array or object. If it's an array (aka the old screensavers) loop. If not, then go straight to the method.

(Note I am not a Swift dev, but just spitballing)

Thoughts?

esetnik commented 6 years ago

Running with the suggestion by @AddoSolutions I hacked together a quick 4k demo. You can live in the future now at https://github.com/esetnik/Aerial/tree/feature/4k.

Warning this is completely experimental at this point. 👍

quantumgolem commented 6 years ago

@esetnik Would this slow older Macs down?

esetnik commented 6 years ago

Yeah I think 4K cpu decoding on macs without decent integrated graphics will cause high cpu usage and battery consumption. I think eventually it’d be nice for Aerial to offer 4K video as an option in settings.

quantumgolem commented 6 years ago

@esetnik I think it would be much more useful allowing the user to convert the videos into 2K and then selecting them, no?

hakusaro commented 6 years ago

@esetnik I compiled your branch, but I get the dreaded issue from #4 and can't enable 4k. Did you do anything specific to get it to work on 10.13.1?

esetnik commented 6 years ago

I have the same problem but it seems worse than #4 because the close and reopen solution doesn’t work. It’s possible that we can’t compile with Xcode 9 🙁

hakusaro commented 6 years ago

(I'm on Xcode 9.1, I might add)

UniverseXXX commented 6 years ago

Didn't work for me either.

alistairmcmillan commented 6 years ago

@esetnik Your fork is Swift 3 code but bundling the Swift 4 libraries. Hence the error message.

BTW You should open the Issues tab on your fork so people can report bugs against it there instead of here. :)

esetnik commented 6 years ago

Thanks for the suggestion @alistairmcmillan. I’ve activated issues tab on my fork. Happy to accept a PR for fixing the swift 4 bundling. I have limited experience building a screensaver so any help on that front is appreciated.

esetnik commented 6 years ago

@esetnik Your fork is Swift 3 code but bundling the Swift 4 libraries. Hence the error message.

@alistairmcmillan can you comment on https://github.com/esetnik/Aerial/issues/1 regarding the correct procedure for bundling swift 3 libraries?

esetnik commented 6 years ago

I compiled a release of my branch with Xcode 8. If you just want 4k screensavers now you can download it here

UniverseXXX commented 6 years ago

@esetnik Thanks for this. I've now installed it. However, I'm looking at cached mov files' details and it seems that all of them are 1080p. Is it supposed to be like that or I'm missing something? Thanks.

esetnik commented 6 years ago

@UniverseXXX please direct all issues with my fork to https://github.com/esetnik/Aerial/issues. I want to avoid polluting the original thread.

I am unable to reproduce the issue of cached videos only 1080p. I have updated my release with more detailed logging information. Please try the following:

Install the latest release of my fork here
rm -f ~/Library/Caches/Aerial/*
open Console and filter for Aerial
open System Preferences -> Screen Saver -> Aerial -> Screen Saver Options -> Cache and hit Download Now

For each download file you should see the url being cached in the logs. It looks like default 13:41:18.856364 -0500 System Preferences uncached: [id=802866E6-4AAF-4A69-96EA-C582651391F1, name=Dubai, type=video, timeofDay=day, url=https://sylvan.apple.com/Aerials/2x/Videos/DB_D001_C005_4K_SDR_HEVC.mov]

You can confirm that the file is indeed 4k by opening in QuickTime directly from the cache or with media-info:

❯ mediainfo ~/Library/Caches/Aerial/DB_D001_C005_4K_SDR_HEVC.mov
General
Complete name                            : /Users/ethan/Library/Caches/Aerial/DB_D001_C005_4K_SDR_HEVC.mov
Format                                   : MPEG-4
Format profile                           : QuickTime
Codec ID                                 : qt   0000.00 (qt  )
File size                                : 446 MiB
Duration                                 : 7 min 30 s
Overall bit rate                         : 8 314 kb/s
Encoded date                             : UTC 2017-07-15 16:53:17
Tagged date                              : UTC 2017-07-16 02:02:24
Writing library                          : Apple QuickTime

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5@High
Codec ID                                 : hvc1
Codec ID/Info                            : High Efficiency Video Coding
Duration                                 : 7 min 30 s
Bit rate                                 : 8 310 kb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 29.970 (30000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.033
Stream size                              : 446 MiB (100%)
Title                                    : Core Media Video
Encoded date                             : UTC 2017-07-16 02:02:22
Tagged date                              : UTC 2017-07-16 02:02:24
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

scherii commented 6 years ago

Yep, 4K. 🙂

mediainfo ~/Library/Caches/Aerial/DB_D011_C009_4K_SDR_HEVC.mov                                                                                                                

General
Complete name                            : ~/Library/Caches/Aerial/DB_D011_C009_4K_SDR_HEVC.mov
Format                                   : MPEG-4
Format profile                           : QuickTime
Codec ID                                 : qt   0000.00 (qt  )
File size                                : 238 MiB
Duration                                 : 4 min 0 s
Overall bit rate                         : 8 294 kb/s
Encoded date                             : UTC 2017-07-14 15:15:56
Tagged date                              : UTC 2017-07-14 20:00:03
Writing library                          : Apple QuickTime

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5@High
Codec ID                                 : hvc1
Codec ID/Info                            : High Efficiency Video Coding
Duration                                 : 4 min 0 s
Bit rate                                 : 8 290 kb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 29.970 (30000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.033
Stream size                              : 237 MiB (100%)
Title                                    : Core Media Video
Encoded date                             : UTC 2017-07-14 20:00:02
Tagged date                              : UTC 2017-07-14 20:00:03
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

JeromeMartinez commented 6 years ago

@SuperDOS @esetnik @scherii it looks like you are happy users of MediaInfo and at the same time of Dolby Vision stuff, FYI we jut have implemented Dolby Vision support in MediaInfo, with detection and display of version/profie/layers, e.g.:

Dolby Vision                             : 1.0, dvhe.stn@fhd30, BL+RPU

You can use the Windows GUI snapshot (updates/other OSes here (mini 20171201)).

albinekb commented 6 years ago

Just an update, I've been running the 4K build on macOS high sierra beta since it was posted here, and it's been great, no issues at all! 🎉 https://github.com/JohnCoates/Aerial/issues/378#issuecomment-346098838

carlashley commented 6 years ago

@spitfire - hey, was looking at your JSON feed per comment https://github.com/JohnCoates/Aerial/issues/378#issuecomment-329960804 - it doesn't validate properly.

Each line containing "accessibilityLabel": has a trailing , character which should be dropped. Tested with https://jsonlint.com/

esetnik commented 6 years ago

@carlashley it’s actually a bug in Apple’s feed 🤣. We should file a radar haha.

carlashley commented 6 years ago

D’oh! facepalm.png I completely glossed over the Apple-ness of the URL! Sorry @spitfire!

Definitely file a RADAR. Have done it for GarageBand loops feed and it has been corrected in the past. Tho maybe Apple might decide to make it harder to find the videos 😉

esetnik commented 6 years ago

Yeah let’s not let them know we’ve found it just yet. Although it’s not valid JSON it still does parse so it doesn’t pose a real technical challenge.

huebs commented 6 years ago

I installed @esetnik's fork over Thanksgiving and it looks great. Since I've been back in my office this week and plugged into my external monitors, it had my CPU and GPU running at 50% load on my 2016 MBP. It also was dropping frames like crazy. I think the 2017 models have better/native support for decoding HEVC. If you regularly use multiple monitors, I'd avoid 4K for now.

esetnik commented 6 years ago

I’m getting a lot of reports related to 4K video playback but it’s odd because the discrete GPU in the 2016 MBP should be capable of decoding HEVC. Can you confirm that playing the cached videos directly in QuickTime also causes performance issues? Do you also have stuttering playback / high cpu usage 4K@60 HEVC video playback in Photos app? I’m trying to figure out if this is a screensaver exclusive issue or extends to other system apps.

carlashley commented 6 years ago

@esetnik - my problem was trying to parse it in python with requests (firstly), then with the standard json library :>

Totally agree - definitely don't want to alert Apple to the fact we're onto it. 💯

huebs commented 6 years ago

@esetnik I'll open an issue on your fork and go into more detail there.

cmfrazier commented 6 years ago

@esetnik I was having playback issues with 4K at first, but after I cached all the videos, playback has been smooth. (Late 2016 MBP) No shutter in QT.

hssuhas commented 6 years ago

@esetnik

Thank you very much. The 4k videos are a delight! Strangely, the issue I had earlier on my mbp 13' (with the workaround of having to enable the clock) is no longer seen!

I see that there are 15 videos now. Any estimate of when we might be treated to the rest of the videos?!! Thanks in advance. Cheers mate!

esetnik commented 6 years ago

@hssuhas I’m glad you like it! Unfortunately Apple only shipped the limited set of 4K videos so that’s all we have for now. Maybe they will release more in the future.

jyavenard commented 6 years ago

HDR videos look crap on an IPad Pro 10.5" bb19761d-3d83-4d9a-a2f1-b335692bb19a 5f650c7e-c0c8-4ca5-a820-eb0547400ded

jyavenard commented 6 years ago

@esetnik

2016 MBP should be capable of decoding HEVC

High Sierra for now only supports HEVC via Intel. The MBP 2016 skylake doesn't do HEVC HDR, only kabylake does (found in 2017 models)

chipped commented 6 years ago

@jyavenard

High Sierra for now only supports HEVC via Intel. The MBP 2016 skylake doesn't do HEVC HDR, only kabylake does (found in 2017 models)

Where did you read that? It also works on Radeon Pro 4xx and 5xx on 2016 and 2017 MacBook Pro.

jyavenard commented 6 years ago

That it works doesn't mean that decoding will be hardware accelerated.

As for where this info is from, it's in Intel spécialisé as well as in AMD.

chipped commented 6 years ago

@jyavenard

That it works doesn't mean that decoding will be hardware accelerated.

As for where this info is from, it's in Intel spécialisé as well as in AMD.

Yes it does, you are wrong.

Watch the WWDC keynote, 29min48sec, it clearly states hardware acceleration on MacBook Pro 2016.

https://www.apple.com/au/apple-events/june-2017/

mattvlasach commented 6 years ago

For what it is worth, I have the early 2016 MBP and the 4k playback is very jumpy.

So hardware accelerated or not, something about that model’s processor/gpu is not handling the format properly for smooth playback.

jyavenard commented 6 years ago

@chipped theres HEVC hardware decoding support for SDR, not HDR in the macbook pro 2016. There's no full HDR HEVC support in skylake Intel period. h265 main10 (the profile used in those videos) is only partially accelerated, just like VP9 was only partially accelerated (and pretty badly at that) Only kabylake added full hardware VP9 and h265 main10

Check the Intel slide on the anandtech skylake presentation. https://www.anandtech.com/show/9562/intels-skylake-gpu-analyzing-the-media-capabilities.

HEVC_VLD_Main10 has a DXVA profile, but it is done partially in the GPU (as specified in the slide above)

For a thorough presentation of performance of skylake VS kabylake https://youtu.be/cI-a4WZWwZc

A quick search of the difference between skylake and kabylake for hevc vp9 HDR will yield 100s of comparison https://www.digitaltrends.com/computing/intel-kaby-lake-s-hdr/ http://www.fudzilla.com/news/graphics/40959-kaby-lake-gpu-learns-hardware-hdr

HEVC hdr on a macbook pro 2016 will play poorly. Including on late 2016 MBP. Try the 2017 MBP, with the 8th gen Intel, it's night and day

chipped commented 6 years ago

@jyavenard You are looking at the wrong place mate.

It uses the Radeon Pro for HEVC hardware accelerated decoding.

@mattvlasach I also have the 2016 MacBook Pro with Radeon Pro 460. What type of 4K media were you using and does the player you use have Hardware Acceleration support?

chipped commented 6 years ago

@jyavenard

“macOS High Sierra adds support for High Efficiency Video Coding (HEVC), with hardware acceleration where available, as well as support for High Efficiency Image File Format (HEIF). Macs with the Intel Kaby Lake processor offer hardware support for Main 10 profile 10-bit hardware decoding, those with the Intel Skylake processor support Main profile 8-bit hardware decoding, and those with AMD Radeon 400 series graphics also support full HEVC decoding.[5] In addition, audio codecs FLAC and Opus will also be supported, but not in iTunes.[11][12]”

https://en.wikipedia.org/wiki/MacOS_High_Sierra

Changes > System > Media

jyavenard commented 6 years ago

@chipped did you even try to open those 4K HDR files on High-Sierra on a MBP 2016? My guess is that you haven't despite your claims. There's no way you could even open those files on a MBP 2016 with any of the Apple player. (quicktime or safari) They are using dolbyvision variant, and there's no public decoders for those yet, not even VLC will play them.

For the others, why do you think this thread exists and so many people report playback issue? (stuttering and so forth).

Edit: on a MBP 15" late 2016 (i7 2.9Ghz skylake i7-6920HQ with Radeon Pro 460) the 4K SDR plays in Qt. VTDecpderXPCService in Activity Monitor is around 108% CPU, QT itself around 8% CPU. So all software decoding.

anyhow, I'm done.... obviously wasting my time

mattvlasach commented 6 years ago

Ok, my MBP is: MacBook Pro (13-inch, 2016, Four Thunderbolt 3 Ports) - 10.13.2 Intel Iris Graphics 550 1536 MB

It does play in QuickTime Player, but shoots the processor well over 100%.

chipped commented 6 years ago

@jyavenard I figured out your stuttering problem. I have attached a screen recording, read this first and then watch it.

You can see the little "d" or "i" in my menu bar. This is an app called gfxCardStatus and it indicates Dedicated in use or Integrated in use. If you click on it shows it any Dependencies that are asking for the Dedicated graphics card. I always turn off Automatic Graphics Switching on my MacBook Pro so it runs off the Radeon Pro all the time using this app.

You guys having probably have Automated Graphics Switching turned on (you can control this in System Preferences > Energy Saver OR select Discrete only in gfxCardStatus menu bar menu. NOTE Must be done after each reboot for either method) and its causing your stuttering problem. Quicktime does not tell macOS it needs a Dependency, so the system doesn't switch to the Radeon Pro. This seems like a flaw and I may submit a bug report about it.

You can see when I manually switch to the Radeon Pro that the Frame Rate is perfect and the Processor on the Radeon is loaded up. This indicates Hardware Acceleration is working on the Radeon Pro. When I switch to Automated Graphics Switching the Integrated Intel is in use and it stutters, you can also see the intel GPU is idle as its being software decoded.

Link to video https://drive.google.com/file/d/1WfKLAp2NpYxs9RWaRk2bUoReOvxe0Ero/view?usp=sharing

Link to gfxCardStatus https://gfx.io

The app I'm using to monitor CPU and GPU stats is iStat Menus https://bjango.com/mac/istatmenus/

You're welcome 👍

JohnCoates / Aerial

Feature: 4K Video Support #378