Add 4K, HDR, HEVC Aerials support and new scenes

kubedzero commented 3 years ago

Overview

This PR was inspired by my desire to have the latest Apple Aerials screensavers in Kodi without the hacky approach I previously used (which was to download the 4K files locally and name them as the old filenames)

I tested on Kodi 19.0 Matrix on MacOS 11 Big Sur, as well as Windows 10 1907. I was not able to test the language packs for ES and PT but I updated them as best I could.

I'll continue testing over the next couple days to ensure it works as expected.

Below are summaries of the different categories of changes in this PR

Offline/Downloader

Bugfix of checksum calculation of local file: hashlib.md5(xbmcvfs.File.read()) would throw a UTF-8 error, so it was changed to hashlib.md5(xbmcvfs.File.readBytes()) to match the behavior of normal file opening
Printouts that output to the Kodi log in debug mode to describe the state of downloading or skipping videos
Updated list of locations for the user to choose to download. I opted to include the current live set from Apple, rather than the extended/hand-written set (those the extended set can still be downloaded by choosing "All")
Downloading of the best match possible in each scene/location, so if 4K/HEVC/HDR is chosen but only 1080P/H264/SDR is available, it will download that instead.

Playlist

Automatic download of resources.tar from Apple, plus the extraction of entries.json (provided the setting is toggled correctly to do so)
Updated parsing to match the new structure of entries.json

ATV

New Settings added for 4K, HEVC, HDR
New Setting added for preserving custom JSON (avoiding the auto-update from the Apple JSON, even when offline mode is turned off)

ChecksumGenerator

Added logic to parse the new JSON out of the tarball
Added logic to compute checksums for all codec/resolution/color-space combinations
Added logic to output the total file count processed, plus the total data downloaded/read

Other

Added README info linking to other Aerial projects as well as statistics and history of the project and this plugin's feature set
New entries.json downloaded from Apple and then hand-edited to contain all non-included unique scenes (as per the Google doc in the README)
Time of Day code removed throughout, as Apple no longer publishes this information in the new JSON format.
- We could add it into the JSON by hand and restore the code to re-enable the feature, but I figure that would be a followup task
Regionalization/Internationalization strings updated to make the GUI more clear in some cases (checksums, Hawaii typo, offline mode)
Checksums added for all codec/resolution/color-space combinations
JetBrains PyCharm Code Reformat, which rearranged imports in a few files
Extensive commenting and variable naming refactoring to aid in readability and self-documentation
Moved some declarations that were only used in once out of commonatv and into the file where they were used
Created some declarations in commonatv that are used among multiple files. Namely, based on the codec/resolution/color-space flags, creation of an ordered list of most-preferred to least-preferred video versions, and a parser for this list.
Added more exclusions to .gitignore
Marked the plugin version as 3.0.1, which will hopefully show as an update in clients

kubedzero commented 3 years ago

Testing the new commit now, will update once I confirm the logging works as expected.

kubedzero commented 3 years ago

Looks good on MacOS:

2021-03-14 10:01:47.161 T:2882378   DEBUG <general>: CPythonInvoker(5, /Users/kz/Library/Application Support/Kodi/addons/screensaver.atv4/entrypointscript.py): instantiating addon using automatically obtained id of "screensaver.atv4" dependent on version 3.0.0 of the xbmc.python api
2021-03-14 10:01:47.273 T:2882378   DEBUG <general>: Downloading the Apple Aerials resources.tar to disk
2021-03-14 10:01:47.808 T:2882378   DEBUG <general>: Extracting entries.json from resources.tar and placing in ./resources
2021-03-14 10:01:47.816 T:2882378   DEBUG <general>: Deleting resources.tar now that we've grabbed entries.json from it
2021-03-14 10:01:47.818 T:2882378   DEBUG <general>: Adding video for location Dubai to playlist
2021-03-14 10:01:47.818 T:2882378   DEBUG <general>: Adding video for location Southern California to Baja to playlist
....
2021-03-14 10:01:47.821 T:2882378   DEBUG <general>: Video available locally, path is: /Users/kz/Desktop/aerials/comp_A103_C002_0205DG_v12_SDR_FINAL_20180706_SDR_4K_HEVC.mov

graysky2 commented 3 years ago

I'm trying out the code now, thanks for all the work. Any guidance on Settings>Offline mode:

Enable offline mode (no video/JSON download) ... is the "no video" meaning no streaming?
Enable custom JSON (no updates from Apple) ... what is this option?

I think most users will simply want to download/cache the official videos. For that, should the first be checked and the 2nd unchecked?

graysky2 commented 3 years ago

I also noticed that the download feature does not check for write permissions to the path defined. The result is that the script attempts to download the content but doesn't write it to disk nor does it end in an error. Might be confusing to users.

kubedzero commented 3 years ago

Let's see if I can word this better:

Enable offline mode puts the entire addon into offline mode. When the playlist is being generated it will use the local copy of entries.json and will only play a video if it's already saved locally. So it disables both the JSON fetching from Apple as well as streaming videos from Apple.
Enable custom JSON is a flag to skip fetching new JSON from Apple. Since the checked-in JSON has been hand-edited to include extra videos (undersea, retired videos, etc), I wanted to provide a feature that allows users to still stream content from Apple but avoid overwriting the custom JSON. Furthermore, if this setting is enabled, a user could theoretically edit the JSON to include whatever video names/links they want and still be able to stream them

This gives us the following behaviors:

Offline mode disabled, custom json disabled: Each activation of Aerials, download new JSON from Apple and then stream videos
(Default) Offline mode disabled, custom json enabled: Skip downloading new JSON from Apple, using the local copy. However, videos can still be streamed. I set this to the default because I figured this is the most common desire: an extended list of videos, but no local storage needed to play them
Offline mode enabled, custom json disabled: No network calls made, so we'll use whatever local JSON and already-downloaded videos we have available
Offline mode enabled, custom json enabled: Same behavior as the above state.

My intention was to give people without the local storage available the option to use the custom JSON with extra videos. I figured that decoupling the settings entirely would be less useful than full offline mode, aka providing a "load local files only" option and a "don't get JSON updates" option. Instead, the Custom JSON setting acts as a kind of subset of the Offline Mode setting.

I guess we could make it a dropdown with a few different choices (the way DPMS is for example), or provide a path entry to a custom JSON. For this PR though, I figured those setting descriptions were sufficient to tell users that offline mode means full offline mode, and custom JSON means no overwriting/updates from Apple's copy of the JSON.

download feature does not check for write permissions to the path defined

Interesting, did it check for write permissions before? I don't see that in the master branch, am I missing something?

Might be confusing to users.

Agreed, though I am not entirely sure how to fix this (and provide proper error handling/CX). Another point of confusion:

Enter Aerials settings, go to Offline category
From default of blank, change the Download Folder to any directory
Click "download videos" at the bottom, and get the error "You need to define a download folder."
Expected behavior: Without needing to leave the settings popup, define or redefine a download folder path and then be able to download to that location

I realized that a user needs to push "OK" to save the download folder setting, and then re-enter the settings and retry the "download videos" option. I'm not sure how to fix this either

graysky2 commented 3 years ago

Thanks for the detailed explanation. Took me a few reads through to understand. I think to maximize the number of videos, use the custom JSON you hand-wrote. That should be 98 videos, yes?

kubedzero commented 3 years ago

Took me a few reads through to understand

Ah sorry, if you have recommendations on how to make it clearer I'm all ears. I tried to keep it succinct but clear enough to understand from the setting names, but I guess I missed the mark.

to maximize the number of videos, use the custom JSON you hand-wrote

Yes! This is the default setting (don't get JSON updates from Apple)

That should be 98 videos, yes?

That's what I see on my system, yes!

graysky2 commented 3 years ago

I wouldn't say you missed the mark, just that in general, the labels on the software should be very simple. I don't know if it's reasonable to expect users to understand the inner workings of the plugin, ie building playlists based on local files vs fetching from web-based. I think most would want to use all the videos so the language of "no updates from Apple" is confusing if disabling that option results in more options using your hand-curated list.

Maybe simplify the options: Enable offline mode (no video/JSON download) --> No streaming or Local files only Enable custom JSON (no updates from Apple) --> Include newest videos or Maximum number of videos

As to the number of videos, I went ahead and enabled both options options, then downloaded all videos again and ended up with 44.9 GiB/188 files. Seems to have pulled down both 2k and 4k videos. Interestingly, I did not enable the Enable 4k (default 1080p) setting. Perhaps that is just for display purposes/not download purposes. I did not see that any of the 4k content were played in kodi.log so I think that last statement is accurate.

For reference:

total 45G
-rw-r--r-- 1 kodi kodi 141M Mar 14 20:05 AK_A004_C012_SDR_20191217_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 285M Mar 14 14:09 AK_A004_C012_SDR_20191217_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  84M Mar 14 14:09 b1-3.mov
-rw-r--r-- 1 kodi kodi 275M Mar 14 14:07 b2-2.mov
-rw-r--r-- 1 kodi kodi  44M Mar 14 14:09 b2-4.mov
-rw-r--r-- 1 kodi kodi 320M Mar 14 14:09 b4-3.mov
-rw-r--r-- 1 kodi kodi 258M Mar 14 14:09 b5-3.mov
-rw-r--r-- 1 kodi kodi  89M Mar 14 14:09 b6-4.mov
-rw-r--r-- 1 kodi kodi 277M Mar 14 14:09 b8-2.mov
-rw-r--r-- 1 kodi kodi 184M Mar 14 14:09 b8-3.mov
-rw-r--r-- 1 kodi kodi 159M Mar 14 20:07 BO_A012_C031_SDR_20190726_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 303M Mar 14 14:13 BO_A012_C031_SDR_20190726_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 177M Mar 14 20:06 BO_A014_C008_SDR_20190719_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 355M Mar 14 14:10 BO_A014_C008_SDR_20190719_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  86M Mar 14 20:07 BO_A014_C023_SDR_20190717_F240F3709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 168M Mar 14 14:11 BO_A014_C023_SDR_20190717_F240F3709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 101M Mar 14 20:06 BO_A018_C029_SDR_20190812_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 201M Mar 14 14:10 BO_A018_C029_SDR_20190812_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  97M Mar 14 20:02 comp_1223LV_FLARE_v21_SDR_PS_FINAL_20180709_F0F5700_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 188M Mar 14 14:03 comp_1223LV_FLARE_v21_SDR_PS_FINAL_20180709_F0F5700_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 108M Mar 14 20:05 comp_A001_C001_120530_v04_SDR_FINAL_20180706_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 206M Mar 14 14:08 comp_A001_C001_120530_v04_SDR_FINAL_20180706_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 130M Mar 14 20:02 comp_A001_C004_1207W5_v23_SDR_FINAL_20180706_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 258M Mar 14 14:02 comp_A001_C004_1207W5_v23_SDR_FINAL_20180706_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 120M Mar 14 20:01 comp_A006_C003_1219EE_CC_v01_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 238M Mar 14 14:00 comp_A006_C003_1219EE_CC_v01_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 119M Mar 14 20:03 comp_A007_C017_01156B_v02_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 231M Mar 14 14:04 comp_A007_C017_01156B_v02_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 166M Mar 14 20:04 comp_A008_C007_011550_CC_v01_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 334M Mar 14 14:07 comp_A008_C007_011550_CC_v01_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 166M Mar 14 20:01 comp_A009_C001_010181A_v09_SDR_PS_FINAL_20180725_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 331M Mar 14 14:01 comp_A009_C001_010181A_v09_SDR_PS_FINAL_20180725_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 187M Mar 14 20:00 comp_A012_C014_1223PT_v53_SDR_PS_FINAL_20180709_F0F8700_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 368M Mar 14 13:59 comp_A012_C014_1223PT_v53_SDR_PS_FINAL_20180709_F0F8700_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 154M Mar 14 20:03 comp_A013_C012_0122D6_CC_v01_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 300M Mar 14 14:04 comp_A013_C012_0122D6_CC_v01_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 119M Mar 14 19:58 comp_A015_C018_0128ZS_v03_SDR_PS_FINAL_20180709__SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 235M Mar 14 13:57 comp_A015_C018_0128ZS_v03_SDR_PS_FINAL_20180709__SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  44M Mar 14 20:04 comp_A083_C002_1130KZ_v04_SDR_PS_FINAL_20180725_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi  88M Mar 14 14:06 comp_A083_C002_1130KZ_v04_SDR_PS_FINAL_20180725_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 164M Mar 14 20:00 comp_A103_C002_0205DG_v12_SDR_FINAL_20180706_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 321M Mar 14 14:00 comp_A103_C002_0205DG_v12_SDR_FINAL_20180706_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 103M Mar 14 20:05 comp_A105_C002_v06_SDR_FINAL_25062018_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 209M Mar 14 14:08 comp_A105_C002_v06_SDR_FINAL_25062018_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 105M Mar 14 20:00 comp_A105_C003_0212CT_FLARE_v10_SDR_PS_FINAL_20180711_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 209M Mar 14 13:59 comp_A105_C003_0212CT_FLARE_v10_SDR_PS_FINAL_20180711_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 101M Mar 14 19:57 comp_A108_C001_v09_SDR_FINAL_22062018_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 202M Mar 14 13:58 comp_A108_C001_v09_SDR_FINAL_22062018_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 148M Mar 14 19:58 comp_A114_C001_0305OT_v10_SDR_FINAL_22062018_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 297M Mar 14 13:56 comp_A114_C001_0305OT_v10_SDR_FINAL_22062018_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 238M Mar 14 20:05 comp_A351_C001_v06_SDR_PS_20180725_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 474M Mar 14 14:09 comp_A351_C001_v06_SDR_PS_20180725_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 165M Mar 14 20:04 comp_C001_C005_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 327M Mar 14 14:06 comp_C001_C005_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 124M Mar 14 20:01 comp_C003_C003_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 238M Mar 14 14:00 comp_C003_C003_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 141M Mar 14 20:04 comp_C004_C003_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 287M Mar 14 14:07 comp_C004_C003_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 101M Mar 14 20:01 comp_CH_C002_C005_PSNK_v05_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 205M Mar 14 14:00 comp_CH_C002_C005_PSNK_v05_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 235M Mar 14 20:03 comp_CH_C007_C004_PSNK_v02_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 474M Mar 14 14:05 comp_CH_C007_C004_PSNK_v02_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 240M Mar 14 20:00 comp_CH_C007_C011_PSNK_v02_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 475M Mar 14 14:00 comp_CH_C007_C011_PSNK_v02_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 236M Mar 14 20:02 comp_DB_D001_C001_PSNK_v06_SDR_PS_20180824_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 469M Mar 14 14:02 comp_DB_D001_C001_PSNK_v06_SDR_PS_20180824_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 221M Mar 14 20:04 comp_DB_D001_C005_COMP_PSNK_v12_SDR_PS_20180912_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 447M Mar 14 14:06 comp_DB_D001_C005_COMP_PSNK_v12_SDR_PS_20180912_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 180M Mar 14 20:03 comp_DB_D002_C003_PSNK_v04_SDR_PS_20180914_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 360M Mar 14 14:04 comp_DB_D002_C003_PSNK_v04_SDR_PS_20180914_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 264M Mar 14 20:02 comp_DB_D008_C010_PSNK_v21_SDR_PS_20180914_F0F16157_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 530M Mar 14 14:01 comp_DB_D008_C010_PSNK_v21_SDR_PS_20180914_F0F16157_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 180M Mar 14 19:58 comp_DB_D011_C010_PSNK_DENOISE_v19_SDR_PS_20180914_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 355M Mar 14 13:56 comp_DB_D011_C010_PSNK_DENOISE_v19_SDR_PS_20180914_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 294M Mar 14 20:01 comp_GL_G002_C002_PSNK_v03_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 588M Mar 14 14:01 comp_GL_G002_C002_PSNK_v03_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 140M Mar 14 19:58 comp_GL_G004_C010_PSNK_v04_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 279M Mar 14 13:56 comp_GL_G004_C010_PSNK_v04_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 291M Mar 14 19:59 comp_GL_G010_C006_PSNK_NOSUN_v12_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 623M Mar 14 13:58 comp_GL_G010_C006_PSNK_NOSUN_v12_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 139M Mar 14 19:58 comp_GMT026_363A_103NC_E1027_KOREA_JAPAN_NIGHT_v18_SDR_PS_20180907_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 274M Mar 14 13:57 comp_GMT026_363A_103NC_E1027_KOREA_JAPAN_NIGHT_v18_SDR_PS_20180907_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 175M Mar 14 20:05 comp_GMT060_117NC_363D_1034_AUSTRALIA_v35_SDR_PS_FINAL_20180731_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 345M Mar 14 14:08 comp_GMT060_117NC_363D_1034_AUSTRALIA_v35_SDR_PS_FINAL_20180731_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  67M Mar 14 20:01 comp_GMT110_112NC_364D_1054_AURORA_ANTARTICA__COMP_FINAL_v34_PS_SDR_20181107_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 136M Mar 14 14:00 comp_GMT110_112NC_364D_1054_AURORA_ANTARTICA__COMP_FINAL_v34_PS_SDR_20181107_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  91M Mar 14 20:05 comp_GMT306_139NC_139J_3066_CALI_TO_VEGAS_v07_SDR_FINAL_22062018_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 176M Mar 14 14:08 comp_GMT306_139NC_139J_3066_CALI_TO_VEGAS_v07_SDR_FINAL_22062018_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  91M Mar 14 19:59 comp_GMT306_139NC_139J_3066_CALI_TO_VEGAS_v08_SDR_PS_20180824_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 177M Mar 14 13:59 comp_GMT306_139NC_139J_3066_CALI_TO_VEGAS_v08_SDR_PS_20180824_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 163M Mar 14 20:02 comp_GMT307_136NC_134K_8277_NY_NIGHT_01_v25_SDR_PS_20180907_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 324M Mar 14 14:02 comp_GMT307_136NC_134K_8277_NY_NIGHT_01_v25_SDR_PS_20180907_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 175M Mar 14 19:57 comp_GMT308_139K_142NC_CARIBBEAN_DAY_v09_SDR_FINAL_22062018_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 351M Mar 14 14:04 comp_GMT308_139K_142NC_CARIBBEAN_DAY_v09_SDR_FINAL_22062018_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 120M Mar 14 20:04 comp_GMT312_162NC_139M_1041_AFRICA_NIGHT_v14_SDR_FINAL_20180706_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 247M Mar 14 13:53 comp_GMT312_162NC_139M_1041_AFRICA_NIGHT_v14_SDR_FINAL_20180706_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 154M Mar 14 20:03 comp_GMT314_139M_170NC_NORTH_AMERICA_AURORA__COMP_v22_SDR_20181206_v12CC_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 311M Mar 14 14:05 comp_GMT314_139M_170NC_NORTH_AMERICA_AURORA__COMP_v22_SDR_20181206_v12CC_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  30M Mar 14 20:03 comp_GMT329_113NC_396B_1105_CHINA_v04_SDR_FINAL_20180706_F900F2700_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi  60M Mar 14 14:04 comp_GMT329_113NC_396B_1105_CHINA_v04_SDR_FINAL_20180706_F900F2700_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  65M Mar 14 19:58 comp_GMT329_113NC_396B_1105_ITALY_v03_SDR_FINAL_20180706_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 136M Mar 14 13:57 comp_GMT329_113NC_396B_1105_ITALY_v03_SDR_FINAL_20180706_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 106M Mar 14 20:00 comp_GMT329_117NC_401C_1037_IRELAND_TO_ASIA_v48_SDR_PS_FINAL_20180725_F0F6300_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 212M Mar 14 14:00 comp_GMT329_117NC_401C_1037_IRELAND_TO_ASIA_v48_SDR_PS_FINAL_20180725_F0F6300_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 117M Mar 14 20:01 comp_H004_C007_PS_v02_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 236M Mar 14 14:01 comp_H004_C007_PS_v02_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 119M Mar 14 20:04 comp_H004_C009_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 233M Mar 14 14:06 comp_H004_C009_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 166M Mar 14 20:02 comp_H005_C012_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 331M Mar 14 14:02 comp_H005_C012_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 119M Mar 14 20:04 comp_H007_C003_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 236M Mar 14 14:05 comp_H007_C003_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 115M Mar 14 20:03 comp_H012_C009_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 231M Mar 14 14:04 comp_H012_C009_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 160M Mar 14 19:58 comp_HK_B005_C011_PSNK_v16_SDR_PS_20180914_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 319M Mar 14 13:57 comp_HK_B005_C011_PSNK_v16_SDR_PS_20180914_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 391M Mar 14 20:05 comp_HK_H004_C001_PSNK_DENOISE_v14_SDR_PS_FINAL_20180731_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 781M Mar 14 14:08 comp_HK_H004_C001_PSNK_DENOISE_v14_SDR_PS_FINAL_20180731_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 322M Mar 14 20:04 comp_HK_H004_C008_PSNK_v19_SDR_PS_20180914_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 627M Mar 14 14:07 comp_HK_H004_C008_PSNK_v19_SDR_PS_20180914_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 206M Mar 14 19:59 comp_HK_H004_C010_PSNK_v08_SDR_PS_20181009_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 416M Mar 14 13:59 comp_HK_H004_C010_PSNK_v08_SDR_PS_20181009_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 167M Mar 14 20:00 comp_L004_C011_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 333M Mar 14 14:00 comp_L004_C011_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 179M Mar 14 19:58 comp_L007_C007_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 357M Mar 14 13:57 comp_L007_C007_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 166M Mar 14 20:03 comp_L010_C006_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 332M Mar 14 14:05 comp_L010_C006_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 143M Mar 14 20:02 comp_L012_c002_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 280M Mar 14 14:02 comp_L012_c002_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 104M Mar 14 20:01 comp_LA_A005_C009_PSNK_ALT_v09_SDR_PS_201809134_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 204M Mar 14 14:00 comp_LA_A005_C009_PSNK_ALT_v09_SDR_PS_201809134_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 134M Mar 14 20:02 comp_LA_A006_C004_v01_SDR_FINAL_PS_20180730_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 281M Mar 14 14:02 comp_LA_A006_C004_v01_SDR_FINAL_PS_20180730_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 175M Mar 14 19:58 comp_LA_A006_C008_PSNK_ALL_LOGOS_v10_SDR_PS_FINAL_20180801_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 352M Mar 14 13:57 comp_LA_A006_C008_PSNK_ALL_LOGOS_v10_SDR_PS_FINAL_20180801_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 392M Mar 14 19:59 comp_LA_A008_C004_ALTB_ED_FROM_FLAME_RETIME_v46_SDR_PS_20180917_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 786M Mar 14 13:58 comp_LA_A008_C004_ALTB_ED_FROM_FLAME_RETIME_v46_SDR_PS_20180917_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  97M Mar 14 20:03 comp_LA_A009_C009_PSNK_v02_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 227M Mar 14 14:05 comp_LA_A009_C009_PSNK_v02_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 256M Mar 14 20:03 comp_LA_A011_C003_DGRN_LNFIX_STAB_v57_SDR_PS_20181002_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 504M Mar 14 14:03 comp_LA_A011_C003_DGRN_LNFIX_STAB_v57_SDR_PS_20181002_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  90M Mar 14 20:03 comp_LW_L001_C003__PSNK_DENOISE_v04_SDR_PS_FINAL_20180803_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 180M Mar 14 14:04 comp_LW_L001_C003__PSNK_DENOISE_v04_SDR_PS_FINAL_20180803_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 210M Mar 14 20:04 comp_LW_L001_C006_PSNK_DENOISE_v02_SDR_PS_FINAL_20180709_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 409M Mar 14 14:06 comp_LW_L001_C006_PSNK_DENOISE_v02_SDR_PS_FINAL_20180709_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 119M Mar 14 20:04 comp_N003_C006_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 239M Mar 14 14:07 comp_N003_C006_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 118M Mar 14 19:58 comp_N008_C003_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 228M Mar 14 13:56 comp_N008_C003_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 235M Mar 14 19:59 comp_N008_C009_PS_v01_SDR_PS_20180925_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 471M Mar 14 13:59 comp_N008_C009_PS_v01_SDR_PS_20180925_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  84M Mar 14 19:59 comp_N013_C004_PS_v01_SDR_PS_20180925_F1970F7193_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 169M Mar 14 13:59 comp_N013_C004_PS_v01_SDR_PS_20180925_F1970F7193_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 137M Mar 14 20:06 CR_A009_C007_SDR_20191113_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 267M Mar 14 14:11 CR_A009_C007_SDR_20191113_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 167M Mar 14 20:06 DL_B002_C011_SDR_20191122_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 341M Mar 14 14:10 DL_B002_C011_SDR_20191122_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 207M Mar 14 20:06 FK_U009_C004_SDR_20191220_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 414M Mar 14 14:11 FK_U009_C004_SDR_20191220_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 288M Mar 14 20:06 g201_AK_A003_C014_SDR_20191113_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 568M Mar 14 14:10 g201_AK_A003_C014_SDR_20191113_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 224M Mar 14 20:06 g201_CA_A016_C002_SDR_20191114_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 446M Mar 14 14:10 g201_CA_A016_C002_SDR_20191114_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  72M Mar 14 20:08 g201_TH_803_A001_8_SDR_20191031_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 143M Mar 14 14:13 g201_TH_803_A001_8_SDR_20191031_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi  64M Mar 14 20:08 g201_TH_804_A001_8_SDR_20191031_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 120M Mar 14 14:13 g201_TH_804_A001_8_SDR_20191031_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 158M Mar 14 20:06 g201_WH_D004_L014_SDR_20191031_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 315M Mar 14 14:11 g201_WH_D004_L014_SDR_20191031_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 320M Mar 14 20:03 HK_H004_C013_2K_SDR_HEVC.mov
-rw-r--r-- 1 kodi kodi 646M Mar 14 14:03 HK_H004_C013_4K_SDR_HEVC.mov
-rw-r--r-- 1 kodi kodi 179M Mar 14 20:07 KP_A010_C002_SDR_20190717_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 356M Mar 14 14:11 KP_A010_C002_SDR_20190717_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 177M Mar 14 20:06 MEX_A006_C008_SDR_20190923_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 354M Mar 14 14:11 MEX_A006_C008_SDR_20190923_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 318M Mar 14 20:07 PA_A001_C007_SDR_20190717_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 637M Mar 14 14:12 PA_A001_C007_SDR_20190717_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 389M Mar 14 20:07 PA_A002_C009_SDR_20190730_ALT01_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 769M Mar 14 14:12 PA_A002_C009_SDR_20190730_ALT01_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 151M Mar 14 20:07 PA_A004_C003_SDR_20190719_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 298M Mar 14 14:12 PA_A004_C003_SDR_20190719_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 157M Mar 14 20:07 PA_A010_C007_SDR_20190717_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 313M Mar 14 14:12 PA_A010_C007_SDR_20190717_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 177M Mar 14 20:07 RS_A008_C010_SDR_20191218_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 355M Mar 14 14:12 RS_A008_C010_SDR_20191218_SDR_4K_HEVC.mov
-rw-r--r-- 1 kodi kodi 165M Mar 14 20:08 SE_A016_C009_SDR_20190717_SDR_2K_HEVC.mov
-rw-r--r-- 1 kodi kodi 331M Mar 14 14:13 SE_A016_C009_SDR_20190717_SDR_4K_HEVC.mov

kubedzero commented 3 years ago

in general, the labels on the software should be very simple

I would actually hope that a toggle is labeled well enough so that a user can determine the setting's impact on the behavior of the system. (Maybe I've gotten too used to the verbosity of JetBrains' settings menus though) Especially without supporting documentation, the addon's setting names are our only chance to communicate what's going on when you toggle a setting on or off.

I worry that No Streaming is opaque enough that the true behavior isn't made clear, namely that users are looking for an airplane/offline mode due to data caps or slow internet or whatever else. Local Files Only obscures the distinction between this setting and the one below, since this one is for videos+JSON and the one below is for JSON only.

Include newest videos or Maximum number of videos is accurate at this particular point in time, but what happens if a user disables the setting, gets updates from Apple, and then enables the setting again? They've lost the custom JSON that ships with the addon, so these setting descriptors no longer apply :( Another use case I imagined was that users can at any point enable the setting and then mess with the JSON however they see fit. For example, they could remove videos and/or resolutions they find unappealing. Or point to custom domains/videos. Or if at some point they mess up the JSON they can disable, get the update from Apple, and be back to a safe spot. I figured the best description would be to state exactly what it does: preventing the addon from fetching new JSON from Apple, and allowing you to preserve whatever custom JSON you want. Sure the default has extra videos and newer copies of videos, but I see that as a bonus rather than the core purpose.

I guess maybe another way of going about this could be to hide the offline mode and other settings behind expert-level settings in Kodi? I think all the settings are basic/0 right now but if we elevated that to expert/3 maybe that could help. https://kodi.wiki/view/Add-on_settings_conversion#Level

We could also restructure it to always keep a copy of the custom entries.json, or define a freeform entry where if blank, use Apple, otherwise point to the location of a JSON file to use which defaults to the one in the addon directory. Lots of ways to tackle this I suppose!

I went ahead and enabled both options options, then downloaded all videos again and ended up with 44.9 GiB/188 files. Seems to have pulled down both 2k and 4k videos.

That's super weird, and I wasn't able to reproduce it on Windows. Maybe there were leftovers from a previous run of the downloader? The offline/downloader code uses the same settings as the normal playlist code to determine which version to download. First, a step to read the settings and then a step to actually get the URL of the preferred quality (or a fallback quality if the preferred quality isn't available)

The only reason I can think that it might have downloaded multiple was that 4K/SDR is the default and a download was run before the settings saved. We see similar behavior with choosing the download path, where it thinks the path is empty until you push OK and then re-enter the settings menu. In other words, the settings dialog has to be exited for the modifications to actually be readable. I think that's due to having the ability to push cancel/defaults on the right side of the dialog, so the settings are volatile until you confirm you want them saved. Maybe something similar is happening with the quality toggle, where 4K/SDR (the default) is the stored setting value until you push OK and exit/re-enter? The file modification times seem to back this up as well, where the 4K/SDR stuff downloaded around 2pm while the 1080p/SDR stuff downloaded at 8pm.

If you could try reproducing this behavior and listing what you did to make it happen, I can maybe look into it more!

enen92 commented 3 years ago

This seems to work fine, so I'll merge. But will work a bit on the settings as I find them confusing. Also things like hevc or 4k can't be enabled by default. Most of kodi devices are really low end

enen92 commented 3 years ago

Submitted in https://github.com/xbmc/repo-scripts/pull/1939 Thanks a lot for your contribution @kubedzero . Highly appreciated!

enen92 / screensaver.atv4