search

jmbannon / ytdl-sub

Lightweight tool to automate downloading and metadata generation with yt-dlp
https://ytdl-sub.readthedocs.io
GNU General Public License v3.0
1.85k stars 71 forks source link

source_thumbnails not saving thumbnails from source as expected #482

Closed tyzbit closed 1 year ago

tyzbit commented 1 year ago

I am trying to configure ytdl-sub to download the avatar_uncropped thumbnail from the channel that created the playlist and save it as poster.jpg but neither playlist_thumbnails or source_thumbnails work for me.

Image I'm using: ghcr.io/jmbannon/ytdl-sub@sha256:9f965b16a2dd0f35f89e30dee741488d0a736f434641fba1a5c02c7426a66d96 (latest as of opening this issue)

config.yaml

configuration:
  working_directory: "/tmp/ytdl-sub"

presets:
  yt_video:
    download:
      download_strategy: "url"

    ytdl_options:
      format: "bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b"
      ignoreerrors: True
      break_on_existing: False
      break_on_reject: False
      final_ext: mp4
      cachedir: "/tmp/ytdl-sub"

    date_range:
      before: "{download_before}"
      after: "{download_after}"

    output_options:
      maintain_download_archive: True
      output_directory: "/media/youtube/videos"
      file_name: "{video_path}.{ext}"
      thumbnail_name: "{video_path}-thumb.{thumbnail_ext}"
      keep_files_after: "{download_after}"

    overrides:
      video_name: "{title_sanitized}-{upload_date}"
      video_type: "{channel}"
      video_path: "{video_type_sanitized}/{video_name}"
      download_before: "now"
      download_after: "19000101"

    video_tags:
      tags:
        title: "{title}"
        date: "{upload_date}"
        description: "{description}"

  yt_playlist:
    preset:
      - yt_video
      - season_by_year__episode_by_download_index

    download:
      download_strategy: "url"
      source_thumbnails:
        - name: "poster.jpg"
          uid: "avatar_uncropped"
        - name: "banner.jpg"
          uid: "banner_uncropped"

    output_options:
      output_directory: "/media/youtube/playlists/{tv_show_name_sanitized}"
      file_name: "{episode_path}.{ext}"
      thumbnail_name: "{episode_path}-thumb.{thumbnail_ext}"

    overrides:
      tv_show_name: "OVERRIDE IN SUBSCRIPTION"
      date: "{upload_date}"
      episode: "{download_index}"
      season: "00"
      episode_padded: "{download_index_padded6}"
      episode_file_name: "S{season}E{episode_padded} - {title_sanitized}"
      episode_path: "{episode_file_name}"

    video_tags:
      tags:
        episode: "{episode}"
        season: "{season_number}"

subscriptions.yaml

roblox:
  preset: yt_playlist
  overrides:
    tv_show_name: Roblox
  download:
    url: "https://www.youtube.com/playlist?list=PLWh_Fb1QWWg0maNyHke_4ltNXsl4Hsm5V"
Logs (click to expand) 
$ docker-compose up
Starting ytdl-sub ... done
Attaching to ytdl-sub
ytdl-sub    | [migrations] started
ytdl-sub    | [migrations] no migrations found
ytdl-sub    | cont-init: info: running /etc/cont-init.d/30-config
ytdl-sub    | usermod: no changes
ytdl-sub    |
ytdl-sub    | -------------------------------------
ytdl-sub    |           _         ()
ytdl-sub    |          | |  ___   _    __
ytdl-sub    |          | | / __| | |  /  \
ytdl-sub    |          | | \__ \ | | | () |
ytdl-sub    |          |_| |___/ |_|  \__/
ytdl-sub    |
ytdl-sub    |
ytdl-sub    | Brought to you by linuxserver.io
ytdl-sub    | -------------------------------------
ytdl-sub    |
ytdl-sub    | To support LSIO projects visit:
ytdl-sub    | https://www.linuxserver.io/donate/
ytdl-sub    | -------------------------------------
ytdl-sub    | GID/UID
ytdl-sub    | -------------------------------------
ytdl-sub    |
ytdl-sub    | User uid:    1000
ytdl-sub    | User gid:    1000
ytdl-sub    | -------------------------------------
ytdl-sub    |
ytdl-sub    | cont-init: info: /etc/cont-init.d/30-config exited 0
ytdl-sub    | [custom-init] No custom files found, skipping...
ytdl-sub    | [ls.io-init] done.
ytdl-sub    | [ytdl-sub] Beginning subscription download for roblox
ytdl-sub    | [ytdl-sub] Subscription full yaml:
ytdl-sub    | presets:
ytdl-sub    |   roblox:
ytdl-sub    |     date_range:
ytdl-sub    |       after: '{download_after}'
ytdl-sub    |       before: '{download_before}'
ytdl-sub    |     download:
ytdl-sub    |       source_thumbnails:
ytdl-sub    |       - name: poster.jpg
ytdl-sub    |         uid: avatar_uncropped
ytdl-sub    |       - name: banner.jpg
ytdl-sub    |         uid: banner_uncropped
ytdl-sub    |       url: https://www.youtube.com/playlist?list=PLWh_Fb1QWWg0maNyHke_4ltNXsl4Hsm5V
ytdl-sub    |     output_options:
ytdl-sub    |       file_name: '{episode_path}.{ext}'
ytdl-sub    |       keep_files_after: '{download_after}'
ytdl-sub    |       maintain_download_archive: true
ytdl-sub    |       output_directory: /media/youtube/playlists/{tv_show_name_sanitized}
ytdl-sub    |       thumbnail_name: '{episode_path}-thumb.{thumbnail_ext}'
ytdl-sub    |     overrides:
ytdl-sub    |       date: '{upload_date}'
ytdl-sub    |       download_after: '19000101'
ytdl-sub    |       download_before: now
ytdl-sub    |       episode: '{download_index}'
ytdl-sub    |       episode_file_name: S{season}E{episode_padded} - {title_sanitized}
ytdl-sub    |       episode_number: '{download_index}'
ytdl-sub    |       episode_number_padded: '{download_index_padded6}'
ytdl-sub    |       episode_padded: '{download_index_padded6}'
ytdl-sub    |       episode_path: '{episode_file_name}'
ytdl-sub    |       season: '00'
ytdl-sub    |       season_number: '{upload_year}'
ytdl-sub    |       season_number_padded: '{season_number}'
ytdl-sub    |       tv_show_name: Roblox
ytdl-sub    |       video_name: '{title_sanitized}-{upload_date}'
ytdl-sub    |       video_path: '{video_type_sanitized}/{video_name}'
ytdl-sub    |       video_type: '{channel}'
ytdl-sub    |     preset: yt_playlist
ytdl-sub    |     video_tags:
ytdl-sub    |       tags:
ytdl-sub    |         date: '{upload_date}'
ytdl-sub    |         description: '{description}'
ytdl-sub    |         episode: '{episode}'
ytdl-sub    |         season: '{season_number}'
ytdl-sub    |         title: '{title}'
ytdl-sub    |     ytdl_options:
ytdl-sub    |       break_on_existing: false
ytdl-sub    |       break_on_reject: false
ytdl-sub    |       cachedir: /tmp/ytdl-sub
ytdl-sub    |       final_ext: mp4
ytdl-sub    |       format: bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b
ytdl-sub    |       ignoreerrors: true
ytdl-sub    |
ytdl-sub    | [ytdl-sub:downloader] ytdl_options: {'ignoreerrors': True, 'outtmpl': '/tmp/ytdl-sub/roblox/%(id)s.%(ext)s', 'writethumbnail': False, 'download_archive': '/tmp/ytdl-sub/roblox/.ytdl-sub-roblox-download-archive.txt', 'daterange': , 'format': 'bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b', 'break_on_existing': False, 'break_on_reject': False, 'final_ext': 'mp4', 'cachedir': '/tmp/ytdl-sub', 'skip_download': True, 'writeinfojson': True}
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube:tab] Extracting URL: https://www.youtube.com/playlist?list=PLWh_Fb1QWWg0maNyHke_4ltNXsl4Hsm5V
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube:tab] PLWh_Fb1QWWg0maNyHke_4ltNXsl4Hsm5V: Downloading webpage
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube:tab] PLWh_Fb1QWWg0maNyHke_4ltNXsl4Hsm5V: Redownloading playlist API JSON with unavailable videos
ytdl-sub    | [ytdl-sub:yt-dlp] [download] Downloading playlist: roblox death sound
ytdl-sub    | [ytdl-sub:yt-dlp] [info] Writing playlist metadata as JSON to: /tmp/ytdl-sub/roblox/PLWh_Fb1QWWg0maNyHke_4ltNXsl4Hsm5V.info.json
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube:tab] Playlist roblox death sound: Downloading 1 items of 1
ytdl-sub    | [ytdl-sub:yt-dlp] [download] Downloading item 1 of 1
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube] Extracting URL: https://www.youtube.com/watch?v=f49ELvryhao
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube] f49ELvryhao: Downloading webpage
ytdl-sub    | [ytdl-sub:downloader] Downloading metadata for roblox death sound
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube] f49ELvryhao: Downloading android player API JSON
ytdl-sub    | [ytdl-sub:yt-dlp] [info] f49ELvryhao: Downloading 1 format(s): 136+140
ytdl-sub    | [ytdl-sub:yt-dlp] [info] Writing video metadata as JSON to: /tmp/ytdl-sub/roblox/f49ELvryhao.info.json
ytdl-sub    | [ytdl-sub:yt-dlp] [info] Writing updated playlist metadata as JSON to: /tmp/ytdl-sub/roblox/PLWh_Fb1QWWg0maNyHke_4ltNXsl4Hsm5V.info.json
ytdl-sub    | [ytdl-sub:yt-dlp] [download] Finished downloading playlist: roblox death sound
ytdl-sub    | [ytdl-sub:downloader] Beginning downloads for https://www.youtube.com/playlist?list=PLWh_Fb1QWWg0maNyHke_4ltNXsl4Hsm5V
ytdl-sub    | [ytdl-sub:downloader] Downloading entry 1/1: Roblox Death Sound Effect
ytdl-sub    | [ytdl-sub:downloader] ytdl_options: {'ignoreerrors': True, 'outtmpl': '/tmp/ytdl-sub/roblox/%(id)s.%(ext)s', 'writethumbnail': True, 'download_archive': '/tmp/ytdl-sub/roblox/.ytdl-sub-roblox-download-archive.txt', 'daterange': , 'format': 'bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b', 'break_on_existing': False, 'break_on_reject': False, 'final_ext': 'mp4', 'cachedir': '/tmp/ytdl-sub'}
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube] Extracting URL: https://www.youtube.com/watch?v=f49ELvryhao
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube] f49ELvryhao: Downloading webpage
ytdl-sub    | [ytdl-sub:yt-dlp] [youtube] f49ELvryhao: Downloading android player API JSON
ytdl-sub    | [ytdl-sub:yt-dlp] [info] f49ELvryhao: Downloading 1 format(s): 136+140
ytdl-sub    | [ytdl-sub:yt-dlp] [info] Downloading video thumbnail 41 ...
ytdl-sub    | [ytdl-sub:yt-dlp] WARNING: Unable to download video thumbnail 41: HTTP Error 404: Not Found
ytdl-sub    | [ytdl-sub:yt-dlp] [info] Downloading video thumbnail 40 ...
ytdl-sub    | [ytdl-sub:yt-dlp] WARNING: Unable to download video thumbnail 40: HTTP Error 404: Not Found
ytdl-sub    | [ytdl-sub:yt-dlp] [info] Downloading video thumbnail 39 ...
ytdl-sub    | [ytdl-sub:yt-dlp] WARNING: Unable to download video thumbnail 39: HTTP Error 404: Not Found
ytdl-sub    | [ytdl-sub:yt-dlp] [info] Downloading video thumbnail 38 ...
ytdl-sub    | [ytdl-sub:yt-dlp] WARNING: Unable to download video thumbnail 38: HTTP Error 404: Not Found
ytdl-sub    | [ytdl-sub:yt-dlp] [info] Downloading video thumbnail 37 ...
ytdl-sub    | [ytdl-sub:yt-dlp] [info] Writing video thumbnail 37 to: /tmp/ytdl-sub/roblox/f49ELvryhao.webp
ytdl-sub    | [ytdl-sub:yt-dlp] [download] Destination: /tmp/ytdl-sub/roblox/f49ELvryhao.f136.mp4
[download]  11.2% of    8.92KiB at    8.91KiB/s ETA 00:00
[download]  33.6% of    8.92KiB at   26.18KiB/s ETA 00:00
[download]  78.4% of    8.92KiB at   59.50KiB/s ETA 00:00
[download] 100.0% of    8.92KiB at   73.25KiB/s ETA 00:00
[download] 100% of    8.92KiB in 00:00:00 at 41.75KiB/s
ytdl-sub    | [ytdl-sub:yt-dlp] [download] Destination: /tmp/ytdl-sub/roblox/f49ELvryhao.f140.m4a
[download]   4.3% of   23.39KiB at    3.68KiB/s ETA 00:06
[download]  12.8% of   23.39KiB at   11.00KiB/s ETA 00:01
[download]  29.9% of   23.39KiB at   25.47KiB/s ETA 00:00
[download]  64.1% of   23.39KiB at   53.62KiB/s ETA 00:00
[download] 100.0% of   23.39KiB at   43.81KiB/s ETA 00:00
[download] 100% of   23.39KiB in 00:00:00 at 36.35KiB/s
ytdl-sub    | [ytdl-sub:yt-dlp] [Merger] Merging formats into "/tmp/ytdl-sub/roblox/f49ELvryhao.mp4"
ytdl-sub    | [ytdl-sub:yt-dlp] Deleting original file /tmp/ytdl-sub/roblox/f49ELvryhao.f136.mp4 (pass -k to keep)
ytdl-sub    | [ytdl-sub:yt-dlp] Deleting original file /tmp/ytdl-sub/roblox/f49ELvryhao.f140.m4a (pass -k to keep)
ytdl-sub    | [ytdl-sub:ffmpeg] Running ffmpeg -i /tmp/ytdl-sub/roblox/f49ELvryhao.mp4 -map 0 -metadata title=Roblox Death Sound Effect -metadata date=20160812 -metadata description=Suggested by: Jenny
ytdl-sub    | Download: https://audiograb.com/opNTQCf4R
ytdl-sub    | SUBSCRIBE TO MY MAIN CHANNEL - https://www.youtube.com/user/Satherax/videos
ytdl-sub    | Leave a suggestion in the comments for my next video :DD
ytdl-sub    | My twitter - https://twitter.com/Satherax -metadata episode=1 -metadata season=2016 -codec copy /tmp/ytdl-sub/roblox/f49ELvryhao.mp4.out.mp4
ytdl-sub    | [ytdl-sub:ffmpeg] Running ffmpeg -bitexact -i /tmp/ytdl-sub/roblox/f49ELvryhao.webp /tmp/ytdl-sub/roblox/f49ELvryhao.jpg
ytdl-sub    | [ytdl-sub] Downloads for roblox:
ytdl-sub    | Files created:
ytdl-sub    | ----------------------------------------
ytdl-sub    | /media/youtube/playlists/Roblox
ytdl-sub    |   .ytdl-sub-roblox-download-archive.json
ytdl-sub    |   S00E000001 - Roblox Death Sound Effect-thumb.jpg
ytdl-sub    |   S00E000001 - Roblox Death Sound Effect.mp4
ytdl-sub    |     Video Tags:
ytdl-sub    |       date: 20160812
ytdl-sub    |       description:
ytdl-sub    |         Suggested by: Jenny
ytdl-sub    |         Download: https://audiograb.com/opNTQCf4R
ytdl-sub    |         SUBSCRIBE TO MY MAIN CHANNEL - https://www.youtube.com/user/Satherax/videos
ytdl-sub    |         Leave a suggestion in the comments for my next video :DD
ytdl-sub    |         My twitter - https://twitter.com/Satherax
ytdl-sub    |       episode: 1
ytdl-sub    |       season: 2016
ytdl-sub    |       title: Roblox Death Sound Effect
ytdl-sub    |

$  ls -laR
total 24
drwxr-xr-x   7 tyzbit  staff   224 Feb 28 19:17 .
drwx------+ 38 tyzbit  staff  1216 Feb 28 18:00 ..
-rw-r--r--@  1 tyzbit  staff  6148 Feb 28 18:53 .DS_Store
drwxr-xr-x@  6 tyzbit  staff   192 Feb 27 22:51 config
-rw-r--r--   1 tyzbit  staff   487 Feb 27 22:26 docker-compose.yaml
drwxr-xr-x   3 tyzbit  staff    96 Feb 28 19:17 playlists
drwxr-xr-x   4 tyzbit  staff   128 Feb 28 19:17 tmp

./config:
total 32
drwxr-xr-x@ 6 tyzbit  staff   192 Feb 27 22:51 .
drwxr-xr-x  7 tyzbit  staff   224 Feb 28 19:17 ..
-rw-r--r--@ 1 tyzbit  staff  6148 Feb 27 22:32 .DS_Store
-rw-r--r--@ 1 tyzbit  staff  1897 Feb 28 19:17 config.yaml
drwxr-xr-x@ 9 tyzbit  staff   288 Feb 27 22:51 examples
-rw-r--r--@ 1 tyzbit  staff   164 Feb 28 18:50 subscriptions.yaml

./config/examples:
total 80
drwxr-xr-x@ 9 tyzbit  staff   288 Feb 27 22:51 .
drwxr-xr-x@ 6 tyzbit  staff   192 Feb 27 22:51 ..
-rw-r--r--@ 1 tyzbit  staff   385 Feb 27 22:51 README.md
-rw-r--r--@ 1 tyzbit  staff  7639 Feb 27 22:51 music_audio_config.yaml
-rw-r--r--@ 1 tyzbit  staff  5099 Feb 27 22:51 music_audio_subscriptions.yaml
-rw-r--r--@ 1 tyzbit  staff  2627 Feb 27 22:51 music_videos_config.yaml
-rw-r--r--@ 1 tyzbit  staff  1041 Feb 27 22:51 music_videos_subscriptions.yaml
-rw-r--r--@ 1 tyzbit  staff  4752 Feb 27 22:51 tv_show_config.yaml
-rw-r--r--@ 1 tyzbit  staff   524 Feb 27 22:51 tv_show_subscriptions.yaml

./playlists:
total 0
drwxr-xr-x  3 tyzbit  staff   96 Feb 28 19:17 .
drwxr-xr-x  7 tyzbit  staff  224 Feb 28 19:17 ..
drwxr-xr-x  5 tyzbit  staff  160 Feb 28 19:17 Roblox

./playlists/Roblox:
total 104
drwxr-xr-x  5 tyzbit  staff    160 Feb 28 19:17 .
drwxr-xr-x  3 tyzbit  staff     96 Feb 28 19:17 ..
-rw-r--r--  1 tyzbit  staff    222 Feb 28 19:17 .ytdl-sub-roblox-download-archive.json
-rw-r--r--  1 tyzbit  staff  11404 Feb 28 19:17 S00E000001 - Roblox Death Sound Effect-thumb.jpg
-rw-r--r--  1 tyzbit  staff  33818 Feb 28 19:17 S00E000001 - Roblox Death Sound Effect.mp4

./tmp:
total 0
drwxr-xr-x  4 tyzbit  staff  128 Feb 28 19:17 .
drwxr-xr-x  7 tyzbit  staff  224 Feb 28 19:17 ..
-rw-r--r--  1 tyzbit  staff    0 Feb 28 19:17 _tmp_ytdl-sub
drwxr-xr-x  2 tyzbit  staff   64 Feb 28 19:17 ytdl-sub

./tmp/ytdl-sub:
total 0
drwxr-xr-x  2 tyzbit  staff   64 Feb 28 19:17 .
drwxr-xr-x  4 tyzbit  staff  128 Feb 28 19:17 ..
jmbannon commented 1 year ago

AFAIK yt-dlp doesn't fetch the source (playlist creator) metadata. For playlists, it is recommended to use the playlist_thumbnail's latest_entry for artwork

jmbannon commented 1 year ago

Btw, you should use the prebuilt presets as your 'base' preset. They are battle-tested to make sure they work in your desired player. https://ytdl-sub.readthedocs.io/en/latest/presets.html#tv-shows

Closing the issue out but feel free to comment with any additional questions/concerns.

tyzbit commented 1 year ago

Thanks for the response, that's a bummer. So does source_thumbnails do nothing then? The docs seem to imply it does what I'm looking for.

Also, thank you for the preset tip. I'll check them out.

jmbannon commented 1 year ago

For playlist URLs it does nothing. For urls like: https://www.youtube.com/@TylerHawkins/playlists?view=1&sort=dd&shelf_id=0

Since it's pointing to a channel with multiple playlists, source will be the channel whereas playlist will be each individual playlist. You can run

ytdl-sub view "https://www.youtube.com/@TylerHawkins/playlists?view=1&sort=dd&shelf_id=0"

and see the playlist/source variables quickly

timbroder commented 1 year ago

@tyzbit what did you end up using for this? running into something similar

tyzbit commented 1 year ago

@timbroder I just don't have playlist thumbnails. I preferred to have none than to let one of the videos in the playlist provide the thumbnail in case I intentionally remove a video from my own playlists. It's not pretty but I still have full functionality.

Two ideas are that you can either drop thumbnails manually or you could add a script that runs after ytdl-sub that gets all the playlists from the config (if you write a shell script, yq would be your friend here) and does a simple GET for the predictable YT channel logo for the playlist then drops it in the right spot.

timbroder commented 1 year ago

@tyzbit thanks! I ended up making some quick ones with https://www.coverlabs.io/ and called it a day