search

jmbannon / ytdl-sub

Lightweight tool to automate downloading and metadata generation with yt-dlp
https://ytdl-sub.readthedocs.io
GNU General Public License v3.0
1.85k stars 71 forks source link

Invalid date format "now-6hour" #723

Open Ang-Andrew opened 1 year ago

Ang-Andrew commented 1 year ago

Looks like the underlying regex is broken for date_range. Here is my config:

  subscriptions:
    preset:
      - "video"
    date_range:
      after: "now-6hour"
    output_options:
      maintain_download_archive: True
      keep_files_after: "now-1week"
    overrides:
      video_type: "subscriptions"

Full output of log:

cat /tmp/ytdl-sub.08uuu_hq
[ytdl-sub] Beginning subscription dry run for A Life Engineered
[ytdl-sub] Subscription full yaml:
presets:
  A Life Engineered:
    date_range:
      after: now-6hour
    download: https://www.youtube.com/channel/UCEHFikgnRuLd1HYKTLrae9Q
    output_options:
      file_name: '{video_path}.{ext}'
      keep_files_after: now-1week
      maintain_download_archive: true
      output_directory: /youtube
      thumbnail_name: '{video_path}.{thumbnail_ext}'
    overrides:
      episode: '{upload_month}{upload_day_padded}'
      season: '{upload_year}'
      video_name: '{upload_date_standardized}.{title_sanitized}'
      video_path: '{video_type_sanitized}/{video_name}'
      video_type: subscriptions
    preset: subscriptions

[ytdl-sub] An uncaught error occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ytdl_sub/main.py", line 26, in main
    _main()
  File "/usr/lib/python3.11/site-packages/ytdl_sub/main.py", line 18, in _main
    ytdl_sub.cli.main.main()
  File "/usr/lib/python3.11/site-packages/ytdl_sub/cli/main.py", line 326, in main
    transaction_logs = _download_subscriptions_from_yaml_files(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/cli/main.py", line 110, in _download_subscriptions_from_yaml_files
    transaction_log = subscription.download(dry_run=dry_run)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_download.py", line 341, in download
    metadata_ytdl_options=subscription_ytdl_options.metadata_builder(),
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_ytdl_options.py", line 110, in metadata_builder
    self._plugin_ytdl_options(DateRangePlugin),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_ytdl_options.py", line 94, in _plugin_ytdl_options
    return audio_extract_plugin.ytdl_options()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/plugins/date_range.py", line 59, in ytdl_options
    source_date_range = to_date_range(
                        ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/utils/datetime.py", line 29, in to_date_range
    return DateRange(start=start, end=end)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 1810, in __init__
    self.start = date_from_str(start, strict=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 1763, in date_from_str
    raise ValueError(f'Invalid date format "{date_str}"')
ValueError: Invalid date format "now-6hour"
[ytdl-sub] Version 2023.09.15+0b62cc1
Please upload the error log file '/tmp/ytdl-sub.08uuu_hq' and make a Github issue at https://github.com/jmbannon/ytdl-sub/issues with your config and command/subscription yaml file to reproduce. Thanks for trying ytdl-sub!
jmbannon commented 1 year ago

try hours instead of hour

Ang-Andrew commented 1 year ago

try hours instead of hour

Hi there, looks like i'm running into the same error:

root@8ad4c4a637b2:/config# ytdl-sub --dry-run --config config.yaml sub 
subscriptions.yaml
[ytdl-sub] Beginning subscription dry run for A Life Engineered
[ytdl-sub] An uncaught error occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ytdl_sub/main.py", line 26, in main
    _main()
  File "/usr/lib/python3.11/site-packages/ytdl_sub/main.py", line 18, in _main
    ytdl_sub.cli.main.main()
  File "/usr/lib/python3.11/site-packages/ytdl_sub/cli/main.py", line 326, in main
    transaction_logs = _download_subscriptions_from_yaml_files(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/cli/main.py", line 110, in _download_subscriptions_from_yaml_files
    transaction_log = subscription.download(dry_run=dry_run)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_download.py", line 341, in download
    metadata_ytdl_options=subscription_ytdl_options.metadata_builder(),
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_ytdl_options.py", line 110, in metadata_builder
    self._plugin_ytdl_options(DateRangePlugin),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_ytdl_options.py", line 94, in _plugin_ytdl_options
    return audio_extract_plugin.ytdl_options()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/plugins/date_range.py", line 59, in ytdl_options
    source_date_range = to_date_range(
                        ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/utils/datetime.py", line 29, in to_date_range
    return DateRange(start=start, end=end)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 1810, in __init__
    self.start = date_from_str(start, strict=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 1763, in date_from_str
    raise ValueError(f'Invalid date format "{date_str}"')
ValueError: Invalid date format "now-6hours"
[ytdl-sub] Version 2023.09.15+0b62cc1
Please upload the error log file '/tmp/ytdl-sub.vwctfv_o' and make a Github issue at https://github.com/jmbannon/ytdl-sub/issues with your config and command/subscription yaml file to reproduce. Thanks for trying ytdl-sub!

tried with day/days and that works out. Looks like its hitting this function which doesn't filter on the hour as per the doc. Think this function is the intended string matching function of data_range 🤔

jmbannon commented 1 year ago

@Ang-Andrew It looks like it's passing the literal "{date_str}" into that field. Can you post your full config

Ang-Andrew commented 1 year ago

@Ang-Andrew It looks like it's passing the literal "{date_str}" into that field. Can you post your full config

Here are my configs

# config.yaml

configuration:
  working_directory: '.ytdl-sub-downloads'

presets:
  video:
    output_options:
      output_directory: "/youtube"
      file_name: "{video_path}.{ext}"
      thumbnail_name: "{video_path}.{thumbnail_ext}"

    overrides:
      video_name: "{upload_date_standardized}.{title_sanitized}"
      video_type: "{channel}"
      video_path: "{video_type_sanitized}/{video_name}"
      season: "{upload_year}"
      episode: "{upload_month}{upload_day_padded}"

  youtube:
    preset:
      - "video"

  watch-later:
    preset: "video"
    output_options:
      maintain_download_archive: True
      keep_files_after: "now-1week"
    overrides:
      video_type: "watch-later"

  playlist:
    preset:
      - "video"

  channel:
    preset:
      - "playlist"

  subscriptions:
    preset:
      - "video"
    date_range:
      after: "now-6hours"
    output_options:
      maintain_download_archive: True
      keep_files_after: "now-1week"
    overrides:
      video_type: "subscriptions"

And my subscriptions.yaml

# Bare-bones config. Here are some useful links to get started:
#   Walk-through Guide:  https://github.com/jmbannon/ytdl-sub/wiki/1.-Introduction
#   Config Examples:     https://github.com/jmbannon/ytdl-sub/tree/master/examples
#   Prebuilt Presets:    https://ytdl-sub.readthedocs.io/en/latest/presets.html
#   Config Docs:         https://ytdl-sub.readthedocs.io/en/latest/config.html
#
# Examples are included in the /config/examples/ directory.
# Any config and subscription can be ran using:
#   ytdl-sub --config /path/to/config.yaml sub /path/to/subscriptions.yaml
#
# Or dry-ran with:
#   ytdl-sub --dry-run --config /path/to/config.yaml sub /path/to/subscriptions.yaml
#

A Life Engineered:
  download: https://www.youtube.com/channel/UCEHFikgnRuLd1HYKTLrae9Q
  preset: subscriptions

Using dry run was enough for me to reproduce the issue. Appreciate the help!

jmbannon commented 1 year ago

@Ang-Andrew could you dry run and also add --log-level debug

jmbannon commented 1 year ago

Or reupload the error log file that it prints in the error message (that contains debug logs/

Ang-Andrew commented 1 year ago

Here's the with output with the log-level debug:

root@8ad4c4a637b2:/config# ytdl-sub --dry-run --config config.yaml sub subscriptions.yaml --log-level debug
[ytdl-sub] Beginning subscription dry run for A Life Engineered
[ytdl-sub] Subscription full yaml:
presets:
  A Life Engineered:
    date_range:
      after: now-1day
    download: https://www.youtube.com/channel/UCEHFikgnRuLd1HYKTLrae9Q
    output_options:
      file_name: '{video_path}.{ext}'
      keep_files_after: now-6hours
      maintain_download_archive: true
      output_directory: /youtube
      thumbnail_name: '{video_path}.{thumbnail_ext}'
    overrides:
      episode: '{upload_month}{upload_day_padded}'
      season: '{upload_year}'
      video_name: '{upload_date_standardized}.{title_sanitized}'
      video_path: '{video_type_sanitized}/{video_name}'
      video_type: subscriptions
    preset: subscriptions

[ytdl-sub:yt-dlp-downloader] ytdl_options: {'ignoreerrors': True, 'outtmpl': '.ytdl-sub-downloads/A Life Engineered/%(id)s.%(ext)s', 'writethumbnail': False, 'ffmpeg_location': '/usr/bin/ffmpeg', 'download_archive': '.ytdl-sub-downloads/A Life Engineered/.ytdl-sub-A Life Engineered-download-archive.json', 'daterange': yt_dlp.utils.DateRange('2023-09-19', '9999-12-31'), 'break_on_reject': True, 'skip_download': True, 'writeinfojson': True}
[ytdl-sub:yt-dlp] [youtube:tab] Extracting URL: https://www.youtube.com/channel/UCEHFikgnRuLd1HYKTLrae9Q
[ytdl-sub:yt-dlp] [youtube:tab] UCEHFikgnRuLd1HYKTLrae9Q: Downloading webpage
[ytdl-sub:yt-dlp] [youtube:tab] Downloading all uploads of the channel. To download only the videos in a specific tab, pass the tab's URL
[ytdl-sub:yt-dlp] [youtube:tab] UCEHFikgnRuLd1HYKTLrae9Q/shorts: Downloading webpage
[ytdl-sub:yt-dlp] [youtube:tab] Downloading as multiple playlists, separated by tabs. To download as a single playlist instead, pass https://www.youtube.com/playlist?list=UUEHFikgnRuLd1HYKTLrae9Q
[ytdl-sub:yt-dlp] [download] Downloading playlist: A Life Engineered
[ytdl-sub:yt-dlp] [info] Writing playlist metadata as JSON to: .ytdl-sub-downloads/A Life Engineered/UCEHFikgnRuLd1HYKTLrae9Q.info.json
[ytdl-sub:yt-dlp] [youtube:tab] Playlist A Life Engineered: Downloading 2 items of 2
[ytdl-sub:yt-dlp] [download] Downloading item 1 of 2
[ytdl-sub:yt-dlp] [download] Downloading playlist: A Life Engineered - Videos
[ytdl-sub:yt-dlp] [info] Writing playlist metadata as JSON to: .ytdl-sub-downloads/A Life Engineered/UCEHFikgnRuLd1HYKTLrae9Q.info.json
[ytdl-sub:downloader] Downloading metadata for A Life Engineered
[ytdl-sub:yt-dlp] [youtube:tab] Playlist A Life Engineered - Videos: Downloading 25 items of 25
[ytdl-sub:yt-dlp] [download] Downloading item 1 of 25
[ytdl-sub:yt-dlp] [youtube] Extracting URL: https://www.youtube.com/watch?v=24zp6pabmrk
[ytdl-sub:yt-dlp] [youtube] 24zp6pabmrk: Downloading webpage
[ytdl-sub:yt-dlp] [youtube] 24zp6pabmrk: Downloading ios player API JSON
[ytdl-sub:yt-dlp] [youtube] 24zp6pabmrk: Downloading android player API JSON
[ytdl-sub:yt-dlp] [youtube] 24zp6pabmrk: Downloading m3u8 information
[ytdl-sub:yt-dlp] [download] 2023-08-14 upload date is not in range yt_dlp.utils.DateRange('2023-09-19', '9999-12-31')
[ytdl-sub:yt-dlp-downloader] RejectedVideoReached, stopping additional downloads (Can be disable by setting `ytdl_options.break_on_reject` to False).
[ytdl-sub:downloader] Beginning downloads for https://www.youtube.com/channel/UCEHFikgnRuLd1HYKTLrae9Q
[ytdl-sub] An uncaught error occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ytdl_sub/main.py", line 26, in main
    _main()
  File "/usr/lib/python3.11/site-packages/ytdl_sub/main.py", line 18, in _main
    ytdl_sub.cli.main.main()
  File "/usr/lib/python3.11/site-packages/ytdl_sub/cli/main.py", line 326, in main
    transaction_logs = _download_subscriptions_from_yaml_files(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/cli/main.py", line 110, in _download_subscriptions_from_yaml_files
    transaction_log = subscription.download(dry_run=dry_run)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_download.py", line 347, in download
    return self._process_subscription(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_download.py", line 292, in _process_subscription
    with self._subscription_download_context_managers():
  File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_download.py", line 168, in _subscription_download_context_managers
    with (
  File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/lib/python3.11/site-packages/ytdl_sub/subscriptions/subscription_download.py", line 143, in _maintain_archive_file
    date_range_to_keep = to_date_range(
                         ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ytdl_sub/utils/datetime.py", line 29, in to_date_range
    return DateRange(start=start, end=end)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 1810, in __init__
    self.start = date_from_str(start, strict=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 1763, in date_from_str
    raise ValueError(f'Invalid date format "{date_str}"')
ValueError: Invalid date format "now-6hours"
[ytdl-sub] Version 2023.09.15+0b62cc1
Please upload the error log file '/tmp/ytdl-sub.evo8fgo5' and make a Github issue at https://github.com/jmbannon/ytdl-sub/issues with your config and command/subscription yaml file to reproduce. Thanks for trying ytdl-sub!