Log progress while downloading

[deldesir]

Is your feature request related to a problem? Please describe. When using lb tubeadd and lb dl with --verbose option, I wished yt_dlp output weren't supressed. This makes it difficult to track the progress of video downloads.

Describe the solution you'd like I would like the ability to see the progress of video downloads. This could be achieved by allowing an option to display the download progress logs during execution.

Describe alternatives you've considered I have considered manually modifying tube_backend.py to enable yt_dlp output

   "default_opts": {
    ...
    "quiet": False,
    "noprogress": False,
    "skip_download": False,
    ...
    }

but a built-in option to control log visibility during execution would be more convenient and user-friendly.

Additional context Having a real-time progress indicator for downloads would make it easier to monitor the status of ongoing downloads.

[chapmanjacobd]

The default logging is minimal so that novel errors are obvious.

You can run with -v, -vv, -vvv, or -vvvv for more verbose logging.

ie. lb dl -vvv or lb dl --verbose --verbose --verbose

You can use the lb download-status command.

[holta]

You can use the lb download-status command.

Interesting. And lb download_status with underscore appears to be equivalent:

https://github.com/chapmanjacobd/library/blob/0c08fd4814aebbc087284543c266f28c207cadd4/xklb/lb.py#L41

Just FYI --help or -h output below (if I'm reading it correctly!) suggests this is not for ongoing monitoring of granular details e.g. during the download of large/individual videos.

(Not a walk in the park to parse animation-like text output of things like rsync --info=progress2 of course ;-)

But we'll find a way eventually! 💯

root@box:~# lb download-status -h
usage: library download-status DATABASE

    Print download queue groups

        library download-status video.db
        ╒═════════════╤══════════════════╤════════════════════╤══════════╕
        │ extractor_key      │ duration         │   never_downloaded │   errors │
        ╞═════════════╪══════════════════╪════════════════════╪══════════╡
        │ Youtube     │ 3 hours and 2.07 │                 76 │        0 │
        │             │ minutes          │                    │          │
        ├─────────────┼──────────────────┼────────────────────┼──────────┤
        │ Dailymotion │                  │                 53 │        0 │
        ├─────────────┼──────────────────┼────────────────────┼──────────┤
        │ Youtube     │ 1 day, 18 hours  │                 30 │        0 │
        │             │ and 6 minutes    │                    │          │
        ├─────────────┼──────────────────┼────────────────────┼──────────┤
        │ Dailymotion │                  │                186 │      198 │
        ├─────────────┼──────────────────┼────────────────────┼──────────┤
        │ Youtube     │ 1 hour and 52.18 │                  1 │        0 │
        │             │ minutes          │                    │          │
        ├─────────────┼──────────────────┼────────────────────┼──────────┤
        │ Vimeo       │                  │                253 │       49 │
        ├─────────────┼──────────────────┼────────────────────┼──────────┤
        │ Youtube     │ 2 years, 4       │              51676 │      197 │
        │             │ months, 15 days  │                    │          │
        │             │ and 6 hours      │                    │          │
        ├─────────────┼──────────────────┼────────────────────┼──────────┤
        │ Youtube     │ 4 months, 23     │               2686 │        7 │
        │             │ days, 19 hours   │                    │          │
        │             │ and 33 minutes   │                    │          │
        ╘═════════════╧══════════════════╧════════════════════╧══════════╛

    Simulate --safe flag

        library download-status video.db --safe

positional arguments:
  database

options:
  -h, --help            show this help message and exit
  --cols [COLS ...], -cols [COLS ...], -col [COLS ...]
                        Include a column when printing (default: None)
  --safe, -safe         Skip generic URLs (default: False)
  --retry-delay RETRY_DELAY, -r RETRY_DELAY
                        Must be specified in SQLITE Modifiers format: N hours, days, months, or years
                        (default: 14 days)
  -v, --verbose

[chapmanjacobd]

Yes, xklb prioritizes interactive and experimental use.

Perhaps you could query the SQLITE database directly to see if any recoverable, unrecoverable errors occurred during the previous download attempt.

[holta]

Perhaps you could query the SQLITE database directly to see if any recoverable, unrecoverable errors occurred

Excellent Suggestion, Thanks! 👌

Context:

• Naturally everyone's a sucker for "Entrancing Animations" and "Exact Percentages" while watching the kettle boil (i.e. during downloads).

• Such Bells & Whistles can be added later, but are not required at this time.

• Robust engineering is our goal at this time.

[deldesir]

Yes, xklb prioritizes interactive and experimental use.

Perhaps you could query the SQLITE database directly to see if any recoverable, unrecoverable errors occurred during the previous download attempt.

Thanks @chapmanjacobd. I will implement your suggestion. I think it will help.

[chapmanjacobd]

Thanks for opening this issue. I realize now that log.debug("\n".join(ydl_full_log)) was commented out. I've pushed up a new version v2.3.001 which will print the full yt-dlp log with four levels of verbosity or more -vvvv. I hope that will help with what you are trying to do.

I also added a quick scan of the downloaded media file to check that the download was not interrupted. If it was then you should see "Media check failed" in the SQLITE database. If that happens when you run lb dl again it should resume from the last byte written. You will need to set the retry time to something small like 1 second lb dl --retry-delay '1 second'

The default is to retry after two weeks because most recoverable errors are not transient errors but take time to resolve [account inactivity, copyright trolls, geofencing / IP block, etc]).

[holta]

You will need to set the retry time to something small like 1 second lb dl --retry-delay '1 second'

The default is to retry after two weeks because most recoverable errors are not transient errors but take time to resolve [account inactivity, copyright trolls, geofencing, etc]).

Critical & vital logistics, Thanks for explaining!

Background: In practice, most Internet-in-a-Box implementers are not professional curriculum designers and so try to provision all content (for their school, clinic, church, prison, family, etc) within a single weekend — and often lose patience (declaring victory!) within ~24h maximum. 😉

[chapmanjacobd]

I should clarify that yt-dlp will retry on network and similar transient errors by default so any mid-download interruption is likely the result of getting IP-blocked by the site that is hosting the video. Retrying after 20 minutes or one hour would likely be a better strategy than immediately retrying but many video site IP blocks last longer than a day.