Feature Request: Ability to Log Failed Downloads

mollyrealized commented 1 year ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm requesting a feature unrelated to a specific site
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2023.07.06 (update instructions) or later (specify commit)
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

Using the "--batch-file" option to download multiple videos generates a substantial amount of output, making it difficult to identify reports of videos that failed to download. A feature that allows the URLs of videos that couldn't be downloaded to be written to a specific text file (log file) would be beneficial both for manual inspection and for allowing error logging while using "-quiet --progress-bar". Current 'abort' commands that halt the download process when failures are detected are not ideal, as they interrupt the entire operation.

Is the description of the issue itself sufficient? and Is there enough context in your bug report? In my view, yes. If anything remains confusing, please let me know. Are you using the latest version? Yes. Is the issue already documented? Not that I can tell through searching open issues. Why are existing options not enough? I do not think there are any logging options such as I describe. Have you read and understood the changes, between youtube-dl and yt-dlp Yes. Does the issue involve one problem, and one problem only? Yes. Is anyone going to need the feature? Yes, I believe this would be needed widely by anyone using the --batch-file option. Is your question about yt-dlp? Yes. Are you willing to share account details if needed? Yes. Is the website primarily used for piracy? No.

Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-quiet', '--progress', '-vU']
[debug] Portable config "C:\Users\Molly\bin\EXE\ydl\yt-dlp.conf": ['--audio-format', 'mp3', '--format', 'mp4', '-o', 'C:\\Users\\Molly\\zInboxes\\Capture\\Videos\\%(duration>%H-%M-%S)s %(title)S.%(ext)s', '--embed-subs', '--sub-lang', 'en', '--restrict-filenames', '--trim-filenames', '150', '--sponsorblock-remove', 'sponsor,interaction,selfpromo,interaction,intro,outro']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.07.06 [b532a3481] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19043-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg n5.0-1-ga66b58d61c-20220117 (setts), ffprobe n5.0-1-ga66b58d61c-20220117, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1855 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.07.06, Current version: stable@2023.07.06
Current Build Hash: 5ff3e702171a50175c34397494e2d18ce35d771c2110b1e59bd173ec2fb352aa
yt-dlp is up to date (stable@2023.07.06)

bashonly commented 1 year ago

Related: #1760, #1761, #2668

pukkandan commented 1 year ago

This has been requested before, but there's no issue open afair. This is however currently difficult to implement. As workaround, you can:

Send stderr to a different log file using shell redirects
Search the combined log for lines starting in ERROR:, or
--print-to-file after_move:original_url success.txt will record all successful URLs, which you can then diff with your --batch-file

Tmalll commented 9 months ago

I also need similar functions, I use in the CMD under Windows >> re-directed output content, but it only has a general log (I guess only download and download successfully downloading and running successfully) But all error outputs are still output on the screen How can I only output the error into the log file? The function of this implementation is the opposite I need :)

kacperwyczawski commented 4 months ago

3. --print-to-file after_move:original_url success.txt will record all successful URLs, which you can then diff with your --batch-file

Thank you, this is really useful

mollyrealized commented 4 months ago

Here is a batch file that handles an input file and then writes anything with the "ERROR" prefix during the output into a separate file. Use at your own risk and no warranty is offered, but it should work ...

@echo off
setlocal

rem Check if the input file argument is provided
if "%~1"=="" (
    echo Usage: %~nx0 inputfile
    exit /b 1
)

rem Get the input file from the command-line argument
set "inputFile=%~1"

rem Verify the input file exists
if not exist "%inputFile%" (
    echo The input file "%inputFile%" does not exist.
    exit /b 1
)

rem Get the current date and time in the format YYYYMMDDHHMMSS
for /f "tokens=2 delims==" %%i in ('wmic os get localdatetime /value ^| find "="') do set datetime=%%i
set "datetime=%datetime:~0,4%%datetime:~4,2%%datetime:~6,2%%datetime:~8,2%%datetime:~10,2%%datetime:~12,2%"

rem Paths to the output files
set "fullOutput=full_output.log"
set "errorFile=errorlog-%datetime%.txt"
set "tempErrorFile=temp_error.txt"

rem Clear previous logs if they exist
if exist "%fullOutput%" del "%fullOutput%"
if exist "%errorFile%" del "%errorFile%"
if exist "%tempErrorFile%" del "%tempErrorFile%"

rem Download videos and log the full output
yt-dlp --batch-file="%inputFile%" > "%fullOutput%" 2>&1

rem Extract lines containing errors and save to temporary error file
findstr /r /c:"ERROR: " "%fullOutput%" > "%tempErrorFile%"

rem Remove "ERROR: " from the beginning of each error line and save to final error file
for /f "tokens=*" %%i in (%tempErrorFile%) do (
    set "line=%%i"
    setlocal enabledelayedexpansion
    echo !line:ERROR: =! >> "%errorFile%"
    endlocal
)

rem Cleanup temporary file
del "%tempErrorFile%"
del "%fullOutput%"

endlocal

The reason why I went this route instead of comparing the original request with the successful downloads is because - at least for me - sometimes I will download stuff that has already been downloaded. In those cases, those (falsely, for my purposes) register as failed downloads.

yt-dlp / yt-dlp