Videos which fail to be downloaded from the HTTP link are still not added to the ZIM

[benoit74]

These are logs found on https://farm.openzim.org/pipeline/787e87e5-f589-45e8-aeb9-9311dadc9700 task from https://farm.openzim.org/recipes/ted_topic_animation recipe.

[ted2zim::2024-03-14 09:48:22,554] DEBUG:Downloading Does planting trees actually cool the planet?
[ted2zim::2024-03-14 09:48:22,937] ERROR:Could not download from https://py.tedcdn.com/consus/projects/00/67/48/001/products/2022e-carolyn-beans-tree-planting-001-fallback-d05dfc19-b9a7-4160-8a97-419ce81f2402-1200k.mp4 for /output/tmpu6uk98x3/videos/120091/video.mp4
[ted2zim::2024-03-14 09:48:22,937] DEBUG:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/ted2zim/scraper.py", line 927, in download_video_files
    save_large_file(video_link, org_video_file_path)
  File "/usr/local/lib/python3.11/site-packages/zimscraperlib/download.py", line 122, in save_large_file
    subprocess.run(
  File "/usr/local/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'wget', '-t', '5', '--retry-connrefused', '--random-wait', '-O', '/output/tmpu6uk98x3/videos/120091/video.mp4', '-c', 'https://py.tedcdn.com/consus/projects/00/67/48/001/products/2022e-carolyn-beans-tree-planting-001-fallback-d05dfc19-b9a7-4160-8a97-419ce81f2402-1200k.mp4']' returned non-zero exit status 8.
[youtube] Extracting URL: bbxmH_Kj7fk
[youtube] bbxmH_Kj7fk: Downloading webpage
[youtube] bbxmH_Kj7fk: Downloading ios player API JSON
[youtube] bbxmH_Kj7fk: Downloading android player API JSON
[youtube] bbxmH_Kj7fk: Downloading m3u8 information
[info] bbxmH_Kj7fk: Downloading subtitles: ar, my, zh-CN, zh-TW, en, fr, id, it, ko, pt-BR, pt-PT, es, vi
[info] bbxmH_Kj7fk: Downloading 1 format(s): 248+251
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.ar.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.ar.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    2.28MiB/s (00:00:00)
[download]    7.00KiB at    3.38MiB/s (00:00:00)
[download]    9.84KiB at    3.40MiB/s (00:00:00)
[download] 100% of    9.84KiB in 00:00:00 at 141.09KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.my.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.my.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    2.30MiB/s (00:00:00)
[download]    7.00KiB at    3.16MiB/s (00:00:00)
[download]   14.55KiB at    4.40MiB/s (00:00:00)
[download] 100% of   14.55KiB in 00:00:00 at 410.85KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.zh-CN.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.zh-CN.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    1.58MiB/s (00:00:00)
[download]    6.13KiB at    2.05MiB/s (00:00:00)
[download] 100% of    6.13KiB in 00:00:00 at 188.03KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.zh-TW.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.zh-TW.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    2.30MiB/s (00:00:00)
[download]    6.14KiB at    2.67MiB/s (00:00:00)
[download] 100% of    6.14KiB in 00:00:00 at 197.22KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.en.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.en.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    2.15MiB/s (00:00:00)
[download]    6.67KiB at    2.70MiB/s (00:00:00)
[download] 100% of    6.67KiB in 00:00:00 at 170.93KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.fr.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.fr.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    1.55MiB/s (00:00:00)
[download]    7.00KiB at    2.39MiB/s (00:00:00)
[download]    8.48KiB at    2.12MiB/s (00:00:00)
[download] 100% of    8.48KiB in 00:00:00 at 241.87KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.id.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.id.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    2.48MiB/s (00:00:00)
[download]    7.00KiB at    3.43MiB/s (00:00:00)
[download]    7.17KiB at    2.48MiB/s (00:00:00)
[download] 100% of    7.17KiB in 00:00:00 at 195.15KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.it.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.it.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    2.37MiB/s (00:00:00)
[download]    7.00KiB at    3.18MiB/s (00:00:00)
[download]    7.27KiB at    2.30MiB/s (00:00:00)
[download] 100% of    7.27KiB in 00:00:00 at 205.14KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.ko.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.ko.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    2.01MiB/s (00:00:00)
[download]    7.00KiB at    2.77MiB/s (00:00:00)
[download]    7.75KiB at    2.15MiB/s (00:00:00)
[download] 100% of    7.75KiB in 00:00:00 at 231.07KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.pt-BR.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.pt-BR.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at  Unknown B/s (00:00:00)
[download]    7.00KiB at    4.50MiB/s (00:00:00)
[download]    7.19KiB at    3.21MiB/s (00:00:00)
[download] 100% of    7.19KiB in 00:00:00 at 231.52KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.pt-PT.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.pt-PT.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    1.91MiB/s (00:00:00)
[download]    7.00KiB at    2.83MiB/s (00:00:00)
[download]    7.76KiB at    2.27MiB/s (00:00:00)
[download] 100% of    7.76KiB in 00:00:00 at 237.33KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.es.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.es.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    2.72MiB/s (00:00:00)
[download]    7.00KiB at    3.59MiB/s (00:00:00)
[download]    7.95KiB at    2.80MiB/s (00:00:00)
[download] 100% of    7.95KiB in 00:00:00 at 227.10KiB/s
[info] Writing video subtitles to: /output/tmpu6uk98x3/videos/120091/video.vi.vtt
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.vi.vtt

[download]    1.00KiB at  Unknown B/s (00:00:00)
[download]    3.00KiB at    2.61MiB/s (00:00:00)
[download]    7.00KiB at    3.43MiB/s (00:00:00)
[download]    8.40KiB at    2.82MiB/s (00:00:00)
[download] 100% of    8.40KiB in 00:00:00 at 267.26KiB/s
[info] Downloading video thumbnail 41 ...
[info] Writing video thumbnail 41 to: /output/tmpu6uk98x3/videos/120091/video.41.webp
[info] Downloading video thumbnail 40 ...
[info] Writing video thumbnail 40 to: /output/tmpu6uk98x3/videos/120091/video.40.jpg
[info] Downloading video thumbnail 39 ...
[info] Writing video thumbnail 39 to: /output/tmpu6uk98x3/videos/120091/video.39.webp
[info] Downloading video thumbnail 38 ...
[info] Writing video thumbnail 38 to: /output/tmpu6uk98x3/videos/120091/video.38.jpg
[info] Downloading video thumbnail 37 ...
[info] Writing video thumbnail 37 to: /output/tmpu6uk98x3/videos/120091/video.37.webp
[info] Downloading video thumbnail 36 ...
[info] Writing video thumbnail 36 to: /output/tmpu6uk98x3/videos/120091/video.36.jpg
[info] Downloading video thumbnail 35 ...
[info] Writing video thumbnail 35 to: /output/tmpu6uk98x3/videos/120091/video.35.webp
[info] Downloading video thumbnail 34 ...
[info] Writing video thumbnail 34 to: /output/tmpu6uk98x3/videos/120091/video.34.jpg
[info] Downloading video thumbnail 33 ...
[info] Writing video thumbnail 33 to: /output/tmpu6uk98x3/videos/120091/video.33.jpg
[info] Downloading video thumbnail 32 ...
[info] Writing video thumbnail 32 to: /output/tmpu6uk98x3/videos/120091/video.32.jpg
[info] Downloading video thumbnail 31 ...
[info] Writing video thumbnail 31 to: /output/tmpu6uk98x3/videos/120091/video.31.jpg
[info] Downloading video thumbnail 30 ...
[info] Writing video thumbnail 30 to: /output/tmpu6uk98x3/videos/120091/video.30.jpg
[info] Downloading video thumbnail 29 ...
[info] Writing video thumbnail 29 to: /output/tmpu6uk98x3/videos/120091/video.29.webp
[info] Downloading video thumbnail 28 ...
[info] Writing video thumbnail 28 to: /output/tmpu6uk98x3/videos/120091/video.28.jpg
[info] Downloading video thumbnail 27 ...
[info] Writing video thumbnail 27 to: /output/tmpu6uk98x3/videos/120091/video.27.webp
[info] Downloading video thumbnail 26 ...
[info] Writing video thumbnail 26 to: /output/tmpu6uk98x3/videos/120091/video.26.jpg
[info] Downloading video thumbnail 25 ...
[info] Writing video thumbnail 25 to: /output/tmpu6uk98x3/videos/120091/video.25.webp
[info] Downloading video thumbnail 24 ...
[info] Writing video thumbnail 24 to: /output/tmpu6uk98x3/videos/120091/video.24.jpg
[info] Downloading video thumbnail 23 ...
[info] Writing video thumbnail 23 to: /output/tmpu6uk98x3/videos/120091/video.23.webp
[info] Downloading video thumbnail 22 ...
[info] Writing video thumbnail 22 to: /output/tmpu6uk98x3/videos/120091/video.22.jpg
[info] Downloading video thumbnail 21 ...
[info] Writing video thumbnail 21 to: /output/tmpu6uk98x3/videos/120091/video.21.webp
[info] Downloading video thumbnail 20 ...
[info] Writing video thumbnail 20 to: /output/tmpu6uk98x3/videos/120091/video.20.jpg
[info] Downloading video thumbnail 19 ...
[info] Writing video thumbnail 19 to: /output/tmpu6uk98x3/videos/120091/video.19.webp
[info] Downloading video thumbnail 18 ...
[info] Writing video thumbnail 18 to: /output/tmpu6uk98x3/videos/120091/video.18.jpg
[info] Downloading video thumbnail 17 ...
[info] Writing video thumbnail 17 to: /output/tmpu6uk98x3/videos/120091/video.17.webp
[info] Downloading video thumbnail 16 ...
[info] Writing video thumbnail 16 to: /output/tmpu6uk98x3/videos/120091/video.16.jpg
[info] Downloading video thumbnail 15 ...
[info] Writing video thumbnail 15 to: /output/tmpu6uk98x3/videos/120091/video.15.webp
[info] Downloading video thumbnail 14 ...
[info] Writing video thumbnail 14 to: /output/tmpu6uk98x3/videos/120091/video.14.jpg
[info] Downloading video thumbnail 13 ...
[info] Writing video thumbnail 13 to: /output/tmpu6uk98x3/videos/120091/video.13.webp
[info] Downloading video thumbnail 12 ...
[info] Writing video thumbnail 12 to: /output/tmpu6uk98x3/videos/120091/video.12.jpg
[info] Downloading video thumbnail 11 ...
[info] Writing video thumbnail 11 to: /output/tmpu6uk98x3/videos/120091/video.11.webp
[info] Downloading video thumbnail 10 ...
[info] Writing video thumbnail 10 to: /output/tmpu6uk98x3/videos/120091/video.10.jpg
[info] Downloading video thumbnail 9 ...
[info] Writing video thumbnail 9 to: /output/tmpu6uk98x3/videos/120091/video.9.webp
[info] Downloading video thumbnail 8 ...
[info] Writing video thumbnail 8 to: /output/tmpu6uk98x3/videos/120091/video.8.jpg
[info] Downloading video thumbnail 7 ...
[info] Writing video thumbnail 7 to: /output/tmpu6uk98x3/videos/120091/video.7.webp
[info] Downloading video thumbnail 6 ...
[info] Writing video thumbnail 6 to: /output/tmpu6uk98x3/videos/120091/video.6.jpg
[info] Downloading video thumbnail 5 ...
[info] Writing video thumbnail 5 to: /output/tmpu6uk98x3/videos/120091/video.5.webp
[info] Downloading video thumbnail 4 ...
[info] Writing video thumbnail 4 to: /output/tmpu6uk98x3/videos/120091/video.4.jpg
[info] Downloading video thumbnail 3 ...
[info] Writing video thumbnail 3 to: /output/tmpu6uk98x3/videos/120091/video.3.webp
[info] Downloading video thumbnail 2 ...
[info] Writing video thumbnail 2 to: /output/tmpu6uk98x3/videos/120091/video.2.jpg
[info] Downloading video thumbnail 1 ...
[info] Writing video thumbnail 1 to: /output/tmpu6uk98x3/videos/120091/video.1.webp
[info] Downloading video thumbnail 0 ...
[info] Writing video thumbnail 0 to: /output/tmpu6uk98x3/videos/120091/video.0.jpg
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.f248.webm

[download]   0.0% of   35.44MiB at  Unknown B/s ETA Unknown
[download]   0.0% of   35.44MiB at    1.68MiB/s ETA 00:21  
[download]   0.0% of   35.44MiB at    2.56MiB/s ETA 00:13
[download]   0.0% of   35.44MiB at    4.07MiB/s ETA 00:08
[download]   0.1% of   35.44MiB at    6.52MiB/s ETA 00:05
[download]   0.2% of   35.44MiB at    8.96MiB/s ETA 00:03
[download]   0.3% of   35.44MiB at   11.50MiB/s ETA 00:03
[download]   0.7% of   35.44MiB at   16.65MiB/s ETA 00:02
[download]   1.4% of   35.44MiB at   25.45MiB/s ETA 00:01
[download]   2.8% of   35.44MiB at   39.81MiB/s ETA 00:00
[download]   5.6% of   35.44MiB at   52.58MiB/s ETA 00:00
[download]  11.3% of   35.44MiB at   55.58MiB/s ETA 00:00
[download]  22.6% of   35.44MiB at   61.29MiB/s ETA 00:00
[download]  27.3% of   35.44MiB at   66.01MiB/s ETA 00:00
[download]  27.4% of   35.44MiB at  Unknown B/s ETA Unknown
[download]  27.4% of   35.44MiB at  Unknown B/s ETA Unknown
[download]  27.4% of   35.44MiB at    5.44MiB/s ETA 00:04  
[download]  27.4% of   35.44MiB at    8.05MiB/s ETA 00:03
[download]  27.4% of   35.44MiB at   12.38MiB/s ETA 00:02
[download]  27.5% of   35.44MiB at   19.86MiB/s ETA 00:01
[download]  27.7% of   35.44MiB at   32.30MiB/s ETA 00:00
[download]  28.1% of   35.44MiB at   29.14MiB/s ETA 00:00
[download]  28.8% of   35.44MiB at   26.45MiB/s ETA 00:00
[download]  30.2% of   35.44MiB at   41.82MiB/s ETA 00:00
[download]  33.0% of   35.44MiB at   59.74MiB/s ETA 00:00
[download]  38.6% of   35.44MiB at   58.11MiB/s ETA 00:00
[download]  49.9% of   35.44MiB at   61.98MiB/s ETA 00:00
[download]  54.6% of   35.44MiB at   66.95MiB/s ETA 00:00
[download]  54.6% of   35.44MiB at  Unknown B/s ETA Unknown
[download]  54.6% of   35.44MiB at  Unknown B/s ETA Unknown
[download]  54.6% of   35.44MiB at    5.37MiB/s ETA 00:03  
[download]  54.7% of   35.44MiB at    7.92MiB/s ETA 00:02
[download]  54.7% of   35.44MiB at   12.09MiB/s ETA 00:01
[download]  54.8% of   35.44MiB at   18.88MiB/s ETA 00:00
[download]  55.0% of   35.44MiB at   30.15MiB/s ETA 00:00
[download]  55.3% of   35.44MiB at   48.72MiB/s ETA 00:00
[download]  56.0% of   35.44MiB at   73.93MiB/s ETA 00:00
[download]  57.4% of   35.44MiB at   25.46MiB/s ETA 00:00
[download]  60.3% of   35.44MiB at   40.85MiB/s ETA 00:00
[download]  65.9% of   35.44MiB at   56.49MiB/s ETA 00:00
[download]  77.2% of   35.44MiB at   68.28MiB/s ETA 00:00
[download]  82.6% of   35.44MiB at   68.65MiB/s ETA 00:00
[download]  82.6% of   35.44MiB at  Unknown B/s ETA Unknown
[download]  82.6% of   35.44MiB at  Unknown B/s ETA Unknown
[download]  82.6% of   35.44MiB at    5.80MiB/s ETA 00:01  
[download]  82.6% of   35.44MiB at    8.54MiB/s ETA 00:00
[download]  82.7% of   35.44MiB at   12.83MiB/s ETA 00:00
[download]  82.8% of   35.44MiB at   20.44MiB/s ETA 00:00
[download]  83.0% of   35.44MiB at   32.84MiB/s ETA 00:00
[download]  83.3% of   35.44MiB at   53.42MiB/s ETA 00:00
[download]  84.0% of   35.44MiB at   80.71MiB/s ETA 00:00
[download]  85.4% of   35.44MiB at   43.28MiB/s ETA 00:00
[download]  88.2% of   35.44MiB at   61.61MiB/s ETA 00:00
[download]  93.9% of   35.44MiB at   72.97MiB/s ETA 00:00
[download] 100.0% of   35.44MiB at   65.58MiB/s ETA 00:00
[download] 100% of   35.44MiB in 00:00:00 at 51.52MiB/s  
[download] Destination: /output/tmpu6uk98x3/videos/120091/video.f251.webm

[download]   0.0% of    5.68MiB at  Unknown B/s ETA Unknown
[download]   0.1% of    5.68MiB at    2.18MiB/s ETA 00:02  
[download]   0.1% of    5.68MiB at    3.57MiB/s ETA 00:01
[download]   0.3% of    5.68MiB at    5.79MiB/s ETA 00:00
[download]   0.5% of    5.68MiB at    9.47MiB/s ETA 00:00
[download]   1.1% of    5.68MiB at   15.71MiB/s ETA 00:00
[download]   2.2% of    5.68MiB at   26.13MiB/s ETA 00:00
[download]   4.4% of    5.68MiB at   43.55MiB/s ETA 00:00
[download]   8.8% of    5.68MiB at   70.34MiB/s ETA 00:00
[download]  17.6% of    5.68MiB at  108.41MiB/s ETA 00:00
[download]  35.2% of    5.68MiB at  110.59MiB/s ETA 00:00
[download]  70.4% of    5.68MiB at   99.90MiB/s ETA 00:00
[download] 100.0% of    5.68MiB at   88.29MiB/s ETA 00:00
[download] 100% of    5.68MiB in 00:00:00 at 63.18MiB/s  
[Merger] Merging formats into "/output/tmpu6uk98x3/videos/120091/video.webm"
Deleting original file /output/tmpu6uk98x3/videos/120091/video.f248.webm (pass -k to keep)
Deleting original file /output/tmpu6uk98x3/videos/120091/video.f251.webm (pass -k to keep)
[ted2zim::2024-03-14 09:48:27,963] DEBUG:Speaker doesn't have an image
[ted2zim::2024-03-14 09:48:28,642] INFO:downloaded /output/tmpu6uk98x3/videos/120091/thumbnail.webp from cache at thumbnail/120091
[ted2zim::2024-03-14 09:48:28,643] WARNING:Multiple video file candidates for 120091 in /output/tmpu6uk98x3/videos/120091. Picking /output/tmpu6uk98x3/videos/120091/video.mp4 out of [PosixPath('/output/tmpu6uk98x3/videos/120091/video.mp4'), PosixPath('/output/tmpu6uk98x3/videos/120091/video.webm')]
[ted2zim::2024-03-14 09:48:28,643] DEBUG:Converting video 120091
[ted2zim::2024-03-14 09:48:28,751] ERROR:Failed to post process video 120091
[ted2zim::2024-03-14 09:48:28,751] DEBUG:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/ted2zim/scraper.py", line 965, in download_video_files
    post_process_video(
  File "/usr/local/lib/python3.11/site-packages/ted2zim/processing.py", line 36, in post_process_video
    reencode(
  File "/usr/local/lib/python3.11/site-packages/zimscraperlib/video/encoding.py", line 47, in reencode
    ffmpeg.check_returncode()
  File "/usr/local/lib/python3.11/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['ffmpeg', '-y', '-i', 'file:/output/tmpu6uk98x3/videos/120091/video.mp4', '-max_muxing_queue_size', '9999', '-codec:v', 'libvpx', '-quality', 'best', '-b:v', '300k', '-maxrate', '300k', '-minrate', '300k', '-qmin', '30', '-qmax', '42', '-vf', "scale='480:trunc(ow/a/2)*2'", '-codec:a', 'libvorbis', '-ar', '44100', '-b:a', '48k', 'file:/tmp/tmpm83hfaka/video.tmp.webm']' returned non-zero exit status 1.

Many problems here:

• Multiple video file candidates for 120091 => why do we have two candidates on the local filesystem?
• FFMPEG fails => to be fixed? (error is probably due to the fact that output file already exists)
• FFMPEG logs are missing => display all STDOUT/STDERR for such errors (probably only in debug mode)

Nota: once task is finished and whole log file uploaded, I will check if we do not have a problem of double-processing of the same video ID.

[benoit74]

The problem is global, for all videos which fails to be downloaded from the HTTP link.

[benoit74]

FYI, I will not work on it in the coming weeks, the scraper is now producing a valid ZIM (these videos are excluded) which is already way better than it has been since few months / years.

Contributions are of course welcomed, it is probably a good first issue.