search

C0D3D3V / bbb-dl

Downlaod BigBlueButton recordings. A simple scrapper for Big Blue Button lessons. Download and merge the audio and video files of a BigBlueButton conference from the conference replay url.
MIT License
65 stars 16 forks source link

Downloader always gets stuck #20

Closed lokpimkd closed 2 years ago

lokpimkd commented 2 years ago

So I am trying to download a video from my course. But the downloader always gets stuck somewhere half way. Tried different encoders same thing, and yes my card does support HEVC

This is the whole log of the downloader:

bbb-dl https://bbb-lb.finki.ukim.mk/playback/presentation/2.3/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416 --encoder hevc_nvenc -ncc -v
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.7 (CPython) - Windows-10-10.0.17763-SP0
[debug] exe versions: ffmpeg 4.4.1-full_build-www.gyan.dev, ffprobe 4.4.1-full_build-www.gyan.dev
[debug] Proxy map: {}
[BBB] Downloading meta informations
[BBB] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading webpage
[BBB] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading XML
[BBB] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading XML
[BBB] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading XML
[BBB] BBB version: 3.11.1+
[BBB] Downloading webcams.webm
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://bbb-lb.finki.ukim.mk/presentation/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416/video/webcams.webm'
[download] Destination: c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416\webcams.webm
[download] 100% of 38.88MiB in 00:05
[BBB] Downloading deskshare.webm
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://bbb-lb.finki.ukim.mk/presentation/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416/deskshare/deskshare.webm'
[download] Destination: c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416\deskshare.webm
[download]  65.1% of 129.00MiB at  7.11MiB/s ETA 00:06
C0D3D3V commented 2 years ago

For me it works, are you sure you are on the latest version?

You can update with pip install -U bbb-dl

C0D3D3V commented 2 years ago

I guess because you are using the -ncc option that you are on windows? Normally you should not need to use that option. What Windows Version are you using? It is also very strange that it downloaded the webcams.webm file but not the deskshare.webm file. Maybe we should update to use yt-dlp instead of youtube-dl

lokpimkd commented 2 years ago

Tried another computer, this time without -ncc and other graphic card. Same thing. One is Windows 10 LTSC other is Windows 10 19044, still gets stuck image

C0D3D3V commented 2 years ago

But on that one you got to 71.3% instead of 65.1% as before. Are you sure you waited long enough? How fast is you internet connection?

C0D3D3V commented 2 years ago

Also there is no error so, in theory it still is doing stuff. So maybe just wait longer. But I will update to yt-dlp anyway, maybe that will help

lokpimkd commented 2 years ago

It's not that the percentage was increased, it sticks in random percentage after 50% as per my tries. I've waited 30+ min, on a 1660 super, and ethernet connection of 100mbs down which for a 100mb video I think should be processed in few minutes if not less. Maybe you are right since youtube-dl was discontinued quite some time ago maybe that's the reason. Please ping me once you make the changes so that I can retest and let you know :)

C0D3D3V commented 2 years ago

Can you test out the new release please? I hope I have not broken anything xD

lokpimkd commented 2 years ago

Now at least we have an error, which is some progress :D 

bbb-dl https://bbb-lb.finki.ukim.mk/playback/presentation/2.3/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416 --encoder hevc_nvenc -v
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.07.18 [135f05e]
[debug] Python 3.8.7 (CPython 64bit) - Windows-10-10.0.17763-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.4.1-full_build-www.gyan.dev (setts), ffprobe 4.4.1-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2021.10.08, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[BBB] Downloading meta informations
[BBB] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading webpage
[BBB] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading XML
[BBB] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading XML
[BBB] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading XML
[BBB] BBB version: 3.11.1+
[BBB] Downloading webcams.webm
[debug] Default format spec: bestvideo*+bestaudio/best
[info] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading 1 format(s): 0
[debug] Invoking http downloader on "https://bbb-lb.finki.ukim.mk/presentation/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416/video/webcams.webm"
[debug] File locking is not supported. Proceeding without locking
[download] Destination: c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416\webcams.webm
[download] 100% of 38.88MiB in 00:05
[BBB] Downloading deskshare.webm
[debug] Default format spec: bestvideo*+bestaudio/best
[info] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading 1 format(s): 0
[debug] Invoking http downloader on "https://bbb-lb.finki.ukim.mk/presentation/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416/deskshare/deskshare.webm"
[download] Destination: c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416\deskshare.webm
[download]  55.8% of 129.00MiB at    7.15MiB/s ETA 00:07[download] Got server HTTP error: The read operation timed out
ERROR: giving up after 0 retries
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Lokpi\AppData\Local\Programs\Python\Python38\Scripts\bbb-dl.exe\__main__.py", line 7, in <module>
    sys.exit(main())
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\bbb_dl\main.py", line 832, in main
    BBBDL(args.verbose, args.no_check_certificate, args.encoder, args.audiocodec).run(
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\bbb_dl\main.py", line 203, in run
    self.ydl.process_ie_result(deskshare_dl)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1578, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 2661, in process_video_result
    self.process_info(new_info)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 3160, in process_info
    success, real_download = self.dl(temp_filename, info_dict)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 2860, in dl
    return fd.download(name, new_info, subtitle)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\downloader\common.py", line 444, in download
    ret = self.real_download(filename, info_dict)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\downloader\http.py", line 392, in real_download
    self.report_error('giving up after %s retries' % retries)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 971, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 903, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

[BBB] Downloading deskshare.webm failed! Downloading deskshare.mp4 instead
[debug] Default format spec: bestvideo*+bestaudio/best
[info] c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416: Downloading 1 format(s): 0
[debug] Invoking http downloader on "https://bbb-lb.finki.ukim.mk/presentation/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416/deskshare/deskshare.mp4"
ERROR: unable to download video data: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\bbb_dl\main.py", line 203, in run
    self.ydl.process_ie_result(deskshare_dl)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1578, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 2661, in process_video_result
    self.process_info(new_info)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 3160, in process_info
    success, real_download = self.dl(temp_filename, info_dict)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 2860, in dl
    return fd.download(name, new_info, subtitle)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\downloader\common.py", line 444, in download
    ret = self.real_download(filename, info_dict)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\downloader\http.py", line 392, in real_download
    self.report_error('giving up after %s retries' % retries)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 971, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 914, in trouble
    raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: giving up after 0 retries

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 3160, in process_info
    success, real_download = self.dl(temp_filename, info_dict)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 2860, in dl
    return fd.download(name, new_info, subtitle)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\downloader\common.py", line 444, in download
    ret = self.real_download(filename, info_dict)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\downloader\http.py", line 375, in real_download
    establish_connection()
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\downloader\http.py", line 131, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 3639, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\urllib\request.py", line 640, in http_response
    response = self.parent.error(
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\urllib\request.py", line 502, in _call_chain
    result = func(*args)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

[BBB] Warning: Downloading deskshare.mp4 failed - No desk was likely shared in this session.
[BBB] Downloading slides
Traceback (most recent call last):
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Lokpi\AppData\Local\Programs\Python\Python38\Scripts\bbb-dl.exe\__main__.py", line 7, in <module>
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\bbb_dl\main.py", line 832, in main
    BBBDL(args.verbose, args.no_check_certificate, args.encoder, args.audiocodec).run(
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\bbb_dl\main.py", line 293, in run
    self._write_slides(slides_infos, self.ydl)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\bbb_dl\main.py", line 354, in _write_slides
    if os.path.exists(encodeFilename(slide.path)):
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\utils.py", line 877, in encodeFilename
    assert isinstance(s, str)
AssertionError
C0D3D3V commented 2 years ago

Mh, I guess your internet connection is very unstable. I can increase the allowed retries.

lokpimkd commented 2 years ago

Trust me dude, I run whole test suites with 100+ automation tests couple of times a day, its never failed me in the past 2years, the thing is the error is trying to find ttps://bbb-lb.finki.ukim.mk/presentation/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416/deskshare/deskshare.webm" [download] Destination: c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416\deskshare.webm which for some reason it doesn't exist. Idk how bbb works you are the expert on that here

C0D3D3V commented 2 years ago

No you read it wrong. You have to concentrate on this error: 

[debug] Invoking http downloader on "https://bbb-lb.finki.ukim.mk/presentation/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416/deskshare/deskshare.webm"
[download] Destination: c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416\deskshare.webm
[download]  55.8% of 129.00MiB at    7.15MiB/s ETA 00:07[download] Got server HTTP error: The read operation timed out
ERROR: giving up after 0 retries
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Lokpi\AppData\Local\Programs\Python\Python38\Scripts\bbb-dl.exe\__main__.py", line 7, in <module>
    sys.exit(main())
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\bbb_dl\main.py", line 832, in main
    BBBDL(args.verbose, args.no_check_certificate, args.encoder, args.audiocodec).run(
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\bbb_dl\main.py", line 203, in run
    self.ydl.process_ie_result(deskshare_dl)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1578, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 2661, in process_video_result
    self.process_info(new_info)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 3160, in process_info
    success, real_download = self.dl(temp_filename, info_dict)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 2860, in dl
    return fd.download(name, new_info, subtitle)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\downloader\common.py", line 444, in download
    ret = self.real_download(filename, info_dict)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\downloader\http.py", line 392, in real_download
    self.report_error('giving up after %s retries' % retries)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 971, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "c:\users\lokpi\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 903, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

The other errors are follow up errors (actually not really errors because they are expected, they also should be quiet if you turn of verbose). So the error is a time out error on this file: https://bbb-lb.finki.ukim.mk/presentation/c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416/deskshare/deskshare.webm

And I think it is a server problem, not your internet connection. I tried to access the file multiple times, I once got a server errors. Maybe I'm lucky because I have Gbit/s internet connection and it downloads so fast that I do not get an server error.

Till I update the code to allow more retries you can just try to download that file, and put it in the directory c6c4c950833fd43c76a3f25238bae901c0c49991-1640786126416/deskshare.webm. bbb-dl is smart enough to not redownload it

lokpimkd commented 2 years ago

Aha, thanks for the explanation, downloaded the deskshare.webm and put it in the folder and works like this. Most likely its a server thing as you said

C0D3D3V commented 2 years ago

Could you test please with the new release, if the deskshare.webm gets downloaded (you can stop the program after that with [CTR+C])? I increased the retires counter to 10.

lokpimkd commented 2 years ago

Tried on 4 videos, works awesome now. Thank you a lot. You can close this now