search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.91k stars 10.01k forks source link

Request to support box.com #5949

Open cadeceus opened 9 years ago

cadeceus commented 9 years ago

I would like youtube-dl to be able to download from box.com, please.

Example: https://app.box.com/s/yrvxrv91fenex2acvo0zdpbogjjb9jrf

RingoTheDog commented 5 years ago

Bump....

New example: https://mlssoccer.app.box.com/s/0evd2o3e08l60lr4ygukepvnkord1o1x/file/510727257538?sb=/details

bquast commented 4 years ago

I third this

zahnz commented 3 years ago

Fourth'd! Is there a way we can get started on this? Can someone post some guidance?

I tried grabbing the DASH mpd manifest URL (e.g. from the Networks tab in Google Chrome) and tried to download with that and with cookies, but I still get a 401 Unauthorized error like this, and youtube-dl eventually gives up. Has anyone ever seen this before?

...
[dashsegments] Total fragments: 623
[download] Destination: test.mp4
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 1 of 10)...
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 2 of 10)...
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 3 of 10)...
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 4 of 10)...
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 5 of 10)...
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 6 of 10)...
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 7 of 10)...
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 8 of 10)...
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 9 of 10)...
[download] Got server HTTP error: HTTP Error 401: Unauthorized. Retrying fragment 1 (attempt 10 of 10)...
ERROR: giving up after 10 fragment retries

I've found that manually downloading random samples of these .m4s files separately with youtube-dl does work though, including init.m4s, 1.m4s, and the last .m4s in the series (in this case, 622.m4s). Cookies aren't even needed. But this is strange. The error message above seems to indicate that we should also get an unauthorized error for each of these m4s fragments, but none in my random sample gave a 401 error, and I'm not even using cookies. It is possible I didn't test enough of them, but it seems that youtube-dl's generic extractor is not doing something right, I guess. Turning on verbose mode doesn't seem to uncover any insight.

Note that this could be automated and they could be stitched together like this post suggests: https://superuser.com/questions/1204497/download-all-m4s-files-of-a-mpeg-dash-stream

Edit: Here are better instructions for stitching the m4s into an mp4: https://stackoverflow.com/questions/52705023/ffmpeg-converting-m4s-to-mp4

Edit: I've made a rudimentary Bash script which automates this process. I have very little Bash experience and even less time, so apologies if this is poorly coded. You'll also have to edit it if the video stream is not 1080p (replace "1080" with whatever the right resolution is), and you may have to edit the audio stream as well (from "0" to something else). You'll need the .m4s file URL from the Networks tab of Google Chrome, which must be open when the video player page loads. Finally, you'll need the number of .m4s files in the stream, which will be shown if you finish playing the video (when it ends, URLs for .m4s files will stop appearing in the Networks tab, and you can just enter the greatest number you see into the script).

#!/usr/bin/bash

# number of .m4s files in stream; get from Networks tab of video player after player finishes
N=650

# URL with init.m4s for video from Networks tab; example below
URL='https://dl.boxcloud.com/api/2.0/internal_files/<some integer ID here>/versions/<some other integer ID here>/representations/dash/content/video/1080/init.m4s?access_token=<quite a long encoded string>&shared_link=<seems to be the URL encoded link you visited to play the video>&box_client_name=box-content-preview&box_client_version=<some version number>'

URLSPLIT=(${URL//video\/1080\/init/ })

# download initial video m4s
youtube-dl "${URLSPLIT[0]}video/1080/init${URLSPLIT[1]}" -o "vinit.m4s"
# download initial audio m4s
youtube-dl "${URLSPLIT[0]}audio/0/init${URLSPLIT[1]}" -o "ainit.m4s"

# download all other m4s
for ((MNUM=1;MNUM<=N;MNUM++));
do
    youtube-dl "${URLSPLIT[0]}video/1080/$MNUM${URLSPLIT[1]}" -o "v$MNUM.m4s"
    youtube-dl "${URLSPLIT[0]}audio/0/$MNUM${URLSPLIT[1]}" -o "a$MNUM.m4s"
done

# concatenate m4s into one
cat vinit.m4s >> vall.m4s
cat ainit.m4s >> aall.m4s
for ((MNUM=1;MNUM<=N;MNUM++)); do cat "v$MNUM.m4s" >> vall.m4s; done
for ((MNUM=1;MNUM<=N;MNUM++)); do cat "a$MNUM.m4s" >> aall.m4s; done

# turn m4s into complete.mp4
ffmpeg -i aall.m4s -i vall.m4s -c:v copy -c:a aac complete.mp4

Hope this helps someone until the youtube-dl devs complete the Box implementation.

hmartinez82 commented 2 years ago

@remitamine @ThirumalaiK I saw this TODO to add the query to each fragment commented out in https://github.com/ytdl-org/youtube-dl/commit/c84f9475b8df0f892b631966159e1649dafe13f0#diff-d6894be6fa794f1c1ff868400eaac26e2fbf780e7b175ad9815d47445fa8c67cR72 Isn't that what's missing?

xnileshx commented 1 year ago

I'm hoping if someone could help me see if they are able to download from https://sunsetlearning.app.box.com/s/23inj0thkhosu34xqpg85v9rs4zgudqk/file/876001645317 All of my attempts have failed. Maybe I'm doing it wrong. 

C:\>youtube-dl https://sunsetlearning.app.box.com/s/23inj0thkhosu34xqpg85v9rs4zgudqk/file/875996902995 extractor Box

[Box] 875996902995: Downloading webpage
[Box] 875996902995: Downloading token JSON metadata
[Box] 875996902995: Downloading file JSON metadata
ERROR: unable to download video data: HTTP Error 403: Forbidden

Verbose

C:\Users\admin\Videos>youtube-dl https://sunsetlearning.app.box.com/s/23inj0thkhosu34xqpg85v9rs4zgudqk/file/876001645317 extractor Box -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://sunsetlearning.app.box.com/s/23inj0thkhosu34xqpg85v9rs4zgudqk/file/876001645317', 'extractor', 'Box', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 2022-11-03-git-5ccd4d3060-full_build-www.gyan.dev, ffprobe 2022-11-03-git-5ccd4d3060-full_build-www.gyan.dev
[debug] Proxy map: {}
[Box] 876001645317: Downloading webpage
[Box] 876001645317: Downloading token JSON metadata
[Box] 876001645317: Downloading file JSON metadata
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://dl2.boxcloud.com/api/2.0/files/876001645317/content?access_token=1%216q7lXhrXftVRrsDHMLg4OGPDJcKBJ8yCrdR_MgbEvfCFJNDRjyb4AK6foQCDgyJYDtnGa-_DeHI6A2XOyGjGVF8wk6P9mxBYCkeqHojdX9HyMhaKonlCZBKgpZ4I0M8lboW0-cnvNIE5KPecWGw0zqSz65PFpRd_wSd0pycsv6Mk_vCZn4KXWy6LwmzpcAfh7-AvhZyqke0Wf1EwnkOZmfLFjieEcZMIN0xmAndvAE2l1zFnz-pU7sXCpif3E5f6iWh4v8fQ7YcEAQYBrG4ow_ROkXt7XYYlwkIc0OSCJF9o_8aO3YCQlGBRikrDRirTPB8JUj2b5aZn0LqzdIsc9_Q0o87cm1QzriOWaQiGrzhMw1UdFMCUti4MaoZJnLIMbGK6J1YssTi73G6rrScR_fpv_NvkFf3jeak_W_JhpWQGfnKdcvXLqR1z9hQ7z3RTFmWeXv9siku5VO5wYOUU0yAhzjUNvY0RlfRHD-6Vg_99JyPUwk4tyip6GpY6-LGimXxsXKO0lpS2YPS4ycLxWPPZBxaNaVp_4YZYeQcXL-tOQtZfZBvrSuA7nMmgibhkogNP&shared_link=https%3A%2F%2Fapp.box.com%2Fs%2F23inj0thkhosu34xqpg85v9rs4zgudqk'
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 1976, in process_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 1915, in dl
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\downloader\common.py", line 366, in download
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\downloader\http.py", line 351, in real_download
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\downloader\http.py", line 116, in establish_connection
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\downloader\http.py", line 110, in establish_connection
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 2288, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden
soppelmann commented 1 year ago

This PR works. Just replace the box.py file in the extractors folder. https://github.com/ytdl-org/youtube-dl/pull/27874

kampiamodikuesto commented 1 year ago 

# number of .m4s files in stream; get from Networks tab of video player after player finishes
N=665

# URL with init.m4s for video from Networks tab; example below
URL="https://public.boxcloud.com/api/2.0/internal_files/>FILENAME>/versions/1373998591020/representations/dash/content/video/1080/init.m4s?access_token=<TOKEN>&box_client_name=box-content-preview&box_client_version=2.95.0"

URLSPLIT=(${URL//video\/1080\/init/ })

# Create audio and video directories if they don't exist
mkdir -p audio
mkdir -p video

# Rimuovi i file esistenti dalla cartella "audio"
rm -f audio/*

# Rimuovi i file esistenti dalla cartella "video"
rm -f video/*

# download initial video m4s
youtube-dl "${URLSPLIT[0]}video/1080/init${URLSPLIT[1]}" -o "video/vinit.m4s"
# download initial audio m4s
youtube-dl "${URLSPLIT[0]}audio/0/init${URLSPLIT[1]}" -o "audio/ainit.m4s"

# download all other m4s
for ((MNUM=1;MNUM<=N;MNUM++));
do
    youtube-dl "${URLSPLIT[0]}video/1080/$MNUM${URLSPLIT[1]}" -o "video/v$MNUM.m4s"
    youtube-dl "${URLSPLIT[0]}audio/0/$MNUM${URLSPLIT[1]}" -o "audio/a$MNUM.m4s"
done

# concatenate m4s into one
cat video/vinit.m4s >> vall.m4s
cat audio/ainit.m4s >> aall.m4s
for ((MNUM=1;MNUM<=N;MNUM++)); do cat "v$MNUM.m4s" >> vall.m4s; done
for ((MNUM=1;MNUM<=N;MNUM++)); do cat "a$MNUM.m4s" >> aall.m4s; done

# turn m4s into complete.mp4
ffmpeg -i aall.m4s -i vall.m4s -c:v copy -c:a aac complete.mp4

All audio and video m4s files are downloaded but then i get and wrong mp4 file impossible to read. This is the console output:


  Metadata:
    major_brand     : iso5
    minor_version   : 512
    compatible_brands: iso5iso6mp41
    encoder         : Lavf60.3.100
  Duration: N/A, bitrate: N/A
  Stream #0:0[0x1](und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fe386c167c0] Could not find codec parameters for stream 0 (Video: h264 (avc1 / 0x31637661), none, 1280x720): unspecified pixel format
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'vall.m4s':
  Metadata:
    major_brand     : iso5
    minor_version   : 512
    compatible_brands: iso5iso6mp41
    encoder         : Lavf60.3.100
  Duration: N/A, bitrate: N/A
  Stream #1:0[0x1](und): Video: h264 (avc1 / 0x31637661), none, 1280x720, SAR 1:1 DAR 16:9, 6144 tbr, 12288 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
File 'complete.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #1:0 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
Output #0, mp4, to 'complete.mp4':
  Metadata:
    major_brand     : iso5
    minor_version   : 512
    compatible_brands: iso5iso6mp41
    encoder         : Lavf60.3.100
  Stream #0:0(und): Video: h264 (avc1 / 0x31637661), none, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 6144 tbr, 12288 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.3.100 aac
frame=    0 fps=0.0 q=-1.0 size=       0kB time=-577014:32:22.77 bitrate=  -frame=    0 fps=0.0 q=-1.0 Lsize=       0kB time=-577014:32:22.77 bitrate=N/A speed=N/A    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[aac @ 0x7fe386c19340] Qavg: nan````
kampiamodikuesto commented 1 year ago

finally this script give me the correct results:

#!/bin/bash

# number of .m4s files in stream; get from Networks tab of video player after player finishes
N=665

# URL with init.m4s for video from Networks tab; example below
URL='https://public.boxcloud.com/api/2.0/internal_files/XXXXXXXX/versions/XXXXXXXXX/representations/dash/content/video/1080/init.m4s?access_token=XXXXXXXXXXXXX&box_client_name=box-content-preview&box_client_version=2.95.0'

URLSPLIT=(${URL//video\/1080\/init/ })

# Create audio and video directories if they don't exist
mkdir -p audio
mkdir -p video

# Rimuovi i file esistenti dalla cartella "audio"
rm -f audio/*

# Rimuovi i file esistenti dalla cartella "video"
rm -f video/*

# download initial video m4s
youtube-dl "${URLSPLIT[0]}video/1080/init${URLSPLIT[1]}" -o "video/vinit.m4s" --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" --referer "https://app.box.com/"
# download initial audio m4s
youtube-dl "${URLSPLIT[0]}audio/0/init${URLSPLIT[1]}" -o "audio/ainit.m4s"

# download all other m4s
for ((MNUM=1;MNUM<=N;MNUM++));
do
    youtube-dl "${URLSPLIT[0]}video/1080/$MNUM${URLSPLIT[1]}" -o "video/v$MNUM.m4s"
    youtube-dl "${URLSPLIT[0]}audio/0/$MNUM${URLSPLIT[1]}" -o "audio/a$MNUM.m4s"
done

# concatenate m4s into one
cat video/vinit.m4s >> vall.m4s
cat audio/ainit.m4s >> aall.m4s
for ((MNUM=1;MNUM<=N;MNUM++)); do cat "video/v$MNUM.m4s" >> vall.m4s; done
for ((MNUM=1;MNUM<=N;MNUM++)); do cat "audio/a$MNUM.m4s" >> aall.m4s; done

# turn m4s into complete.mp4
ffmpeg -i aall.m4s -i vall.m4s -c:v copy -c:a aac complete.mp4
dirkf commented 1 year ago

The solution was already posted in https://github.com/ytdl-org/youtube-dl/issues/5949#issuecomment-1387121658: test the PR.