search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.9k stars 10k forks source link

sibnet.ru does not download #18182

Closed LyoSU closed 3 years ago

LyoSU commented 5 years ago

Please follow the guide below

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.11.07. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

Before submitting an issue make sure you have:

What is the purpose of your issue?

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

youtube-dl https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/ -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.11.07
[debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-34-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.4-0ubuntu0.18.04.1, ffprobe 3.4.4-0ubuntu0.18.04.1
[debug] Proxy map: {}
[generic] э: Requesting header
WARNING: Falling back on generic information extractor.
[generic] э: Downloading webpage
[generic] э: Extracting information
[generic] э: Downloading MPD manifest
WARNING: Failed to download MPD manifest: HTTP Error 403: Forbidden
[generic] э: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://video.sibnet.ru/shell.php?videoid=3485877'
[download] Destination: Кот и невидимая преграда-shell.php
[download] 100% of 9.52KiB in 00:00

Video from the site is loaded as php instead of the video itself. This is clearly visible on the penultimate line. I saw that you were previously addressed with this problem, but the question was supposedly closed by a duplicate. The problem still exists, tested on version 2018.11.07

ghost commented 5 years ago

I can confirm this with version 2018.11.23, looks like they updated the player data, now it doesn't include the key and other parameters, e.g.:

https://dv25.sibnet.ru/32/29/93/3229930.m3u8?st=MXCJ_3JuXXVBtPKw2_Tvbw&e=1543369000&stor=25&noip=1&start=

became:

https://video.sibnet.ru/v/5be6d130afd927ae6241de07cdd78f12/3464673.m3u8

And without these keys it returns 403 - forbidden.

ghost commented 5 years ago

Using the headers solves it for me, here bash script:

ydl-sib

#!/bin/bash
url="$1"
shift
youtube-dl --add-header "Host: video.sibnet.ru" \
--add-header "User-Agent: Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36" \
--add-header "Accept: */*" \
--add-header "Referer: $url" \
--add-header "Range: bytes=0-" "$url" "$@"

use it like this:

ydl-sib 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/'

Successfully downloads, outputs: 

[generic] э: Requesting header
WARNING: Falling back on generic information extractor.
[generic] э: Downloading webpage
[generic] э: Extracting information
[download] Destination: Кот и невидимая преграда - видео ролик смотреть на Video.Sibnet.Ru-э.mp4
[download] 100% of 514.57KiB in 00:02
ghost commented 5 years ago

I created some bash script which retrieves direct URL and video title from Sibnet video URL: (codename cusi, from CUrl and SIbnet)

cusi

Updated (2019-02-25 14:47:54):

Updated (2019-02-25 17:01:38):

Updated (2019-02-25 17:19:11):

#!/bin/bash
if [[ "$1" == "" ]]; then
  echo "     cusi By BladeMight, v1.0"
  echo "$0 <sibnet-url> <*commands-str>"
  echo "In command str you can use variables:"
  echo " \\\$du = direct url"
  echo " \\\$ti = title"
  echo " \\\$id = id"
  echo " \\\$au = accept url"
  echo "You must escape variables, so they can be processed in code!, Or you can use single quotes ''"
  echo "command str examples:"
  echo "  cusi <url> 'aria2c \"\$du\" -o \"\$ti\"'"
  echo "  cusi <url> 'ffmpeg -i \"\$du\"'"
  exit 0
fi
url="$1"
H1="Accept-Encoding: identity;q=1, *;q=0"
H2="Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,zh-CN;q=0.6,zh;q=0.5"
H3="User-Agent: Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36"
H4="chrome-proxy: frfr"
H5="Accept: */*"
H6="Connection: keep-alive"
H7="Range: bytes=0-"
H8="Referer: $url"
resp=$(curl -s -H "$H1" -H "$H2" -H "$H3" -H "$H4" -H "$H5" -H "$H6" -H "$H7" --compressed "$url" | iconv -f cp1251 -t utf-8 )
accepturl=$(echo "$resp" | grep -iP '\d+\.mp4' | sed -re 's/.*(\/v\/.*?mp4)", type.*/https:\/\/video.sibnet.ru\1/g')
id=$(echo "$resp" | grep -i DOCTYPE | sed -re 's/.*php\?videoid=([0-9]*).*/\1/g')
echo "Video-Id: $id"
title=$(echo "$resp" | grep -iP 'videoname' | sed -re 's/.*?videoName.>(.*?)<\/h1>.*/\1/g')
echo "Video-Title: $title"
echo "Accept-URL: $accepturl"
directurl=$(curl -I "$accepturl" -s -H "$H1" -H "$H2" -H "$H3" -H "$H4" -H "$H5" -H "$H6" -H "$H7" -H "$H8" --compressed | awk 'BEGIN{IGNORECASE=1} match($0, /location: (.*)/, z) {printf "https:%s", z[1]}')
echo "Direct-URL: $directurl"
if [[ "$2" != "" ]]; then
  echo "Eval mode."
  au="$accepturl"
  du="$directurl"
  ti=$(echo "$title" | sed -re 's/\\|\/|\*|:|\"|<|>|\?|\|//g') # safe title
  eval $2
fi

Use it like this:

$ cusi 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/'
Video-Id: 3485877
Video-Title: Кот и невидимая преграда
Accept-URL: https://video.sibnet.ru/v/b191e90609fb5399adb254d0b25d84e5/3485877.mp4
Direct-URL: https://dv98.sibnet.ru/32/46/57/3246579.mp4?st=4OqCJPY1IMlq4gxP4VzYrA&e=1551128000&stor=26&noip=1

The trick is to use Android user agent, then in the Sibnet video player there is Mp4 Direct Url, then you just need to GET it with correct headers.

And you can use it with aria2c(or just modify the script):

cusi 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/'  | awk '/Title:/{ $1=""; TITLE=substr($0, 2, length($0))}/Direct-URL:/{ $1=""; URL=substr($0,2,length($0)) } END { gsub(/\\|\/|\*|:|\"|<|>|\?|\|/, "", TITLE); printf "aria2c \"%s\" -o \"%s.mp4\"", URL, TITLE}' | sh

With command str mode, you can use even it simpler:

cusi 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/' 'aria2c "$du" -o "$ti"'

or even directly to player, e.g. ffplay:

cusi 'https://video.sibnet.ru/rating/video3485877-Kot_i_nevidimaya_pregrada/%D1%8D/' 'ffplay -i "$du"'

run cusi without arguments to see help.

bato3 commented 5 years ago

one header what you need is "Referer

youtube-dl https://video.sibnet.ru/shell.php?videoid=3422904 --add-header "Referer: https://video.sibnet.ru/shell.php?videoid=3422904"

btw @BladeMight in current cusi you remove 2 first lines and I have error: awk: line 1: syntax error at or near ,

ghost commented 5 years ago

@bato3 Somehow two lines were out in cusi...(i edited the post) Btw, I created repository: BladeMight/cusi, and added some other scripts. I measured the Sibnet download speed of youtube-dl's (MPD/m3u8) and cusi's (direct mp4) and seems like cusi + aria2c is 6-8 faster:

[generic] shell: Requesting header
WARNING: Falling back on generic information extractor.
[generic] shell: Downloading webpage
[generic] shell: Extracting information
[generic] shell: Downloading MPD manifest
[generic] shell: Downloading m3u8 information
[dashsegments] Total fragments: 9
[download] Destination: Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.fdash-v1-x3.mp4
[download] 100% of 5.21MiB in 00:10
[dashsegments] Total fragments: 9
[download] Destination: Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.fdash-a1-x3.m4a
[download] 100% of 862.29KiB in 00:07
[ffmpeg] Merging formats into "Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.mp4"
Deleting original file Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.fdash-v1-x3.mp4 (pass -k to keep)
Deleting original file Доставщик пиццы спросил разрешения сыграть на фортепиано-shell.fdash-a1-x3.m4a (pass -k to keep)

real    0m27.466s
user    0m0.031s
sys     0m0.015s
Video-Id: 3422904
Video-Title: Доставщик пиццы спросил разрешения сыграть на фортепиано
Accept-URL: https://video.sibnet.ru/v/ac3b4b2ebbab7610e5fee5a4b43be09f/3422904.mp4
Direct-URL: https://dv98.sibnet.ru/31/99/63/3199632.mp4?st=1RP51i0cM7pf2iByV3X6xg&e=1551548000&stor=9&noip=1
Eval mode: [#aria]

03/02 11:29:50 [NOTICE] Downloading 1 item(s)
[#cfd09d 5.6MiB/6.0MiB(92%) CN:4 DL:1.9MiB]
03/02 11:29:54 [NOTICE] Download complete: C:/msu/Доставщик пиццы спросил разрешения сыграть на фортепиано.mp4

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
cfd09d|OK  |   1.9MiB/s|C:/msu/Доставщик пиццы спросил разрешения сыграть на фортепиано.mp4

Status Legend:
(OK):download completed.

real    0m7.273s
user    0m0.242s
sys     0m0.394s

About the headers, to get mp4 (not m3u8/mpd) you need the header of UA of Android.

professeurjt commented 11 months ago

yt-dl not working for sibnet , I just tried a simple method and it works well if it helps you to be able to download from sibnet; I don't know if I have the right to leave you the link! but hey I just do it if it can help. Thank you and good day.

youtu.be/xbRl8Vs7pkE