search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.42k stars 9.96k forks source link

[douyu.com] HTTP 403 error Forbidden every 5 minutes #12316

Closed brlala closed 7 years ago

brlala commented 7 years ago

What is the purpose of your issue?

Refer to next comment for log
  1. No account is needed to view the streams/download it. The video streams fine in web browser.
  2. I'm using 'you' instead of 'youtube-dl' to call.
brlala commented 7 years ago

When trying to stream with VLC, able to stream for 5 minutes

D:\>you -o - "https://www.douyu.com/656971"   | vlc -
[DouyuTV] 656971: Downloading room info
[DouyuTV] 656971: Downloading m3u8 information
[download] Destination: -
ffmpeg version N-83585-ga5c1c7a Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-zlib
  libavutil      55. 47.100 / 55. 47.100
  libavcodec     57. 80.101 / 57. 80.101
  libavformat    57. 66.102 / 57. 66.102
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 73.100 /  6. 73.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, hls,applehttp, from 'http://hls1a.douyucdn.cn/live/656971rdHj2s0v3M_550/playlist.m3u8?wsSecret=fe0dc39a673d0a5b46cbf5b4aa6d782c&wsTime=1488370442':
  Duration: N/A, start: 4491.011011, bitrate: N/A
  Program 0
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 960x540, 23 tbr, 90k tbn, 46 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, mono, fltp
    Metadata:
      variant_bitrate : 0
Output #0, mpegts, to 'pipe:':
  Metadata:
    encoder         : Lavf57.66.102
    Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 960x540, q=2-31, 23 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, mono, fltp
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
skipping 1 segments ahead, expired from playlists0:00:09.02 bitrate= 763.5kbits/s speed=1.02x
skipping 3 segments ahead, expired from playlists0:00:48.04 bitrate= 685.5kbits/s speed=   1x
skipping 2 segments ahead, expired from playlists0:01:02.22 bitrate= 586.4kbits/s speed=0.926x
skipping 1 segments ahead, expired from playlists0:02:23.87 bitrate= 636.3kbits/s speed=   1x
skipping 1 segments ahead, expired from playlists0:02:39.02 bitrate= 630.8kbits/s speed=0.995x
skipping 1 segments ahead, expired from playlists0:03:47.78 bitrate= 649.3kbits/s speed=0.997x
skipping 1 segments ahead, expired from playlists0:03:53.57 bitrate= 642.9kbits/s speed=0.993x
skipping 2 segments ahead, expired from playlists0:03:59.64 bitrate= 635.3kbits/s speed=0.986x
skipping 1 segments ahead, expired from playlists0:04:08.29 bitrate= 621.2kbits/s speed=0.997x
skipping 2 segments ahead, expired from playlists0:04:17.26 bitrate= 619.0kbits/s speed=0.981x
skipping 2 segments ahead, expired from playlists0:04:26.20 bitrate= 606.9kbits/s speed=0.987x
skipping 2 segments ahead, expired from playlists0:04:33.04 bitrate= 594.1kbits/s speed=0.974x
skipping 3 segments ahead, expired from playlists0:04:42.06 bitrate= 582.7kbits/s speed=0.975x
[http @ 00000000035112a0] HTTP error 403 Forbidden:04:54.04 bitrate= 566.7kbits/s speed=0.976x
[hls,applehttp @ 00000000024524a0] Failed to reload playlist 0
[http @ 0000000003510700] HTTP error 403 Forbidden
[hls,applehttp @ 00000000024524a0] Failed to reload playlist 0
frame= 5327 fps= 18 q=-1.0 Lsize=   20621kB time=00:04:56.97 bitrate= 568.8kbits/s speed=0.985x
video:14738kB audio:3915kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 10.551546%
ERROR: unable to download video data: [WinError 2] The system cannot find the file specified: '-'

When trying to download, pops up error HTTP 403 after 5 minutes

D:\>you https://www.douyu.com/656971
[DouyuTV] 656971: Downloading room info
[DouyuTV] 656971: Downloading m3u8 information
[download] Destination: 救命！我变成小姐姐了！ 2017-03-01 21_20-656971.mp4
ffmpeg version N-83585-ga5c1c7a Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-zlib
  libavutil      55. 47.100 / 55. 47.100
  libavcodec     57. 80.101 / 57. 80.101
  libavformat    57. 66.102 / 57. 66.102
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 73.100 /  6. 73.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, hls,applehttp, from 'http://hls1a.douyucdn.cn/live/656971rdHj2s0v3M_550/playlist.m3u8?wsSecret=6e201f0b18b8f7a313fb11ce029cc090&wsTime=1488371141':
  Duration: N/A, start: 5193.009011, bitrate: N/A
  Program 0
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 960x540, 23 tbr, 90k tbn, 46 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, mono, fltp
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:鏁戝懡锛佹垜鍙樻垚灏忓濮愪簡锛?2017-03-01 21_20-656971.mp4.part':
  Metadata:
    encoder         : Lavf57.66.102
    Stream #0:0: Video: h264 (High) ([33][0][0][0] / 0x0021), yuv420p, 960x540, q=2-31, 23 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([64][0][0][0] / 0x0040), 44100 Hz, mono, fltp
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
skipping 1 segments ahead, expired from playlists0:01:13.59 bitrate= 652.2kbits/s speed=0.989x
skipping 1 segments ahead, expired from playlists0:01:28.62 bitrate= 627.3kbits/s speed=0.987x
[http @ 00000000035c9120] HTTP error 403 Forbidden:04:59.43 bitrate= 635.7kbits/s speed=1.02x
[hls,applehttp @ 00000000006324a0] Failed to reload playlist 0
[http @ 00000000035c9120] HTTP error 403 Forbidden
[hls,applehttp @ 00000000006324a0] Failed to reload playlist 0
frame= 6755 fps= 23 q=-1.0 Lsize=   23543kB time=00:04:59.97 bitrate= 643.0kbits/s speed=1.02x
video:18665kB audio:4704kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.746939%
[ffmpeg] Downloaded 24108527 bytes
[download] 100% of 22.99MiB
brlala commented 7 years ago

From @lexcn he said that a valid sign value cannot be obtained, probably due to invalid key.

无法计算出可用的sign值，youtube-dl使用的是同一个key也出现了同样的问题

yan12125 commented 7 years ago

When trying to download, pops up error immediately

I'm not sure whether it's a bug of youtube-dl on Windows or ffmpeg. It works for almost 5 minutes on Linux:

$ youtube-dl -v "https://www.douyu.com/1516303"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.douyu.com/1516303']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.02.28
[debug] Git HEAD: aa9cc2ecb
[debug] Python version 3.6.0 - Linux-4.10.1-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.2.4, ffprobe 3.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[DouyuTV] 1516303: Downloading room info
[DouyuTV] 1516303: Downloading m3u8 information
[debug] Invoking downloader on 'http://hls3a.douyucdn.cn/live/1516303rzXhnTa0N/playlist.m3u8?wsSecret=d3ef6d06a50baa747bd3123ad8d97271&wsTime=1488380673'
[download] Destination: 一个人吃饭 旅行 走走停停... 2017-03-01 23_59-1516303.mp4
[debug] ffmpeg command line: ffmpeg -y -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
' -i 'http://hls3a.douyucdn.cn/live/1516303rzXhnTa0N/playlist.m3u8?wsSecret=d3ef6d06a50baa747bd3123ad8d97271&wsTime=1488380673' -c copy -f mp4 'file:一个人吃饭 旅行 走走停停... 2017-03-01 23_59-1516303.mp4.part'
ffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.1 (GCC) 20170109
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Input #0, hls,applehttp, from 'http://hls3a.douyucdn.cn/live/1516303rzXhnTa0N/playlist.m3u8?wsSecret=d3ef6d06a50baa747bd3123ad8d97271&wsTime=1488380673':
  Duration: N/A, start: 63.936011, bitrate: N/A
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (Baseline) ([27][0][0][0] / 0x001B), yuv420p, 368x640, 90k tbr, 90k tbn, 180k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, mono, fltp
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:一个人吃饭 旅行 走走停停... 2017-03-01 23_59-1516303.mp4.part':
  Metadata:
    encoder         : Lavf57.56.101
    Stream #0:0: Video: h264 (Baseline) ([33][0][0][0] / 0x0021), yuv420p, 368x640, q=2-31, 90k tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([64][0][0][0] / 0x0040), 44100 Hz, mono
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
skipping 3 segments ahead, expired from playlists0:00:02.30 bitrate=1578.9kbits/s speed=0.341x    
skipping 2 segments ahead, expired from playlists0:00:14.44 bitrate= 550.1kbits/s speed=0.935x                                          
skipping 6 segments ahead, expired from playlists0:00:23.78 bitrate= 554.8kbits/s speed=0.709x                                          
skipping 4 segments ahead, expired from playlists0:00:44.24 bitrate= 384.4kbits/s speed=0.867x                                          
skipping 3 segments ahead, expired from playlists0:00:59.91 bitrate= 376.4kbits/s speed=0.926x                                          
skipping 5 segments ahead, expired from playlists0:01:11.44 bitrate= 373.4kbits/s speed=0.873x                                          
skipping 4 segments ahead, expired from playlists0:01:29.18 bitrate= 342.2kbits/s speed=0.932x                                          
skipping 5 segments ahead, expired from playlists0:01:44.81 bitrate= 343.2kbits/s speed=0.926x                                          
skipping 6 segments ahead, expired from playlists0:02:02.59 bitrate= 334.8kbits/s speed=0.911x                                          
skipping 2 segments ahead, expired from playlists0:02:23.10 bitrate= 314.3kbits/s speed=0.984x                                          
skipping 5 segments ahead, expired from playlists0:02:32.55 bitrate= 330.8kbits/s speed=0.936x                                          
skipping 3 segments ahead, expired from playlists0:02:50.61 bitrate= 321.0kbits/s speed=0.967x                                          
skipping 5 segments ahead, expired from playlists0:03:02.60 bitrate= 320.8kbits/s speed=0.953x                                          
skipping 5 segments ahead, expired from playlists0:03:20.73 bitrate= 319.4kbits/s speed=0.953x                                          
skipping 4 segments ahead, expired from playlists0:03:38.89 bitrate= 317.4kbits/s speed=0.967x                                          
skipping 3 segments ahead, expired from playlists0:03:53.50 bitrate= 315.2kbits/s speed=0.977x                                          
skipping 3 segments ahead, expired from playlists0:04:05.27 bitrate= 315.4kbits/s speed=0.978x                                          
skipping 4 segments ahead, expired from playlists0:04:17.37 bitrate= 316.1kbits/s speed=0.977x                                          
skipping 5 segments ahead, expired from playlists0:04:33.08 bitrate= 318.2kbits/s speed=0.965x                                          
[http @ 0x5648423e4a00] HTTP error 403 Forbidden00:04:50.20 bitrate= 312.9kbits/s speed=0.975x                                          
[hls,applehttp @ 0x5648423d96e0] Failed to reload playlist 0                                                                            
[http @ 0x5648423e47c0] HTTP error 403 Forbidden                                                                                        
[hls,applehttp @ 0x5648423d96e0] Failed to reload playlist 0                                                                            
frame= 1360 fps=4.6 q=-1.0 Lsize=   11188kB time=00:04:50.48 bitrate= 315.5kbits/s speed=0.975x                                         
video:10211kB audio:994kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exception ignored in: <_io.FileIO name=6 mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedWriter name=6>
[ffmpeg] Downloaded 11456746 bytes
[download] 100% of 10.93MiB

Could you try to run the ffmpeg command in the [debug] line directly and see what's the output?

that a valid sign value cannot be obtained, probably due to invalid key.

Nope. In current version there are no more keys.

brlala commented 7 years ago

I'm using a .exe file instead of the Python. What is the command to run to debug my ffmpeg?

Out of curiosity, why does the stream stops at 5 minutes using youtube-dl but it does not happens when you watch the stream in Chrome? Thanks.

yan12125 commented 7 years ago

What is the command to run to debug my ffmpeg?

This one:

ffmpeg -y -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
' -i 'http://hls3a.douyucdn.cn/live/1516303rzXhnTa0N/playlist.m3u8?wsSecret=d3ef6d06a50baa747bd3123ad8d97271&wsTime=1488380673' -c copy -f mp4 'file:一个人吃饭 旅行 走走停停... 2017-03-01 23_59-1516303.mp4.part'

it does not happens when you watch the stream in Chrome

youtube-dl uses the API for iPhone, which returns m3u8, while on desktop douyu still uses flv.

brlala commented 7 years ago

Sorry I'm new to this where should i key in the command in CMD?

C:\Windows\system32>ffmpeg -y -headers
ffmpeg version N-83657-g7e4f32f Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-zlib
  libavutil      55. 47.100 / 55. 47.100
  libavcodec     57. 81.100 / 57. 81.100
  libavformat    57. 66.102 / 57. 66.102
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 74.100 /  6. 74.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Unrecognized option 'headers'.
Error splitting the argument list: Option not found

youtube-dl uses the API for iPhone, which returns m3u8, while on desktop douyu still uses flv.

So does the token on iPhone gets reset every 5 minutes? What's the difference between watching on iPhone and using youtube-dl?

yan12125 commented 7 years ago

IIRC CMD does not support multiline commands. Try this:

ffmpeg -y -i "http://hls3a.douyucdn.cn/live/1516303rzXhnTa0N/playlist.m3u8?wsSecret=d3ef6d06a50baa747bd3123ad8d97271&wsTime=1488380673" -c copy -f mp4 "file:一个人吃饭 旅行 走走停停... 2017-03-01 23_59-1516303.mp4.part"

You'll need to replace the m3u8 url as it gets expired every 5 minutes.

So does the token on iPhone gets reset every 5 minutes? What's the difference between watching on iPhone and using youtube-dl?

Actually I use Safari with iPhone user-agent and it doesn't really play. Only an error message says that it can't load the m3u8. It would be great if someone can try it on a real iPhone and see what's going on.

chacha1024 commented 7 years ago

Some one get the NEW API KEY here

yan12125 commented 7 years ago

I know that. As that work has no license declaration, youtube-dl can't use it.

brlala commented 7 years ago

@yan12125 https://github.com/spacemeowx2/DouyuHTML5Player/blob/master/package.json MIT license provided here. Use as you wish.

package.json 里写的是 MIT, 随便用吧

brlala commented 7 years ago

@debugzxcv

https://gist.github.com/debugzxcv/85bb2750d8a5e29803f2686c47dc236b https://github.com/spacemeowx2/DouyuHTML5Player/commit/5065e5e

yan12125 commented 7 years ago

Again, it's @spacemeowx2 that finds out a simple API to get a working FLV URL. Thanks!