chrippa / livestreamer

Command-line utility that extracts streams from various services and pipes them into a video player of choice. No longer maintained, use streamlink or youtube-dl instead.
http://livestreamer.io/
BSD 2-Clause "Simplified" License
3.88k stars 583 forks source link

Unable to download Twitch.tv segments (11/21 segments) #620

Closed tamnm closed 9 years ago

tamnm commented 9 years ago

OS: Windows 8 x64 Area: South-East Asia VLC Version: 2.1.5 Rincewind LiveStreamer version: livestreamer-v1.10.2-win32-setup.exe StreamUrl: http://www.twitch.tv/jamielinux Logs:


C:\Users\ManhTam>livestreamer http://www.twitch.tv/jamielinux worst --loglevel d
ebug
[cli][info] Found matching plugin twitch for URL http://www.twitch.tv/jamielinux

[cli][info] Available streams: audio, source (worst, best)
[cli][info] Opening stream: source (hls)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 605 to queue
[cli][debug] Pre-buffering 8192 bytes
[stream.hls][debug] Adding segment 606 to queue
[stream.hls][debug] Adding segment 607 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 608 to queue
[stream.hls][debug] Adding segment 609 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 610 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 611 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 612 to queue
[stream.hls][debug] Adding segment 613 to queue
[stream.hls][debug] Download of segment 605 complete
[cli][info] Starting player: 'C:\Program Files (x86)\VideoLAN\VLC\vlc.exe'
[cli][debug] Writing stream to output
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 614 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 615 to queue
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 616 to queue
[stream.hls][debug] Adding segment 617 to queue
[stream.hls][debug] Download of segment 606 complete
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 618 to queue
[stream.hls][debug] Download of segment 607 complete
[stream.hls][debug] Adding segment 619 to queue
[stream.hls][debug] Download of segment 608 complete
[stream.hls][error] Failed to open segment 609: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000609-JOPO.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 609: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000609-JOPO.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 609: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000609-JOPO.ts (404 Client Error: Not Found)
[stream.hls][debug] Reloading playlist
[stream.hls][debug] Adding segment 623 to queue
[stream.hls][debug] Adding segment 624 to queue
[stream.hls][debug] Download of segment 610 complete
[stream.hls][debug] Adding segment 625 to queue
[stream.hls][error] Failed to open segment 611: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000611-PYCC.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 611: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000611-PYCC.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 611: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000611-PYCC.ts (404 Client Error: Not Found)
[stream.hls][debug] Adding segment 626 to queue
[stream.hls][error] Failed to open segment 612: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000612-JQ75.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 612: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000612-JQ75.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 612: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000612-JQ75.ts (404 Client Error: Not Found)
[stream.hls][debug] Adding segment 627 to queue
[stream.hls][error] Failed to open segment 613: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000613-Wd8A.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 613: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000613-Wd8A.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 613: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000613-Wd8A.ts (404 Client Error: Not Found)
[stream.hls][debug] Adding segment 628 to queue
[stream.hls][error] Failed to open segment 614: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000614-9wvn.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 614: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000614-9wvn.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 614: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000614-9wvn.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 615: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000615-TzdP.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 615: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000615-TzdP.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 615: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000615-TzdP.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 616: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000616-U6Mn.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 616: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000616-U6Mn.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 616: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000616-U6Mn.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 617: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000617-zXzM.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 617: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000617-zXzM.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 617: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000617-zXzM.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 618: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000618-A5YF.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 618: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000618-A5YF.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 618: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000618-A5YF.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 619: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000619-eUg7.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 619: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000619-eUg7.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 619: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000619-eUg7.ts (404 Client Error: Not Found)
[stream.hls][error] Failed to open segment 623: Unable to open URL: http://video17.sfo01.hls.twitch.tv/hls46/jamielinux_12023834576_171385834/chunked/index-0000000623-Qmju.ts (404 Client Error: Not Found)
chrippa commented 9 years ago

Looks like your connection to Twitch is too slow, causing you to fall behind on downloading segments, eventually reaching a point where the segments are no longer available.

In some cases it can be your ISP limiting each individual connection, in which case the --hls-segment-threads option might help, since it will open multiple connections.

trocknet commented 9 years ago

I met the same problem. when fetch a segment failed(i watch afreecatv and met http 400),it did not fetch the next segment immediately, it will wait a long time,even exceed the timeout option.(a segment's duration is 3 seconds,so i set timeout to 3). and the queue segment number will increase faster then the fetch segment number. but when the 2 number distance reach to 10,it will fetch the next immediately. if set --hls-segment-threads=10 might help? btw, is there not a option the set the queue size?change this line might help?

def __init__(self, reader, size=10, retries=None, threads=None, timeout=None):
chrippa commented 9 years ago

btw, is there not a option the set the queue size?change this line might help?

Since we added parallel downloads it's now possible to fill the queue if all threads are downloading, I've raised the limit to 20 in f4c083b2a836d8cc2cd73a9036c5d89f868c32b9.

trocknet commented 9 years ago

Er......I think i should reduce the value. Since i watching a live,not a vod. When a thread fetch a segment failed,it will wait a long time.Is this a bug?

chrippa commented 9 years ago

When a thread fetch a segment failed,it will wait a long time.Is this a bug?

By default a segment won't be skipped until 3 attempts to download it have been made, you can use the --hls-segment-attempts option to change this.

trocknet commented 9 years ago

my setting are: http-timeout=8 hls-segment-timeout=5 hls-timeout=15 hls-segment-attempts=1

and these 2 days,I set --hls-segment-threads=10 , it work much better.

trocknet commented 9 years ago

I add time to debug output:

C:\Users\rocknet>livestreamer -l debug afreeca.com/rlaxordyd best
[cli][info] Found matching plugin afreecatv for URL afreeca.com/rlaxordyd
[cli][info] Available streams: live (worst, best)
[cli][info] Opening stream: live (hls)
[stream.hls][debug] 2014-12-21 12:47:59 Reloading playlist
[cli][debug] Pre-buffering 8192 bytes
[stream.hls][debug] 2014-12-21 12:47:59 Adding segment 1247 to queue
[stream.hls][debug] 2014-12-21 12:47:59 fetch:1247
[stream.hls][debug] 2014-12-21 12:47:59 Adding segment 1248 to queue
[stream.hls][debug] 2014-12-21 12:48:00 Adding segment 1249 to queue
[stream.hls][debug] 2014-12-21 12:48:00 fetch:1248
[stream.hls][debug] 2014-12-21 12:48:00 fetch:1249
[stream.hls][debug] 2014-12-21 12:48:03 Reloading playlist
[stream.hls][debug] 2014-12-21 12:48:03 Download of segment 1247 complete
[cli][info] Starting player: "d:\Program Files (x86)\VideoLAN\VLC\vlc.exe" --fil
e-caching=5000
[stream.hls][debug] 2014-12-21 12:48:03 Adding segment 1250 to queue
[stream.hls][debug] 2014-12-21 12:48:03 fetch:1250
[cli][debug] Writing stream to output
[stream.hls][debug] 2014-12-21 12:48:06 Reloading playlist
[stream.hls][debug] 2014-12-21 12:48:06 Adding segment 1251 to queue
[stream.hls][debug] 2014-12-21 12:48:06 fetch:1251
[stream.hls][debug] 2014-12-21 12:48:09 Reloading playlist
[stream.hls][debug] 2014-12-21 12:48:09 Adding segment 1252 to queue
[stream.hls][debug] 2014-12-21 12:48:09 fetch:1252
[stream.hls][debug] 2014-12-21 12:48:12 Reloading playlist
[stream.hls][debug] 2014-12-21 12:48:13 Adding segment 1253 to queue
[stream.hls][debug] 2014-12-21 12:48:13 fetch:1253
[stream.hls][debug] 2014-12-21 12:48:16 Reloading playlist
[stream.hls][debug] 2014-12-21 12:48:20 Adding segment 1254 to queue
[stream.hls][debug] 2014-12-21 12:48:20 fetch:1254
[stream.hls][debug] 2014-12-21 12:48:20 Adding segment 1255 to queue
[stream.hls][debug] 2014-12-21 12:48:20 fetch:1255
[stream.hls][debug] 2014-12-21 12:48:20 Download of segment 1248 complete
[stream.hls][debug] 2014-12-21 12:48:23 Reloading playlist
[stream.hls][debug] 2014-12-21 12:48:23 Adding segment 1256 to queue
[stream.hls][debug] 2014-12-21 12:48:23 fetch:1256
[stream.hls][debug] 2014-12-21 12:48:24 Download of segment 1249 complete
[stream.hls][debug] 2014-12-21 12:48:24 Download of segment 1250 complete
[stream.hls][debug] 2014-12-21 12:48:24 Download of segment 1251 complete
[stream.hls][debug] 2014-12-21 12:48:26 Download of segment 1252 complete
[stream.hls][debug] 2014-12-21 12:48:26 Download of segment 1253 complete
[stream.hls][debug] 2014-12-21 12:48:26 Download of segment 1254 complete
[stream.hls][debug] 2014-12-21 12:48:26 Reloading playlist
[stream.hls][debug] 2014-12-21 12:48:26 Adding segment 1257 to queue
[stream.hls][debug] 2014-12-21 12:48:26 fetch:1257
[stream.hls][debug] 2014-12-21 12:48:26 Download of segment 1255 complete
[stream.hls][debug] 2014-12-21 12:48:26 Download of segment 1256 complete
[stream.hls][debug] 2014-12-21 12:48:29 Reloading playlist
[stream.hls][debug] 2014-12-21 12:48:29 Adding segment 1258 to queue
[stream.hls][debug] 2014-12-21 12:48:29 fetch:1258
[cli][info] Player closed
[stream.hls][debug] Closing worker thread
[stream.hls][debug] Closing writer thread
[cli][info] Stream ended

and my config:

http-timeout=10
hls-segment-timeout=3
hls-timeout=20
hls-segment-attempts=1
hls-live-edge=5
stream-types=hls,rtmp,hds,http,akamaihd
stream-segment-threads=10
hls-segment-threads=10

when network is not good,this will happen. notice this:

12:48:00 fetch:1248
12:48:20 Download of segment 1248 complete

the time exceed the timeout option.

And this:

[stream.hls][debug] 2014-12-21 12:48:24 Download of segment 1249 complete
[stream.hls][debug] 2014-12-21 12:48:24 Download of segment 1250 complete
[stream.hls][debug] 2014-12-21 12:48:24 Download of segment 1251 complete
[stream.hls][debug] 2014-12-21 12:48:26 Download of segment 1252 complete
[stream.hls][debug] 2014-12-21 12:48:26 Download of segment 1253 complete
[stream.hls][debug] 2014-12-21 12:48:26 Download of segment 1254 complete

it display together.

Seems to have a problem with thread synchronization?

chrippa commented 9 years ago

No, I think this is normal. It's just that the debug message is a little bit misleading, "Download of segment XXX complete" actually means "Download of segment XXX complete AND has been written to the buffer".

What is happening here is that we must wait to write each segment to the buffer in the correct order to avoid playing the segments in the wrong order. If the first segment in the queue is downloaded slower than the ones after, it will look like they finish at the same time since we must process them in order.

the time exceed the timeout option.

The timeout may not mean exactly what you think, here is a quote from the requests documentation:

timeout is not a time limit on the entire response download; rather, an exception is raised if the server has not issued a response for timeout seconds (more precisely, if no bytes have been received on the underlying socket for timeout seconds).

trocknet commented 9 years ago

hm,is there any workround can give me the timeout what i need? reduce the queue size? if one segment take a long time to download, it will stop and delay the whole stream...

trocknet commented 9 years ago

requests has 3 timouts: ConnectTimeout, ReadTimeout, Timeout.they may help?

chrippa commented 9 years ago

I don't think making the timeout apply to the whole response download will help and will only introduce more issues. For example, if a segment has reached 90% completion, but the timeout is reached then we would have to start downloading that segment again which would probably only slow things down more.

The reality is that Afreeca does not have very good bandwidth to countries outside South Korea and I don't think we can overcome this entirely through software.