Feedback

[ryancsh]

Hello swent,

This is in response to the comment asking if I would like to try out this tool. I have only downloaded a few twitch VoDs so far and here are my first impressions.

I really like:

• The progress bar. Downloading 3-4 hour VoDs takes a while with my connection and being able to estimate how long is left is really useful.
• The auto-generated name. Uses the twitch title which is really convenient.

Other than that, I ran into a bug where the VoD got stuck downloading at about 72%. I killed it, launched a new download using the same URL and it worked fine. Maybe the first time my network connection dropped? I don't know, just writing here to let you know this did happen, but I can't reproduce it.

A few QoL improvements I that think would be nice:

1) Ability to parse out extra stuff from the URL (for example, https://www.twitch.tv/videos/1203465185?filter=archives&sort=time does not work, but removing the part after "?filter" does https://www.twitch.tv/videos/1203465185).

2) It would be useful to have access to the speed limiter in interactive mode
(that's the mode I think most users would use).

3) Ability to list some VoD URLs in a file and download all of them one after the other.
This would be useful since I tend to download in batches when nobody is home / using the network.

4) Potentially ability to pause in the middle of a download. (or after finishing the current segment).
Just a thought, not sure if this is actually valuable.

5) Potentially the ability to self-regulate download speed using some metric? The limiter is usually good enough, but if the ISP doesn't guarantee minimum speeds (they don't in my country), it can be hard to pick a good speed limit.
Instead, we could have the tool download a segment as fast as it can, and then pause for the same amount of time it took to download that segment.
Using more concrete numbers, let's say the tool requests some metric percent_pause_between_segments, ranging from 0 to infinity, where 0 means no pause (let's suppose 50% in this example). While downloading a segment, the downloader measures the time it took for that segment, let's say t_last = 30 seconds. After finishing, the downloader pauses for percent_pause_between_segments / 100 * t_last. In this case, that would be 0.50 * 30 seconds = 15 seconds, before starting the next segment.
While this does leave bandwidth unused, it does make sure that other applications don't get bandwidth starved without needing the user to input a hard speed limit which won't adjust to network conditions.

Anyway, these are my thoughts after trying it out so far. I think it's pretty good already, and some of those QoL improvements would make it even more convenient to use.

Cheers.

[swent]

Hey,

thanks for the feedback, its very aprpeciated! Do you know low long your download has been running before encountering the "freeze"? It is very possible that my code is not yet fully error-free so I'd like to narrow down on the problem. Would you be interested in an estimated time beind displayed?

About your QoL suggestions:

1. Yes agreed, will look into it
2. I thought about that but in the end there will be even more options to adjust the downloader behaviour, asking users to ALWAYS enter them all the time will also lead to bad user experience. I think just giving very common options in interactive mode vs having all the tiny options being available as cli arguments will lead to a solid experience. Its easy to use but if you need specific features, you can get them via CLI. Don't you think so?
3. Can you please explain? You mean when the downloader splits them off into multiple parts?
4. Pause can be done, maybe just use a specific key while in the download screen? Will implement I think.
5. This is a much more complicated feature than it seems. Most/all of the public big downloaders out there do not have such a feature - for good reasons. "Getting it right" is a nightmare and even having the procedure you explained nailed down might not lead to the effect you are hoping for. Let me think about it, maybe I can come up with something.

On a sidenote: I tried to generate an m3u8 file that can be used in players like VLC to watch VODs but it seems this doesn't work. Do you know if there are other tools that successfully offer such a feature?

Thanks for all the testing! If you find more problems, information on how to reproduce and maybe the vod link is appreciated :)

Cheers.

[ryancsh]

Hello again,

I ran into another freeze. Not sure if it is the same bug. This time the downloading kept progressing forward, but the Written | X / 529 line got stuck at about halfway through (around 300 something). The line Downloaded | X / 529 kept progressing until 528, despite the written segment number not advancing. Eventually I killed it and retried with the same link. Worked after.

The annoying part is that it's not consistently reproducible. Doubt it's a problem with the link. I know this is not enough information but I'm letting you know for now; maybe you could guess where the issue might be. If I figure out how to consistently make the bug happen, I'll let you know.

About the suggestions:

2. I see what you mean, too many interactive prompts get annoying. I don't mind passing in command-line arguments but, right now, to get the list of available options, I'm looking at the github repository, which is somewhat not user friendly. It would be nice to have a command line argument that prints out all available options, for example twitch-unjail -h or something.

3. No, I meant downloading different VoDs with a single command.
   Let's say I have 3 separate VoDs that I'd like to download. For example, let's say I have a "vods.txt" that contains 3 URLs:

   https://www.twitch.tv/videos/1206008671
   https://www.twitch.tv/videos/1203936617
   https://www.twitch.tv/videos/1203465185

   It would be nice if twitch-unjail could read the URLs from "vods.txt" and download all of them, one at a time, skipping the URLs that don't work for whatever reason. Batch downloading essentially. The command would look something like twitch-unjail -file "vods.txt".

4. Yeah, I do realize it's complicated, but the idea is to have the tool use a somewhat conservative speed limit that adjusts on the fly to network conditions. The main reason I'd like something similar is because I could then set twitch-unjail up to download a VoD (or multiple) and walk away, knowing that it won't hog all bandwidth and slow regular web browsing (or even regular downloads) to a crawl for others at home.
   In my case, the VoD is not a super high priority, as long as it eventually gets done.
   If you can find a better way to achieve the same effect, I'm all for it.

Not sure about m3u8 files. I tried that with twitch-recover from the other repo and couldn't play it with VLC either. Always downloaded as .mp4 afterwards. Maybe try looking into FFmpeg if you really want to use m3u8 files? FFmpeg tends to be good at converting to and from all sorts of weird formats, but figuring out which command-line arguments to use if you want to do something complicated is a pain.

I'll let you know if i run into other issues or if I figure out how to reproduce the freezing bug consistently.

Cheers.

[swent]

Hey,

okay, I will try to guess the most likely ways it might fail to write and add additional handling or maybe just printing additional data in case of odd behaviour, that might help us find the problem in the future. How long would a download with ~530 parts usually take for you? Whats the average speed you are loading at? Using the -mbps option or without?

2. Gotcha, will definitly implement something like that
3. Okay, will add an option to point it to a batch-download, but this might take a while since currently all the internal data is structured to be used one time and then dismissed (no re-use for multiple downloads intended). Its not a hard task, it just takes a little to switch things around and make sure everything is working properly.

Yeah that's the same I was seeing with m3u8 files, I recently added an option to the downloader to load an unmuted m3u8 but it's not able to play anywhere so I don't know if I messed the file up somehow or this might just not work on "regular" players. Will probably drop that option again.

Cheers.

[swent]

All tasks:

1. Option to create a log file to track down problems/freezes when downloading
2. Better url parsing to ignore extra params in urls
3. Extr cli --help argument to show all available options
4. Option to feed a filepath into the app instead of a vod url, file may contain multiple urls to be downloaded one after another
5. Pause option while downloading
6. Potentially variable and self-adjusting speed limit

[swent]

Hey,

could you check out the latest release? I added most of the discussed features and changes. I was sadly unable to track down your freeze problem but I added a --log option.

Just be warned, the log file might get pretty big (up to 100+ MB of size). I would still ask you kindly if you could enable it for some of your downloads or even do some extra downloads with logging enabled. If you encounter the freeze while logging is active, I think I have a good chance of finding the problem.

Full disclosure: the log will contain the save path and parts of the vod url.

For the --file option (batch download) you can use a regular multiline file where each line is a separate url or use the json file format.

I also changed a lot of other code around, so there might be new/other bugs, just let me know if you encounter some.

Cheers.

[ryancsh]

Hello again,

About my download speeds:

If nobody is using the network at all, I get about 2 megabytes/s with no speed limiter. But if I try to use all bandwidth available, loading webpages takes 10+ seconds. I think the issue is that my ISP routers allocate bandwidth per user account instead of per TCP connection. This happens anytime I try to download large files.

As a side note, the latency is nowhere as severe with servers that use LEDBAT-alike algorithms for congestion. For example, it doesn't happen with torrents when I completely disable TCP and use only μTP, even if I'm making full use of the bandwidth capacity. Lately, I've noticed more regular servers using LEDBAT-alike algorithms for traffic collision avoidance. I can tell because when I try to load a webpage, the download speed immediately drops and recovers after I finish loading the webpage. The latency does still increase, but it's tolerable; about 1 second instead of 10+ (My regular latency is about 0.3 - 0.5 seconds).

Because of the intolerable latency, I generally limit twitch-unjail with -mbps 1 even if I'm the only one using the network. If other people at home are using the network, I limit it even further with -mbps 0.5 to avoid being too much of a nuisance.

---

Just downloaded the RC4. For the -h option, I think it would be good if the options are separated into two groups: required vs optional. Not a huge problem but it's a nice QoL improvement. For example, -n NAME is optional in that the user doesn't have to type it in if they don't care about a custom name for the file.
Writing down the default behaviour of optional arguments would also be nice for new users. What I mean is something like the following:

Required arguments:
  --url URL             Url to download

Optional arguments:
  --name / -n NAME      Custom save file name. Defaults the twitch VoD title.

Other than that, I'll download some more VoDs later and I'll let you know if I find anything.

Cheers.

[swent]

Hey,

okay, still looking into congestion control, not an area I know much about . Currently you will notice that you can enable/disable and even increase/decrease the static speed limit on the fly, be warned tho, on low speeds it is better to start the app with --mbps option since that will tune the threads for lower speeds, it should not hurt performance to just activate it on the fly but it might result in a more spiky write speed reportings.

About the --help: yeah that's the plan, it's just a quick first version ;) afterall the readme text here on github is also split into required vs optional.