Closed triplesixman closed 2 years ago
Dont know if it helps, but since downloading with youtube-dl didnt work anymore (because of the speed limit), i started 2 days ago using clipgrab that i always used on windows.
I downloaded the whole rambalac channel with clipgrab 3.9.6, hundreds of high quality 4k-8k videos, over 1,2TB. And had no single speed drop down. Non stop 110MB/s (1gig fiber).
And thats wonder me. Because if i try the same with youtube-dl, i get speed limited.
And what wonder me the most, is that clipgrab use youtube-dl for downlaoding. 2021-06-06 to be exact.
Maybe it helps you, if you guys conntact the guy from clipgrab why it is working with his tool? I know him from linuxforen.de, because 10-12 years ago, he was/is a user from the forum, wich presented this tool and asked for help and so on.
I believe I'm successfully calculating the updated value of n
in my experimental branch for pytube if anybody wants to reference the code. However, this calculated value of n
is different from the value present on stream URLs with the ratebypass
parameter, which seems a little unusual. The calculated value of n
doesn't seem to lead to errors while downloading.
Furthermore, when I substituted the value of n
from the URL with the ratebypass
parameter into other URLs, there don't seem to be any errors while downloading.
I have not yet tested whether this solves the throttling problem. Any help with doing so is greatly appreciated.
I don't know if this can help anything, but ETA was from the start about the size of the video itself (~ one hour). "best" in this case is only 360p. youtube-dl 2021.06.06 - Xubuntu 20.04.2
rhfor@rhfor-PC:~/Downloads$ youtube-dl -f best https://youtu.be/Lg8ONTGiUOc --sub-lang pt-BR --sub-format srt
[youtube] Lg8ONTGiUOc: Downloading webpage
[youtube] Lg8ONTGiUOc: Downloading MPD manifest
[download] Destination: Cosmos com Carl Sagan - Episódio 01 - Shores of Cosmic Ocean-Lg8ONTGiUOc.mp4
[download] 68.8% of 165.35Mib at 43.61KiB/s ETA 20:11
PS: I downloaded the episode 2 in 10 seconds.
this youtube download throtteling has to stop. I bought a new hdd to watch my youtube offline and what happened? downloadspeed at 70 kbs!!! That takes decades. It is terrible. Everyone has the right to watch his videos offline. It is the only way to download playlists and more, automatically.
Youtube is also able to block youtube-dl completely. Youtube already did this during the music copyright thing. Didn't matter which youtube-dl version you used, it didn't work, blocked nothing worked. It can be done anytime by google.
this is so annoying don't you think?
i just tried downloading a video and this happened:
[youtube] Downloading login page
[youtube] Looking up account info
WARNING: Unable to look up account info: HTTP Error 400: Bad Request
[youtube] P9vmxl4sXQM: Downloading webpage
[youtube] P9vmxl4sXQM: Downloading MPD manifest
[download] Destination: /Users/[ommited]/Data/downloads/YouTube/NA _NA/NA- Shinigami Eyes By Toddsson _P9vmxl4sXQM.f137.mp4
[download] 34.0% of 38.14MiB at 20.11KiB/s ETA 21:20
this is getting worse and worse
I believe I'm successfully calculating the updated value of
n
in ~my experimental branch for~ pytube if anybody wants to reference the code. However, this calculated value ofn
is different from the value present on stream URLs with theratebypass
parameter, which seems a little unusual. The calculated value ofn
doesn't seem to lead to errors while downloading.Furthermore, when I substituted the value of
n
from the URL with theratebypass
parameter into other URLs, there don't seem to be any errors while downloading.~I have not yet tested whether this solves the throttling problem. Any help with doing so is greatly appreciated.~
A pytube user tested the changes I made to calculate n
based on the js in the base.js
file, and did not see a slowdown in 100+ videos. They ran into other errors, but I believe those errors are unrelated to the new code for ciphering n
. It looks like there are ~9-10 unique function calls in the javascript for computing n, but most of them are pretty simple.
The weird part was modifying the list that was passed as a reference to the function calls to get it to calculate correctly. Effectively, the way that js code works is as follows:
n
into an array of characters.n
, and null
(calling this array c
for clarity).null
in c
with references to c
itself.c
with arguments of different elements of c
(notably, c
itself gets left- and right-shifted by some of these functions. This is why the references to c
are placed in the array). These calls are all made sequentially within the try
section of the try-catch
block.c
has references to the array made from n
, the actual array of n
gets changed and can be un-split into a new string value of n
.I don't have the time to figure out how to incorporate this into youtube-dl
myself, but maybe somebody else can use this info to write a fix.
A pytube user tested the changes I made to calculate
n
based on the js in thebase.js
file, and did not see a slowdown in 100+ videos. They ran into other errors, but I believe those errors are unrelated to the new code for cipheringn
. It looks like there are ~9-10 unique function calls in the javascript for computing n, but most of them are pretty simple.The weird part was modifying the list that was passed as a reference to the function calls to get it to calculate correctly. Effectively, the way that js code works is as follows:
1. Split `n` into an array of characters. 2. Create an array of integers, strings, anonymous functions, references to `n`, and `null` (calling this array `c` for clarity). 3. Replace all instances of `null` in `c` with references to `c` itself. 4. Call the functions in `c` with arguments of different elements of `c` (notably, `c` itself gets left- and right-shifted by some of these functions. This is why the references to `c` are placed in the array). These calls are all made sequentially within the `try` section of the `try-catch` block. 5. Because `c` has references to the array made from `n`, the actual array of `n` gets changed and can be un-split into a new string value of `n`.
I don't have the time to figure out how to incorporate this into
youtube-dl
myself, but maybe somebody else can use this info to write a fix.
Thank you SOO much @tfdahlin for your efforts. I'm gonna try my hand at this. Glad that it can be implemented in Python. But does your code have any dependencies on packages which are not part of Youtube-DL? Anything for decrypting the Javascript for example?
But does your code have any dependencies on packages which are not part of Youtube-DL?
pytube is dependency-free.
But does your code have any dependencies on packages which are not part of Youtube-DL?
danny-wu is correct, pytube has no package dependencies.
Anything for decrypting the Javascript for example?
A while back I wrote a parser to replace the regex that was being used to identify the bounds of javascript objects, and was able to use that for parsing the different pieces of the javascript. From there, I use regexes to identify individual functions within the arrays, and map them to the relevant ciphering function
Hi, all. I solved this issue by using https://github.com/iTaybb/pySmartDL My example code:
import time
from pySmartDL import SmartDL
from urllib3 import PoolManager
from os.path import exists
LINK = "ANY_LINK"
SAVE_AS = "ANY_FILENAME"
try:
pool = PoolManager()
response = pool.request(
"GET", LINK, preload_content=False)
file_size = int(response.headers.get(
"Content-Length"))
# `4009252` is optimal for 100Mbit/s
threadsVideo = int(file_size / 4009252)
except Exception as e:
print("Exception", e)
threadsVideo = 50
print("threadsVideo", threadsVideo)
obj = SmartDL(LINK, SAVE_AS,
threads=threadsVideo)
obj.start(blocking=False)
while not obj.isFinished():
# Doing something ...
# print("Speed: %s" % obj.get_speed(human=True))
# print("Already downloaded: %s" % obj.get_dl_size(human=True))
# print("Eta: %s" % obj.get_eta(human=True))
# print("Progress: %d%%" % (obj.get_progress()*100))
# print("Progress bar: %s" % obj.get_progress_bar())
# print("Status: %s" % obj.get_status())
# print("\n"*2+"="*50+"\n"*2)
time.sleep(0.5)
if obj.isSuccessful():
print("downloaded file to '%s'" % obj.get_dest())
print("download task took %ss" % obj.get_dl_time(human=True))
print("File hashes:")
print(" * MD5: %s" % obj.get_data_hash('md5'))
print(" * SHA1: %s" % obj.get_data_hash('sha1'))
print(" * SHA256: %s" % obj.get_data_hash('sha256'))
else:
print("There were some errors:")
for e in obj.get_errors():
print(str(e))
secondsBeforeTempVideoSaved = 0
while secondsBeforeTempVideoSaved < 10:
if not exists(SAVE_AS):
print("Video not saved, waiting 500ms ...")
time.sleep(0.5)
secondsBeforeTempVideoSaved += 0.5
else:
break
print("Video downloaded")
can confirm it's still an issue
It went away for us for 2 weeks and is now back. Any other updates? Would be willing to put a bounty in crypto for this.
If somebody wants to fix this in the next couple of days (so we have time to archive unlisted videos before the 23rd), I'd also be willing to drop a bit into @tzarebczan's bounty pot.
try the workaround in yt-dlp. @tzarebczan @rebane2001
yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K
see https://github.com/yt-dlp/yt-dlp/releases/tag/2021.07.07
+1, no issues with yt-dlp's workaround. Archiving thousands of videos fine without throttle.
I am switching to dlp now, the additional features and active community development has won me over.
Just do the same. It's mostly compatible, was a drop in replacement for me.
On Wed, 14 Jul 2021, 3:08 am LE, @.***> wrote:
try the workaround in yt-dlp. @tzarebczan https://github.com/tzarebczan @rebane2001 https://github.com/rebane2001
yt-dlp "your video link here" --extractor-args youtube:player_client=android --throttled-rate 100K
see https://github.com/yt-dlp/yt-dlp/releases/tag/2021.07.07
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-879256177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CVVCF6E2LUZSNML7TKBDTXRXHNANCNFSM463R5FEQ .
-- https://www.canva.com/Empowering the world to design Share accurate information on COVID-19 and spread messages of support to your community. Here are some resources https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates that can help. https://twitter.com/canva https://facebook.com/canva https://au.linkedin.com/company/canva https://twitter.com/canva https://facebook.com/canva https://au.linkedin.com/company/canva https://instagram.com/canva
Glad to hear that @danny-wu . Any issues with formats or download failures otherwise?
I've also been in contact with the DLP dev on Discord, he's great. Sent him a nice tip via paypal last week for his support and continued efforts. Anyone using it should consider it also: https://ko-fi.com/pukkandan
It went away for us for 2 weeks and is now back. Any other updates? Would be willing to put a bounty in crypto for this.
That's the spirit!! A bounty would work wonders.
@tfdahlin appears to have solved the problem with his pytube project. I'm just waiting for someone to implement in youtube-dl. If you don't mind using another repo, then use pytube to download the actual video files. https://github.com/pytube/pytube/issues/1033
try the workaround in yt-dlp. @tzarebczan @rebane2001
yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K
see https://github.com/yt-dlp/yt-dlp/releases/tag/2021.07.07
I can vouch for this workaround as well. Switched to using it a few hours ago and already a mind-blowing difference. Downloading is so much more efficient now. Perfect!
try the workaround in yt-dlp. @tzarebczan @rebane2001
yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K
see https://github.com/yt-dlp/yt-dlp/releases/tag/2021.07.07I can vouch for this workaround as well. Switched to using it a few hours ago and already a mind-blowing difference. Downloading is so much more efficient now. Perfect!
Is there a reason why youtube-dl can't do the same thing they did to go around the throttling?
Is there a reason why youtube-dl can't do the same thing they did to go around the throttling?
Likely because it's not a proper solution. It's a workaround until someone implements decoding the n
parameter, which is the real problem.
Is there a reason why youtube-dl can't do the same thing they did to go around the throttling?
Likely because it's not a proper solution. It's a workaround until someone implements decoding the
n
parameter, which is the real problem.
A workaround that works is better than no solution at all imo, but I suppose anyone who thinks that way can just use the fork, it's a drop-in replacement after all.
Is there a reason why youtube-dl can't do the same thing they did to go around the throttling?
Likely because it's not a proper solution. It's a workaround until someone implements decoding the
n
parameter, which is the real problem.
I see, that makes sense
I have a 100 MBPS connection but the fix with y_dl is downloading at 1 MBPS.
yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K
If i change the rate to something like 500k it retries and dies.
The fix is amazing it is better than 76 KBPS rate for some random videos with youtube_dl. Is there any other way to increase this?
thanks!
yt-dlp "https://www.youtube.com/watch?v=exampleURL" --extractor-args youtube:player_client=android --throttled-rate 100K
yt-dlp != yt-dl.
@skwolvie report it here https://github.com/yt-dlp/yt-dlp/issues/new
I can confirm I am experiencing this same issue. I'm running Windows 10, and I keep the program updated. I use youtube-dl mainly through some automated batch-download Batch scripts I have written. The intermittently super-slow download speeds are a real pain.
Just use yt-dlp. My setup is more complicated but it was a drop in replacement for youtube-dl.
yt-dlp is a more actively maintained project at this point.
On Sun, 25 Jul 2021, 2:48 pm Obscure2020, @.***> wrote:
I can confirm I am experiencing this same issue, intermittently. I'm running Windows 10, and I keep the program updated. I use youtube-dl mainly through some automated batch-download Batch scripts I have written. The intermittently super-slow download speeds are a real pain.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-886146212, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CVVCGHXMVAWILAH7N5CLTZOJQXANCNFSM463R5FEQ .
-- https://www.canva.com/Empowering the world to design Share accurate information on COVID-19 and spread messages of support to your community. Here are some resources https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates that can help. https://twitter.com/canva https://facebook.com/canva https://au.linkedin.com/company/canva https://twitter.com/canva https://facebook.com/canva https://au.linkedin.com/company/canva https://instagram.com/canva
Don't worry if you have complicated batch scripts. Just switch "youtube-dl" to "yt-dlp" and keep everything else the same. It's been working flawlessly for me.
On Sun, 25 Jul 2021, 5:57 pm Danny Wu, @.***> wrote:
Just use yt-dlp. My setup is more complicated but it was a drop in replacement for youtube-dl.
yt-dlp is a more actively maintained project at this point.
On Sun, 25 Jul 2021, 2:48 pm Obscure2020, @.***> wrote:
I can confirm I am experiencing this same issue, intermittently. I'm running Windows 10, and I keep the program updated. I use youtube-dl mainly through some automated batch-download Batch scripts I have written. The intermittently super-slow download speeds are a real pain.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-886146212, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CVVCGHXMVAWILAH7N5CLTZOJQXANCNFSM463R5FEQ .
-- https://www.canva.com/Empowering the world to design Share accurate information on COVID-19 and spread messages of support to your community. Here are some resources https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates that can help. https://twitter.com/canva https://facebook.com/canva https://au.linkedin.com/company/canva https://twitter.com/canva https://facebook.com/canva https://au.linkedin.com/company/canva https://instagram.com/canva
I have this too and I find if I set youtube-dl to resume and restart the script it would regain the speed!
unfortunately development on this project seems to have stalled. yt-dlp project is aggressively addressing this throttling problem.
Does yt-dlp have the same -U command-line auto update feature?
Does yt-dlp have the same -U command-line auto update feature?
Yes.. Please read the documentation before asking questions.
to be frank i have noticed that all i have to do to stop the 50 kb downloads for a day or two is visit the youtube page and redownload the cookies from there. So it's probably a problem of how the cookies are updated in the cookie file.
unfortunately development on this project seems to have stalled. yt-dlp project is aggressively addressing this throttling problem.
Yes, but yt-dpl does not seem to support Python 2, so it seems I'm stuck with youtube_dl.. I manually added an exception to my projects, to go around this problem for now.
Yes, but yt-dpl does not seem to support Python 2, so it seems I'm stuck with youtube_dl
Sounds like you have much bigger problems if you have to be staying on Python 2.
Sounds like you have much bigger problems if you have to be staying on Python 2.
There are a lot of libraries that are not supported in Python 3 Also, big frameworks like Qt also stopped supporting various things that I need, in their updated versions.. Its a nightmare to support complicated apps while everybody changes compatibilities under your feet.. :angry:
OT: @noembryo Isn't virtenv a thing? https://python.land/virtual-environments/virtualenv
OT: @noembryo Isn't virtenv a thing? https://python.land/virtual-environments/virtualenv
Well, my problem is not how to install different versions of Python/libraries. The problem arises when an app that uses something from Qt 4 that does not exists in Qt 5/6 (e.g. like Phonon) also uses a library that suddenly stops supporting Python 2.. :skull:
.. but lets not hijack the thread anymore.. Thank you for your time. :+1:
Since there haven't been any takers, I thought to poke at it and write down what I found (while writing a simple transcoder for another scripting language and for myself - it already works, but it'd count as an external dependency). I will explain what Youtube's n token function does. This is not a patch and is only of interest to other devs if they wish to implement it themselves or improve the current interpreter. But if you wanted to know what useless job Google developers get paid for, continue reading!
Here's your keygen music while reading. Sort of.
Overview of the control flow (already known):
n
- your video access tokenn
based on internal logic. This logic is inside player...base.js
n
is modified by a cryptic functionn
is sent back to server as proof that we're an official client. If you send n
unmodified, the server will eventually throttle you.This cryptic function is currently named "lha" at line 1162 of the current player. My explanation is based on this player. The function code is randomly generated with each player release. It consists of:
n
as stringb = a.split("")
this is where n
will be modified letter by letterc
- the really long array that's supposed to confuse youn
that holds the key to Narnia. If anything goes wrong, an enhanced_except_
is returned along with original n
n
and the seed needed to generate the cryptic function: "enhanced_except_nJMB4-z-_w8_" + original_n
- the random base64 letters are the parameters used by the code generator to create this function. Seed = nJMB4-z-
(changed with each player) and w8
is probably the generator version (hadn't changed yet).The c
array contains 3 references to itself and 3 references to b
. The b's are assigned at the time of array creation, the c's are filled in where the placeholder null
was placed (indexes 3,4,9). Three different references to the same variable are supposed to make correlation harder? They didn't. Idk what's that useful for.
(4) Function calls look like this: c[39](c[3], c[22])
where:
b
or c
- to which the function will applyWhat are the long numbers for? Like 1222607021
and -1989164346
- these are indexes into either b/c
: simply something like 1222607021 % 52 = 25
aka c[25]
If the argument is a string, it is used for "translateA/B" functions (see below). Nothing is eval'ed
or anything (even if you see javascript keywords)
Here's a visual representation (for player 2840754e
, Aug 4/5th 2021) of what the shuffling of data looks like:
Arrows track objects of c
between shuffles. If all arrows are straight on this diagram, it means a shuffle on b
was performed instead (The above diagram describes this order: CCC, BBB, C, BB, C, BB, CC, B, CCCCC, B, CCCCC, BB, CCC, BBBBB
). There're only 3 kinds of shuffle operations: swap two elements, reverse entire array and move all elements by E positions.
Now it's time to explain what kind of shuffle functions there are.
for (e = (e % d.length + d.length) % d.length; e--;) d.unshift(d.pop())
d.splice(-e).reverse().forEach(function(f) { d.unshift(f) })
d.splice(e, 1)
d.splice(0, 1, d.splice(e, 1, d[0])[0])
for (var e = d.length; e;) d.push(d.splice(--e, 1)[0])
As you can see some functions are implemented TWICE with different code. It is beyond me why anyone would do this. I really hope it was a "puzzle task" for an intern and not a real attempt.
Alright, so you call some simple functions that shuffle both c
and b
, nothing complicated. While you could incrementally deduce and reduce the "randomized" index values inside the try-catch block without executing any code, the functions translateA and translateB require you to execute their code (either interpreter or reimplementation).
However you do NOT need to implement the switch-case statement. Whoever created this Rube-Goldberg machine was not as smart as they imagine. Let's take a look at the full function code for "translateA/B"
function(d, e) {
for (var f = 64, h = []; ++f - h.length - 32;) {
switch (f) {
case 91:
f = 44;
continue;
case 123:
f = 65;
break;
case 65:
f -= 18;
continue;
case 58:
f = 96;
continue;
case 46:
f = 95
}
h.push(String.fromCharCode(f))
}
d.forEach(function(l, m, n) {
this.push(n[m] = h[(h.indexOf(l) - h.indexOf(this[m]) + m - 32 + f--) % h.length])
}, e.split(""))
}
Here's the equivalent function with my comment:
function(d, e) {
// look, there's really no need for overengineering even if your IQ is below average
// and you can't write iterators to generate this, like I show below:
h = ["0", "1" ... "a", "b", ... "A", ... "Y", "Z", "-", "_"]; // shortened array for readability
d.forEach(function(l, m, n) {
this.push(n[m] = h[(h.indexOf(l) - h.indexOf(this[m]) + 64) % h.length])
}, e.split(""))
}
case 65:
in the switch then the array is base64 in the following order: 0aA-_
, otherwise it is Aa0-_
(A-Za-z0-9-_)m
increases while f--
decreases and -32
remains constant, the expression + m - 32 + f--
can be replaced with a constant +64
The translateA/B functions modify both N and themselves (this
) hence modifying the LUT hence requiring an interpreter, I do not think it is possible to reduce this one (unlike everything else).
At the end of all these, in essence, trivial transformations, you get the new N. That's all this long-winded function does. I hope you had fun reading <3
The problem is the reimplementation of youtube-dl's basic interpreter because its current approach is a dead-end: I tried in the other scripting language and it works... with ever more hacks applied. Trust me and start with a basic parser. Or use a library, that's what they're for ;)
PS: If you want to have fun with Google engineers, let their cryptic function deliberately "crash" by sending the "enhanced_except" message to their endpoint. They track these messages to react to bugs. Why not send them a message? :sunglasses: Here's the command to get the current "enhanced_except" marker:
curl -L "https://youtube.com/$(curl -L https://youtube.com/ | grep -oP 's/player/([^/]+)/player_ias\.vflset/[\w_]+/base\.js' | head -n 1)" | grep -oP 'enhanced_except[\w_-]+'
Then all you need to do is modify and repeat the /videoplayback?
call from F12 (Browser's Network tab): Replace value of &n with enhanced_except_..._YourMessage
(don't use special symbols)
byebye~
PPS: Bonus containing 81 base.js players - collection-of-81-youtube-players.zip
@OmniSexualCofeeBean Outstanding analysis! So I take it that Youtube-DL as it is currently designed cannot be modified/extended to accommodate this new change. What other package can one use, which Youtube-DL can import, to outsource this decryption algorithm to?
If you can come up with a complete working solution for us (maybe your own fork of youtube-dl?), @tzarebczan has put up a crypto bounty for anyone who cracks the problem. I'd be willing to add a small amount to this. I don't have much in my Bitcoin wallet. He didn't state an amount, but whatever it is, you can claim it if you can solve this problem for us. We would all be very grateful. Would it be worth your while?
@shoxie007 youtube-dl has JSInterpreter which is used in signature decryption. The same can be done for "n" parameter
@shoxie007 youtube-dl has JSInterpreter which is used in signature decryption. The same can be done for "n" parameter
If you read @OmniSexualCofeeBean 's analysis, the decryption happens at random DURING the streaming. The n value doesn't necessarily stay the same. Will JSInterpreter be able to handle this? I don't know if there is anything in youtube-dl that can intervene during a download and re-send a request with a different n value.
What other package can one use, which Youtube-DL can import, to outsource this decryption algorithm to?
@shoxie007 any JS interpreter whether NodeJS or anything smaller. Alternatively one of the available Python JS interpreters. The problem is youtube-dl's own stance against dependencies because I do not see any other reason to write an own interpreter (like this basic JSInterp, but Python 2 & 3 support too). Surely someone may decide to rewrite it from scratch to expand the needed functionality. Actually my entire analysis is irrelevant if someone decided to use an external interpreter. My only goal was to understand their (flawed) logic.
Just to reiterate: youtube-dlp uses a different API call that currently does not require n
manipulation.
decryption happens at random DURING the streaming. The n value doesn't necessarily stay the same.
No, it only happens once after getting the n
value from the server. I don't know what led you to this idea. The following only meant that the lha
function code (which is called to modify n
) is changed with each different base.js player.
The function code is randomly generated with each player release.
@OmniSexualCofeeBean Thanks for the clarification. I saw the arbitrary shuffling of data (CCC, BBB, C, BB, C, BB etc) and assumed that it would need a new n when this happens. A little confused. I am NOT adept with Javascript and the inner workings of streaming and media players. Hence the reason I haven't come up with a solution myself.
dude... 😅 omni clearly said the shuffling happens before...
He just gave us a visual representation of a "really long array the is supposed to confuse you" because he likes to humour us and show us some of the needless passages that this silly function does. That's not a "visual" of the "stream".
i mean, i find it quite funny too: several passages clearly show the entire array stays completely untouched. 😂 That's some high level "nothing" that wastes processor time.
In the end what we have is an original value, divided, that goes through some code which rearranges it around. And then gets sent back to the origin.
That is all.
FWIW, I published this innertube library yesterday. Its for nodejs, though. It does the n transform, url decipher & sapisid hashing (for logged-in accounts) by extracting the functions from the yt player. Over the past month I have tested it on 1000's of videos with no slowdowns. I know its not a fix for this project, but it may help some frustrated by this issue.
@gatecrasher777 Many thanks for your efforts and the share. How would I get Youtube-DL to use it? Give us a rough idea, if you please.
In the ReadMe you state "To get the most from ytcog, all you need is an active age-verified YouTube account." Can your solution work without logging in? Will it at least download non-age-restricted videos? I would NOT want to download 1000 videos or more with a logged in account. I wouldn't want high bandwidth use and multiple automated requests associated with my account, for fear of what can happen.
@shoxie007 - It works for most things without supplying a cookie. It recognizes that you are not logged in, though. Because of that it will likely give 403 on age-restricted video and it might start giving 429 if you use it a lot (although I have never seen a 429 on innertube requests, logged in or out). The test examples as they are don't use a cookie, and still work fine.
I really can't say how youtube-dl could use it. I'm planning to release a cli version in the future. But of course, it is only a youtube solution. youtube-dl does so much more.
After 2 months and 100 comments on this post, I switched to yt-dlp (https://github.com/yt-dlp/yt-dlp) a few weeks ago and I must say that everything works perfectly.
Quite surprising that the problem has been present for at least 4 months and is still ongoing, while youtube remains the most used platform by youtube-dl.
It's also surprising that we didn't get any comment from @dstftw .
Nevertheless, I don't close this post for those who are interested. Cya and thanks to the participants for their research work.
Checklist
Verbose log
While the download was in progress, I ran the exact same command in another terminal, another folder, and the download was completed in a few seconds:
Description
Hello,
Important: The problem is random, maybe 1 chance on 8 to produce it. You have to download several videos in a row (about 10) to notice it.
Since a few weeks, randomly a youtube video can be slowed down to 48 Kio/s, so it takes 5-10 minutes to download a short video of 5 minutes instead of 4, 5 seconds, often the download does not succeed and stops after a few minutes.
This happens on several servers, several internet providers as well as with my private connection.
I even managed to launch a second download in parallel when the first one was taking time, the second one finished in 1 seconds, the first one in 5 minutes and was interrupted before the end. Same video, same connection, same command. (tested only with ipv4 because I don't have ipv6 on my servers or with my internet connection).
Attached is an extraction of the results with --dump-pages. ko.txt ok.txt