sealedtx / java-youtube-downloader

Simple, almost zero-dependency java parser for retrieving youtube video metadata
Other
413 stars 115 forks source link

download video very slow #97

Closed hk200 closed 2 years ago

hk200 commented 2 years ago

Hi when I did a test using download api that use the test url "https://www.youtube.com/watch?v=SbLMmKs3gzw" found very slow( about 40~50K/s) at multiple test location (the network max speed is very high), how can I fix this issue?

best regards

hk200 commented 2 years ago

test code appended VideoTest .java.txt

sealedtx commented 2 years ago

@hk200 hi, I looked through the code and your example and I can't see clearly what might cause such slow download speed. Investigation is required, you may start from looking at headers which are sent by browser and by youtube-downloader for same Format.url()

sealedtx commented 2 years ago

@hk200 can you also check other formats, it might be problem only with DASH formats

hk200 commented 2 years ago

ok,many thanks, appreciated.I will check that. The speed of download was very high before and everything is ok, but something strange these days, every test point at different location is just 50~60k max download speed by now. I am very confused by this condition. I checked the code, network and hardware and seems found no odd by now. did you run this code for test and get the high speed result by your side? have been struggling this issue for several days.

best regards,

hk200 commented 2 years ago

@hk200 can you also check other formats, it might be problem only with DASH formats

ok, I check for this, many thanks.

xibr commented 2 years ago

There is a similar problem on youtube-dl, maybe the same reason https://github.com/ytdl-org/youtube-dl/issues/29326

hk200 commented 2 years ago

There is a similar problem on youtube-dl, maybe the same reason ytdl-org/youtube-dl#29326

ok, many thanks , I check the refer issue together. best regards,

hk200 commented 2 years ago

There is a similar problem on youtube-dl, maybe the same reason ytdl-org/youtube-dl#29326

I checked the refer issue and some of my testing points are standalone servers at US and UK. all download speed at these servers are about 50~60K. I am not sure wether it is the code problem or the source video problem, still in handling.

hk200 commented 2 years ago

@hk200 hi, I looked through the code and your example and I can't see clearly what might cause such slow download speed. Investigation is required, you may start from looking at headers which are sent by browser and by youtube-downloader for same Format.url()

what is DASH format? and how can fix this format change to others at api?

sealedtx commented 2 years ago

@hk200 look at itags and comments: https://github.com/sealedtx/java-youtube-downloader/blob/70aeb61f1fe6fbc28a2dfd36f17ec6e2de46d21c/src/main/java/com/github/kiulian/downloader/model/videos/formats/Itag.java

hk200 commented 2 years ago

@sealedtx I check the code, many thanks.

@hk200 look at itags and comments: https://github.com/sealedtx/java-youtube-downloader/blob/70aeb61f1fe6fbc28a2dfd36f17ec6e2de46d21c/src/main/java/com/github/kiulian/downloader/model/videos/formats/Itag.java

hk200 commented 2 years ago

we change and try all formats of testing video at different locations and cannot fix this issue. @sealedtx need further help best regards, hk

sealedtx commented 2 years ago

@hk200 what about issue in youtube-dl? is there any solution? it was mentioned that fork yt-dlp has implemented workaround, can you check it? if you can find it I may try implement it here

hk200 commented 2 years ago

@hk200 what about issue in youtube-dl? is there any solution? it was mentioned that fork yt-dlp has implemented workaround, can you check it? if you can find it I may try implement it here

@sealedtx current issue I met is download video is very slow and I am just using java-based java-youtube-downloader. I am not y2-dl project user. someone metioned the similar y2 download throttling issue at other project has been fixed by one go-language based project. https://github.com/89z/mech but I am not go-language user, just use java language.

best regards, hk

sealedtx commented 2 years ago

@hk200 Here might be one of workarounds from yt-dlp https://github.com/yt-dlp/yt-dlp/commit/51d9739f8031fb37d8e25b0e9f1abea561e3d2e3 seems like just re-extracting video if speed goes below threshold

Also might be usefull:

We know the cause for throttling and two workarounds. There is nothing else that can be said in this thread by anybody but the original maintainer coming back after 4 months and copying one of the solutions:

  • spoof android
  • correctly calculate &n and modify the way we sent range requests Fix throttled streaming by using a new route anxdpanic/plugin.video.youtube#215 (comment)

first one spoof android - as I understand is already implemented in yt-dlp: --extractor-args youtube:player_client=android second one is not implemented yet

need research about how to spoof android and use android client for extractions

hk200 commented 2 years ago

this project is used by pure java client. I try to check the hints that you mentioned, thx. Is that possible that this feature will added to our project to handle this issue? @sealedtx best regards, hk

sealedtx commented 2 years ago

@hk200 to be honest, I have no time to work on this project now, but this issue looks like critical and I might spend some time next week looking for a solution

hk200 commented 2 years ago

many thanks, we are blocked by this issue for many days. @sealedtx

ok

best regards, hk

sealedtx commented 2 years ago

also may be useful for android spoofing https://github.com/TeamNewPipe/NewPipeExtractor/issues/562

I cheked this endpoint, but I got

<div style="margin-left: 4em;">
            <h1>We're sorry...</h1>
            <p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p>
</div>

Seems like Google protects this endpoint from being scrapped

sealedtx commented 2 years ago

@hk200 checkout 3.0.2 release, I have implemented spoofing android client https://github.com/sealedtx/java-youtube-downloader/commit/b84d42f931c0f34a748567dadb0682b23b44c14b

Resulting extractions seems to have higher download speed than web client

hk200 commented 2 years ago

ok, many thanks. I'll check it out. Can I use it in the pure java client in linux environment instead of android system to fix the throttling issue? @sealedtx

best regards, hk

sealedtx commented 2 years ago

@hk200

in linux environment instead of android system

sure, it is just request metadata to spoof that client is android

hk200 commented 2 years ago

ok, many thanks for support, I'll check it out and have a full test of it. @sealedtx

best regards, hk

sealedtx commented 2 years ago

@hk200 did you check it?

xibr commented 2 years ago

I tested with iTag 22 and got full speed.

hk200 commented 2 years ago

@hk200 did you check it?

I make a fully test of this issue and the update passes the test. @sealedtx

best reagrds,

hk