search

sealedtx / java-youtube-downloader

Simple, almost zero-dependency java parser for retrieving youtube video metadata
Other
432 stars 120 forks source link

Requesting video info is broken #130

Open iexavl opened 2 months ago

iexavl commented 2 months ago

The body in parseVideoAndroid function is broken. It gets a faulty response from youtube with no streamData. I checked pytube and replaced it with this:

        String body =
                "{" +
                "  \"videoId\": \"" + videoId + "\"," +
                "  \"context\": {" +
                "    \"client\": {" +
                "      \"hl\": \"en\"," +
                "      \"gl\": \"US\"," +
                "      \"clientName\": \"ANDROID_MUSIC\"," +
                "      \"clientVersion\": \"5.16.51\"," +
                "      \"androidSdkVersion\": 30" +
                "    }" +
                "  }" +
                "}";

With that it works.

iexavl commented 2 months ago

Never mind. I thought it worked, I was wrong. Also I just noticed that pytube also doesn't work for me. Is this a problem of mine or is it broken for everyone?

iexavl commented 2 months ago

Okay it actually appears to be broken for one particular video? The playability status of that video is 'This video is not available', but it is available in youtube. If I go to the video through the browser it works.

sealedtx commented 2 months ago

@iexavl it may require age verification or smth like that, try it in browser without login (incognito tab)

iexavl commented 2 months ago

@iexavl it may require age verification or smth like that, try it in browser without login (incognito tab)

Not age restricted. It's also not only that video, but it's also not every video. It's kind of weird. Originally it was this video: https://www.youtube.com/watch?v=ZxnD5YrMIgI, but then I found a couple more that don't work

punkydie commented 2 months ago

yes the downloader no longer works ): videoInfo is allways null.

iexavl commented 2 months ago

yes the downloader no longer works ): videoInfo is allways null.

It can work by deleting the part that uses innertube, in other words commenting out parseVideoAndroid, but that makes things quite slow since the downloader has to parse the html.

iexavl commented 2 months ago

https://github.com/iv-org/invidious/blob/a021b93063f3956fc9bb3cce0fb56ea252422738/src/invidious/yt_backend/youtube_api.cr#L483 I found this and tried it. The only thing I am missing is serviceIntegrityDimensions, because from what I can tell it's generated. Maybe I will try to add it tomorrow. The error did change a bit, but it's still an error.

sealedtx commented 2 months ago

@iexavl have a look at https://github.com/ytdl-org/youtube-dl there might be a recent fix for this that we can implement here

wiojosjosd commented 2 months ago

@sealedtx can u fix that now?hope soon.

iexavl commented 2 months ago

@iexavl have a look at https://github.com/ytdl-org/youtube-dl there might be a recent fix for this that we can implement here

I looked it over and tried what I could find. Unless I missed something there hasn't been a fix yet. This seems to be something specific to the android client, probably some additional argument. I tried the IOS client and it works:

String body =
                "{" +
                "  "videoId": "" + videoId + ""," +
                "  "context": {" +
                "    "client": {" +
                "      "hl": "en"," +
                "      "gl": "US"," +
                "      "clientName": "IOS"," +
                "      "clientVersion": "17.36.4"," +
               // "      "androidSdkVersion": 30" +
                "    }" +
                "  }" +
                "}";

Unfortunately it seems to be quite constrained for qualities. On 1080p 60fps video I got 720p 30fps max. I find it slightly odd that I can't see any recent issues on the github page for youtube-dl or any other projects that use the googleapis.

punkydie commented 2 months ago

but with this i dont have sound

punkydie commented 2 months ago

bestVideoWithAudioFormat returns zero with the IOS client. I think we need a better solution here.

punkydie commented 2 months ago

String body = "{" + " \"videoId\": \"" + videoId + "\"," + " \"context\": {" + " \"client\": {" + " \"hl\": \"en\"," + " \"gl\": \"US\"," + " \"clientName\": \"ANDROID_MUSIC\"," + " \"clientVersion\": \"5.16.51\"," + " \"androidSdkVersion\": 30" + " }" + " }" + "}"; with this it works

iexavl commented 2 months ago

String body = "{" + " \"videoId\": \"" + videoId + "\"," + " \"context\": {" + " \"client\": {" + " \"hl\": \"en\"," + " \"gl\": \"US\"," + " \"clientName\": \"ANDROID_MUSIC\"," + " \"clientVersion\": \"5.16.51\"," + " \"androidSdkVersion\": 30" + " }" + " }" + "}";

with this it works

It doesn't work for everything. It works for some videos, for others not. And the IOS is indeed a bad option. I have no idea what they did though and I am running out of ideas as to how to find it. Perhaps we should wait for the youtube-dl guys to notice or just post an issue there.

punkydie commented 2 months ago

and yes you're right unfortunately it doesn't work with every video ):

punkydie commented 2 months ago 
    String body =
            "{" +
            "  \"videoId\": \"" + videoId + "\"," +
            "  \"context\": {" +
            "    \"client\": {" +
            "      \"hl\": \"en\"," +
            "      \"gl\": \"US\"," +
            "      \"clientName\": \"ANDROID_CREATOR\"," +
            "      \"clientVersion\": \"22.36.102\"," +
            "      \"androidSdkVersion\": 30 " +
            "    }" +
            "  }" +
            "}";

this works very good and fast and with all videos too that doesnt worked with ANDROID_MUSIC

punkydie commented 2 months ago

ANDROID_TV 2.19.1.303051424

can use up to 1080p 60FPS and works too

sealedtx commented 2 months ago

@punkydie can you play a bit with these params and find best one, which works with most of videos, it should cover most of Itags

Then I'll update code or you can submit a pull request if you'd like to be a contributor. Thank you

punkydie commented 2 months ago

It might not be bad if the downloader checks all clients in a loop from good to bad until the video works. For example, ANDROID_VR also works, which supports up to 2160p and 60FPS. But not for every video. But it would be a shame to do without it.

iexavl commented 2 months ago

https://github.com/zerodytrash/YouTube-Internal-Clients?tab=readme-ov-file#clients Check this out. ANDROID_TV indeed seems to be 1080p60. It should do the job at least until the ANDROID client works. I tested it on the previous videos that were broken and it worked.

iexavl commented 2 months ago

@punkydie can you play a bit with these params and find best one, which works with most of videos, it should cover most of Itags

Then I'll update code or you can submit a pull request if you'd like to be a contributor. Thank you

Nevermind. Just use the web client. Here :

        String body =
                "{" +
                "  \"videoId\": \"" + videoId + "\"," +
                "  \"context\": {" +
                "    \"client\": {" +
                "      \"hl\": \"en\"," +
                "      \"gl\": \"US\"," +
                "      \"clientName\": \"WEB\"," +
                "      \"clientVersion\": \"2.20220918\"," +
                "    }" +
                "  }" +
                "}";

It works and it's 4320p60 HDR, same as android.

punkydie commented 2 months ago

nope doesnt work on me ANDROID_TV works best with all Videos that i have tested and very fast

iexavl commented 2 months ago

nope doesnt work on me ANDROID_TV works best with all Videos that i have tested and very fast

Seriously? I guess the android clients are the most sensible. Although I suppose we can take a look at the other clients. Can you check out the other clients in the list and see if they work for you?

punkydie commented 2 months ago

Yes, just test it. Works perfectly. Haven't found a video yet that doesn't work.

iexavl commented 2 months ago

Yes, just test it. Works perfectly. Haven't found a video yet that doesn't work.

can you try MWEB (2.20220918) , TVHTML5 (7.20220918) , TVHTML5_CAST (1.1), WEB_EMBEDDED_PLAYER (9.20220918), WEB_CREATOR (1.20220918) for now?

punkydie commented 2 months ago

yes i can

punkydie commented 2 months ago

The WEB Clients doesnt work. and TVHTML Clients not on all Videos.

iexavl commented 2 months ago

The WEB Clients doesnt work. and TVHTML Clients not on all Videos.

this is some really odd stuff. I guess we will be walking on eggshells with these clients until someone finds out what's going on. This also means that my assumption that the problem was only in android clients was wrong.

punkydie commented 2 months ago

Yes, I build into my mod (Minecraft) that I can change the clients if necessary without always having to do that in the code.

iexavl commented 2 months ago

Maybe it would be better to have multiple clients and have the requests be retried in case one client fails to return the streamingData @sealedtx

punkydie commented 2 months ago

Well, that wouldn't actually be a good idea. Because e.g. ANDROID_MUSIC also works, but it is very slow compared to the other formats. This means that if the downloader stops here because the video has been loaded, you have a slow, sometimes stuttering playback.

I can only emphasise once again that ANDROID_TV works perfectly. I have now tested over 50 different videos and for me (Minecraft) 1080p is enough.

iexavl commented 2 months ago

Well, that wouldn't actually be a good idea. Because e.g. ANDROID_MUSIC also works, but it is very slow compared to the other formats. This means that if the downloader stops here because the video has been loaded, you have a slow, sometimes stuttering playback.

We can order the clients by how fast they are or how much qualities they have and we can write it so that you can freely choose which client you want to use or the order of clients. That way if some client breaks, we don't need to update the entire codebase, but the person experiencing the problem can just programatically change which client they want to use.

punkydie commented 2 months ago

But what interests me. Isn't it a matter of time before Youtube finally switches off API V1? Will the downloader also be possible for API V3?

sealedtx commented 2 months ago

We can order the clients by how fast they are or how much qualities they have and we can write it so that you can freely choose which client you want to use or the order of clients. That way if some client breaks, we don't need to update the entire codebase, but the person experiencing the problem can just programatically change which client they want to use.

Would be great to have clear understanding, which client can provide more video formats (and cover more videos overall) and which client can provide faster download speed, mb build some enum or constants class or just leave a link to clients By default most reliable client should be used, and if user wants better quality, try to switch to another by specifying client in config

But what interests me. Isn't it a matter of time before Youtube finally switches off API V1? Will the downloader also be possible for API V3?

Not sure about v3, didn't follow updates for a while

iexavl commented 2 months ago

We can order the clients by how fast they are or how much qualities they have and we can write it so that you can freely choose which client you want to use or the order of clients. That way if some client breaks, we don't need to update the entire codebase, but the person experiencing the problem can just programatically change which client they want to use.

Would be great to have clear understanding, which client can provide more video formats (and cover more videos overall) and which client can provide faster download speed, mb build some enum or constants class or just leave a link to clients By default most reliable client should be used, and if user wants better quality, try to switch to another by specifying client in config

But what interests me. Isn't it a matter of time before Youtube finally switches off API V1? Will the downloader also be possible for API V3?

Not sure about v3, didn't follow updates for a while

well I did say that, but sorting by download speed and reliability might be a bit difficult, because they would be hard to test for. It won't be difficult to write a test for them, but they would require sending requests. If one does that too much at the very least they are gonna get one of the nasty "are you a bot" captchas every time they try to use a google service.

punkydie commented 2 months ago

lol Android TV no longer works. So that really gets on my nerves. I'm now going to build in the mod so that I can choose the client myself.

iexavl commented 2 months ago

lol Android TV no longer works. So that really gets on my nerves. I'm now going to build in the mod so that I can choose the client myself.

I wrote what I thought it would look like when I woke up today. If you got up to anything mind sharing your idea? This is roughly what my vision of it is: draft @sealedtx should I move along with this? (ignore the lack of immutability problems)

sealedtx commented 2 months ago

@iexavl looks fine, I've left suggestions

iexavl commented 2 months ago

@iexavl looks fine, I've left suggestions

I answered to them so you can check that out if you want. I also made a more full, working example here

(ignore the test class I forgot to delete) I added ClientTraits to Request, even though for now not all requests use innertube, if all do eventually it will come in handy. The way it's made right now if a request fails with one client you can just make another one with another client and re-send it. In the case of getVideoInfo that would be when a YoutubeException.BadPageException is thrown.

sealedtx commented 2 months ago

@iexavl you can create pull request when ready, I'll review

iexavl commented 2 months ago

@iexavl you can create pull request when ready, I'll review

Alright, the base functionality is done. After that it can be extended so clients can be sorted by quality formats and other stuff if possible. This reminds me, while I was testing the changes I noticed that the WEB and MWEB clients also broke. When I try to use them I get a bunch of "Error deciphering is required but no js url". That will probably also have to get looked at. I'm not sure if something I did broke them, but the other clients work fine in that regard. I have set ANDROID_TV as highest-priority client for now.

iexavl commented 1 month ago

Okay, ANDROID_TV also stopped working. It returns an HTTP 403. I am changing the highest priority to IOS... For now it's still working for me at least. We also have to perform some tests to refresh what the actual highest/lowest qualities for a given client is, because the information in the clients is likely outdated. It says IOS is max 720p, but I got 1080p.

pH-Valiu commented 1 month ago

Thank you guys for being so quick with a fix here. Really appreciating it :) Ran into the problem of no videoInfo being available as well...

iexavl commented 1 month ago

Thank you guys for being so quick with a fix here. Really appreciating it :) Ran into the problem of no videoInfo being available as well...

Well, unfortunately this isn't quite a fix. It's more of a temporary solution. The root of the problem is that we have no idea what youtube did. Maybe at some point when I get my new phone I can try to figure out what's going on with the clients.

punkydie commented 1 month ago

Somehow it's not like before. Sometimes I get “playabilityStatus={reason=Please sign in, status=LOGIN_REQUIRED}}” for videos in the playerresponse and the videos do not load. Sometimes it doesn't work sometimes. No matter which client.

punkydie commented 1 month ago

ok i have built a recursivecall function that only returns videoinfo when it is no longer null. works fine.

iexavl commented 1 month ago

ok i have built a recursivecall function that only returns videoinfo when it is no longer null. works fine.

Well that was the original idea anyway. You should also check for !videoinfo.formats().isEmpty() , because sometimes a response is returned, but without any formats.

sealedtx commented 1 month ago

@iexavl https://github.com/sealedtx/java-youtube-downloader/commit/e4235e913e921d1c684dd950d8c82085558d2603 have a look at this change I mentioned previously, I'm going to release new library version with your changes, just verifying if you ok with this

iexavl commented 1 month ago

@iexavl e4235e9 have a look at this change I mentioned previously, I'm going to release new library version with your changes, just verifying if you ok with this

To be honest I still think the try catch should be there. Because right now, when an exception occurs in the retry when we catch InvalidJsUrl if,for some reason, we can't extract it and an exception happens the exception will be thrown , but the onError callback will just never fire and I don't think that's the expected behavior. The problem is the catch below that just won't catch it, because this piece of code isn't in the try block. If you want test it yourself, replace the code in extractJsUrlFromConfig with just an exception throw and see if a callback fires.

sealedtx commented 1 month ago

@iexavl oh, I got it now, will bring it back, thanks