403 error when calling GetManifestAsync on suicide-restricted videos

wasd52030 commented 5 months ago

Version

6.3.16

Platform

.NET 6.0.423/Windows 11

Steps to reproduce

using YoutubeExplode;
using YoutubeExplode.Videos.Streams;

var yt=new YoutubeClient();
var url="https://www.youtube.com/watch?v=4QXCPuwBz2E";

var videotManifest=await yt.Videos.Streams.GetManifestAsync(url);

Details

Expected Behavior: Returns valid StreamManifest Actual Behavior: HttpRequestException thrown

Complete error repor

System.Net.Http.HttpRequestException: Response status code does not indicate success: 403 (Forbidden).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at YoutubeExplode.Videos.Streams.StreamClient.TryGetContentLengthAsync(IStreamData streamData, String url, CancellationToken cancellationToken) in /_/YoutubeExplode/Videos/Streams/StreamClient.cs:line 63
   at YoutubeExplode.Videos.Streams.StreamClient.GetStreamInfosAsync(IEnumerable`1 streamDatas, CancellationToken cancellationToken)+MoveNext() in /_/YoutubeExplode/Videos/Streams/StreamClient.cs:line 114
   at YoutubeExplode.Videos.Streams.StreamClient.GetStreamInfosAsync(IEnumerable`1 streamDatas, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.ToListAsync[T](IAsyncEnumerable`1 source) in /_/YoutubeExplode/Utils/Extensions/AsyncCollectionExtensions.cs:line 49
   at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.ToListAsync[T](IAsyncEnumerable`1 source) in /_/YoutubeExplode/Utils/Extensions/AsyncCollectionExtensions.cs:line 49
   at YoutubeExplode.Videos.Streams.StreamClient.GetStreamInfosAsync(VideoId videoId, PlayerResponse playerResponse, CancellationToken cancellationToken) in /_/YoutubeExplode/Videos/Streams/StreamClient.cs:line 218
   at YoutubeExplode.Videos.Streams.StreamClient.GetStreamInfosAsync(VideoId videoId, CancellationToken cancellationToken) in /_/YoutubeExplode/Videos/Streams/StreamClient.cs:line 276
   at YoutubeExplode.Videos.Streams.StreamClient.GetManifestAsync(VideoId videoId, CancellationToken cancellationToken) in /_/YoutubeExplode/Videos/Streams/StreamClient.cs:line 292

Checklist

[X] I have looked through existing issues to make sure that this bug has not been reported before
[X] I have provided a descriptive title for this issue
[X] I have made sure that this bug is reproducible on the latest version of the package
[X] I have provided all the information needed to reproduce this bug as efficiently as possible
[ ] I have sponsored this project
[ ] I have not read any of the above and just checked all the boxes to submit the issue

wasd52030 commented 5 months ago

When I perform a large number of downloads at once, there is a higher probability of occurrences exceeding a certain threshold (greater than 50).

Tyrrrz commented 5 months ago

I can definitely see it happening on CI, but I don't have the time to investigate. Any help is welcome.

dylansunny commented 5 months ago

I think these Youtube videos need age confirmation. That's why the downloads are not accessible.

wasd52030 commented 5 months ago

I think these Youtube videos need age confirmation. That's why the downloads are not accessible.

It displays a similar page I knew from the start that this video was about suicide, but I just want to listen to the music

Tyrrrz commented 5 months ago

I had some time to investigate this issue, but unfortunately didn't come up with a solution. However, it seems that the deciphering process did not change and YoutubeExplode still performs that part correctly. My guess is that YouTube added a new form of verification for the TVHTML5 client (that we use for age-restricted videos), but I have not been able to reverse-engineer it yet.

vadmium commented 5 months ago

FYI I recently found that if stream URLs have an “n=. . .” query parameter, I have to transform the values or I get 403 responses. The transform function is in the player Java Script. To find the function, look where it is called, just after the .get("n") method call. For instance it is the yma function in the following.

yma = function(a) {
    var b = a.split("")
    /* . . . */
    return b.join("")
};
/* . . . */
(b = a.get("n"))
    && (b = yRa[0](b),
        a.set("n", b),
        yRa.length || yma(""))
/* . . . */
var yRa = [yma];

[Edit: discussion of the function from 2021: https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-866674128]

Tyrrrz commented 5 months ago

Thanks @vadmium.

I remember previously the n parameter was only responsible for whether the downloads are throttled or not. Is it now required to access the stream altogether?

Tyrrrz commented 3 months ago

Another video test case: https://www.youtube.com/watch?v=vcw5THyM7Jo

Tyrrrz / YoutubeExplode