thorio / KGrabber

Userscript for extracting links from kissanime.ru and similar sites.
https://greasyfork.org/en/scripts/383649-kissgrabber
GNU General Public License v3.0
33 stars 12 forks source link

Downloading HydraX Streams #35

Open eslamsidereel opened 4 years ago

eslamsidereel commented 4 years ago

This script is amazing and after adding the idm export feature it became even more amazing. but since kissanime changed from rapidvideo to hydrax, i cant download any anime. I searched a lot but couldn't find a way to generate links from hydrax stream. It seems to be impossible so i had to use nova server and solve all the captcha to get the download links. If anybody knows a way to generate links from hydrax stream that would be a great help.

thorio commented 4 years ago

I tried (and failed) to do this a while ago. You can find my attempts on the hydrax branch. I got to the point of getting all the data they use to assemble the stream, but i wasn't able to assemble it or even play their assembled streams outside of their website. Perhaps someone more knowledgeable about HLS can crack this, in which case I'd be happy to write my findings down in detail.

thorio commented 4 years ago

Hydrax seems to have been removed. Re-open when it is spotted again.

Tankettekek commented 4 years ago

Hey it is back...

thorio commented 4 years ago

Indeed it is. I've re-added it to the script, although the links don't even work outside of an iframe anymore.

Shubhodeep84 commented 4 years ago

Hydrax got The Alliance for creativity (ACE) protection which was created on 2017 they are anti-piracy Organization. They was even shutdown Openload and streaming at 31 October 2019. I guess Hydrax thought better form alliance get protected then shutdown as well.

KartikSoneji commented 3 years ago

@thorio I spent some time messing around with HydraX, and so far this is what I have been able to reverse engineer:

If the embed URL is https://playhydrax.com/?v=VMG9wjJtM

To download the video, first you need to make a POST request to https://ping.idocdn.com/.

curl "https://ping.idocdn.com/" -d "slug=VMG9wjJtM"

Response:

{
    "status": true,
    "url": "m0weXd1eG1pdS54eXo=a",
    "sources": ["sd", "hd"],
    "isCdn": false,
    "isHls": false,
    "ads": "https://unreshiramor.com/4/3239564/"
}

Then, you can start downloading the video.

curl "https://www.4vs5osstge.buzz/" -H "Referer: https://playhydrax.com/?v=VMG9wjJtM" -o "video.mp4"

Notes

I haven't use IDM a lot, so I don't know if this method can be used with it. PowerShell does have an in-built Invoke-WebRequest command, and MacOS and Linux ship with curl so we can generate download scripts.

JambDev commented 3 years ago

It appears that the "www." in the URL actually determines it's quality. Ex. https://www.4vs5osstge.buzz/ would download 720p and https://4vs5osstge.buzz/ would download 360p

I still have no idea how it deciphers the link from "m0weXd1eG1pdS54eXo=a" EDIT: I actually did end up figuring out how to "decode" the URL provided by ping.idocdn.com All you have to do is move the last character in the string to the beginning and then decode it in base64.

thorio commented 3 years ago

am0weXd1eG1pdS54eXo= decodes to jm0ywuxmiu.xyz, which is different from 4vs5osstge.buzz but still works. Maybe the url doesn't actually matter? When you open the dev console on the kissanime page it tells you it's a google drive proxy, so the servers might all be identical.

@KartikSoneji I don't think IDM supports Headers outside of the browser extensions.

KartikSoneji commented 3 years ago

@JambDev

It appears that the "www." in the URL actually determines it's quality.

Hmm, quite clever.

EDIT: I actually did end up figuring out how to "decode" the URL provided by ping.idocdn.com All you have to do is move the last character in the string to the beginning and then decode it in base64.

Great! How did you figure it out? I would never have thought that a company would use something like this. Looks like HydraX is going for security through obscurity.

Still haven't been able to test HSL though.

@thorio Interestingly, you don't have to use the url returned from https://ping.idocdn.com. For example, I used 4vs5osstge.buzz for all requests, and it worked for (all of my) ~140 videos. If you resolve 4vs5osstge.buzz and jm0ywuxmiu.xyz with nslookup, you will find that they point to different ip addresses. Maybe the multiple urls are a form of load-balancing? But it is good that we can decode them now.

@KartikSoneji I don't think IDM supports Headers outside of the browser extensions.

I was afraid of that. There has been some discussion about allowing custom headers in VLC for remote playback, but it does not seem to be implemented currently.

How about generating a PowerShell/cmd/shell script to download the videos? I still need to work out some details (like hd/sd selection and fixing the base64) but this is what I have so far:

# downloadHydraXVideo <slug> <output file>
downloadHydraXVideo(){  
    url64=$(curl "https://ping.idocdn.com/" -d "slug=$1" | grep -Po '(?<="url":")[a-zA-Z0-9=+-]*?(?=",)');

    # TODO: Fix the base64 string by moving the last character to the front.
    curl "$(echo $url64 | base64 -d)" -# -C - -H "Referer: https://playhydrax.com/?v=$1" -o "$2";
}

downloadHydraXVideo "VMG9wjJtM" "Video 1.mp4"
downloadHydraXVideo "TiDe3FPHK" "Video 2.mp4"
#...
thorio commented 3 years ago

@KartikSoneji I believe aria2c supports custom headers, as well as multithreaded downloads. I looked into it a while ago, I don't remember why I didn't pursue it further. We should be able to export a file with all the urls and headers required.

KartikSoneji commented 3 years ago

@thorio Agree, but that would be an external dependency. curl is installed on almost all systems, and Invoke-WebRequest ships with all modern versions of Windows. But it can be good as an additonal option though.

thorio commented 3 years ago

There are already aria2c and IDM exporters in the script. I don't believe curl can be multithreaded on the same download? I'll go ahead and add an action to ping idocdn and get the url, as well as a new aria2c exporter for this. Feel free to open a PR for the curl version.

KartikSoneji commented 3 years ago

I don't believe curl can be multithreaded on the same download?

Unfortunately, no :(

I'll go ahead and add a step to ping idocdn and get the url

If there is a large delay (> 5-6 hours) between pinging idocdn and downloading the video then you might get an Access Denied error. The only way that worked reliably is to ping idocnd right before trying to download the video.

I don't know if that is possible in aria2c without some shell scripting.

Feel free to open a PR for the curl version.

OK, will do that.

thorio commented 3 years ago

I pushed the action to feature/hydrax-download. You can copy one of the existing exporters and edit it accordingly.

KartikSoneji commented 3 years ago

@thorio Thanks, will take a look.

shinji257 commented 3 years ago

I wrote up a quick bash script that you can provide the stub to and it will retrieve the video. It assumes 720p for the video resolution. I don't know what would happen if that wasn't available and wasn't tested. If you were grabbing a series that only had a 360p version you can remove the www. in the url that it accesses in order to get that one. I might implement a check that handles it. Not sure. For now just remove the www. in the url if you know you can't get a 720p version. It will automatically end up grabbing the 360p version of a 720p version isn't availabele. Resulting JSON says SD where it would normally have SD/HD as the available formats and the download is SD only even with the www. on the front.

--insecure is needed on the second curl statement as it seems that at least one of the domains to download the video from doesn't have a valid SSL certificate so curl aborted on it.

EDIT: If you couldn't tell the default filename is $STUB.mp4 as in if the stub is 18IX9abCy it translates to 18IX9abCy.mp4. This is assuming you don't give it a filename in the first place. Wrap whatever filename you want to give in quotes if it has spaces.

You can replace the curl near the end with aria2c by using this line. aria2c --check-certificate=false "https://www.$URL/" --referer="https://playhydrax.com/?v=$STUB" -o "$OUT"

#!/usr/bin/env bash

# Video stub
STUB=$1

# Output filename.  If none set in command then default to stub.mp4
OUT=$2
if [ -z "$OUT" ]; then OUT="$STUB.mp4"; fi

# Grab JSON data from server
JSON=`curl -s "https://ping.idocdn.com/" -d "slug=$STUB"`

# Cut away and eventually get domain to download video from. -- Load balancing?
# A bit messy here.  First cut away excess data and remove enclosing quotes.
# After use a bit of possible bashism to rotate string so that the last character is in front of first character.
# Finally store the URL for the retrieval.
URLB64=`echo $JSON | cut -d, -f2 | cut -d: -f2`
URLB64=`sed -e 's/^"//' -e 's/"$//' <<<"$URLB64"`
LURL=${#URLB64}
URLB64=${URLB64:$LURL-1:1}${URLB64:0:$LURL-1}
URL=`echo $URLB64 | base64 -d`

# Testing to see if HD exists.  If it doesn't warn to user that SD is being downloaded instead.
if ! [[ "`echo $JSON | cut -d[ -f2 | cut -d] -f1`" =~ .*"hd".* ]]; then echo -e "\e[31m[WARN]\e[39m No HD version.  Downloading SD instead."; fi

curl --insecure "https://www.$URL/" --referer "https://playhydrax.com/?v=$STUB" -o "$OUT"
JambDev commented 3 years ago

I've noticed that the requests HydraX's player is making don't directly fetch the mp4 anymore. At the time of writing this, the current method (in the replies to this issue) still works. I am afraid that it might break though with this new web socket approach they are using

shinji257 commented 3 years ago

This is no longer needed. KissAnime.ru shut down permanently a few minutes ago.

JambDev commented 3 years ago

welp

thorio commented 3 years ago

@JambDev When I originally tried this (before this issue was opened) they were using HLS, so they might be changing things around a lot.

@shinji257 There is still kissanime.nz (#39), which also has a hydrax option (Hserver). They also don't seem to have captchas? I will look at adding support for it again when I get the chance, but any help is appreciated.

shinji257 commented 3 years ago

@thorio Ahh. I was unaware that it also used hydrax.

thorio commented 3 years ago

@shinji257 Yes, and if you look closely you can still find the familiar ping to idocdn in the network tab. So it's just a matter of figuring out how they get the link to the iframe.

KartikSoneji commented 3 years ago

@thorio @shinji257

kissanime.nz is down as well, so maybe the outage is temporary?

image

shinji257 commented 3 years ago

KissAnime.nz is still working here.

On Sat, Aug 15, 2020, 9:18 AM Kartik Soneji notifications@github.com wrote:

@thorio https://github.com/thorio @shinji257 https://github.com/shinji257

kissanime.nz is down as well, so maybe the outage is temporary?

[image: image] https://user-images.githubusercontent.com/25801283/90313154-da746080-df27-11ea-99ed-853907388ede.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thorio/KGrabber/issues/35#issuecomment-674395493, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB7XDNJTQ3IE5SDSRIYFC3SA2DKDANCNFSM4JTINIAA .

KartikSoneji commented 3 years ago

Yay, kissanime.nz works for me now!

KartikSoneji commented 3 years ago

I looked at https://kisscartoon.nz/ and https://kimcartoon.si/ (#46)

For https://kisscartoon.nz/Cartoon/Angry-Birds.88861/Movie-HD?id=71028

# server = hserver | xserver | ptserver | oserver
curl "https://kisscartoon.nz/ajax/anime/load_episodes_v2?s=<server>" -d "episode_id=71028"

Using hserver:

curl "https://kisscartoon.nz/ajax/anime/load_episodes_v2?s=hserver" -d "episode_id=71028"
{
    "status": true,
    "value": "<iframe src=\"https:\/\/embed.streamx.me?k=756ecb2dfb240248235934ab3ac1b943&li=71028&tham=1597515866&lt=hydrax&check_hot=1&qlt=720p&spq=p&prv=bWVkaWEvdGh1bWIvMTcwNDA4XzA1NDYvQW5ncnktQmlyZHMuODgxNTktTW92aWUtSEQuanBnO21lZGlhL3RodW1iLzE3MDQwOF8wNTQ2L0FuZ3J5LUJpcmRzLjg4MTU5LU1vdmllLUhEX21vYmlsZS5qcGc%3D&key=57443b94a0716a10f23334135e9979e1&ua=5f2bf4be52e6a7bdd37bca5ebabb7b2c\" width=\"100%\" height=\"100%\" style=\"border:none;overflow:hidden\" scrolling=\"no\" frameborder=\"0\" allowTransparency=\"true\" allowFullScreen=\"true\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\"><\/iframe>",
    "embed": true,
    "html5": false,
    "type": "",
    "sv": "hydrax",
    "download_get": ""
}

Using the url from the iframe's src. (referer is needed)

curl \
    --referer "https://kisscartoon.nz/" \
    "https://embed.streamx.me?k=756ecb2dfb240248235934ab3ac1b943&li=71028&tham=1597515866&lt=hydrax&check_hot=1&qlt=720p&spq=p&prv=bWVkaWEvdGh1bWIvMTcwNDA4XzA1NDYvQW5ncnktQmlyZHMuODgxNTktTW92aWUtSEQuanBnO21lZGlhL3RodW1iLzE3MDQwOF8wNTQ2L0FuZ3J5LUJpcmRzLjg4MTU5LU1vdmllLUhEX21vYmlsZS5qcGc%3D&key=57443b94a0716a10f23334135e9979e1&ua=5f2bf4be52e6a7bdd37bca5ebabb7b2c"

returns a lot of HTML, but the interesting bit is

<script>
    new Player({
        "key": "e6d3ce91839c03915de9125be5ed1f47",
        "type": "slug",
        "value": "3aOgVSRom39",
        "width": "100%",
        "height": "100%",
        "cast": "{}",
        "autostart": true,
        tracks: [[]]
    });
</script>

Which finally gives us the slug as 3aOgVSRom39.

The same works for https://kimcartoon.si/, simply by replacing kisscartoon.nz with kimcartoon.si.

Update: https://kissanime.nz also seems to use the same setup, but they have a LOT of scraping protection (captchas, csrf tokens).

Thanhcaro commented 3 years ago

Please help me download this type of link: https://playhydrax.com/?v=GDUc-uLQN Sorry I don't know how to use script, I install on Chrome and open website but still can not load video.

KartikSoneji commented 3 years ago

Hi @Thanhcaro Please refer to the usage instructions: https://github.com/thorio/KGrabber#installation-and-usage

Thanhcaro commented 3 years ago

Hi @Thanhcaro Please refer to the usage instructions: https://github.com/thorio/KGrabber#installation-and-usage

The script is currently compatible with:

kissanime.ru kimcartoon.to kissasian.sh kisstvshow.to

my website is not in this list.

KartikSoneji commented 3 years ago

@Thanhcaro

https://github.com/thorio/KGrabber#compatibility

If you'd like to add a site, please open a new issue on GitHub.

Can you open an issue?

Thanhcaro commented 3 years ago

@Thanhcaro

https://github.com/thorio/KGrabber#compatibility

If you'd like to add a site, please open a new issue on GitHub.

Can you open an issue?

Oh Sorry. I will open new issue. Thanks for your help.

Thanhcaro commented 3 years ago

Sorry, i thought script can download all hydrax link from any website. I already open new issue, pls do me a favour. Thank you.

Vào 00:46, T.3, 6 Th10, 2020 Kartik Soneji notifications@github.com đã viết:

Can you let us know the name of your website?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/thorio/KGrabber/issues/35#issuecomment-703785742, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQZLVXLK6I7JTD7NFWAGKS3SJIA6XANCNFSM4JTINIAA .

mrwunderbar666 commented 3 years ago

I have been lurking this issue for a while now and was able to implement the hydrax downloader in Python.

But recently, something about the https://ping.iamcdn.net/ has changed. It now returns a 502 error.

Using the question from #54 as an example:

curl "https://ping.iamcdn.net/" -d "slug=7fUvtRbiYk"

Returns:

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.19.6</center>
</body>
</html>

However, it appears that the following steps remained the same. I could download files via trial and error by using the old domains such as: curl "https://x00vfu277x.buzz/" -H "Referer: https://playhydrax.com/?v=TZTopeD-V" -o "test01.mp4"

I tried to dig in the js files for clues, but couldn't find anything :/

KartikSoneji commented 3 years ago

@mrwunderbar666 Seems like the ping.iamcdn.net step is no longer necessary.

Running:

curl -L https://playhydrax.com/?v=TZTopeD-V

The interesting bit:

<script type="text/javascript">
    //...
    if ("undefined" != typeof ADS)
        new ADS(atob("eyJzdGF0dXMiOnRydWUsIm1kNV9pZCI6MTAzNTc3MjUsInNvdXJjZXMiOlsic2QiLCJoZCJdLCJzb3VyY2VfYnJlYWtzIjpbXSwiZG9tYWluIjoiNDd5YW9odWhpbi5idXp6IiwidXNlX3AycCI6dHJ1ZSwidXNlX2Rvd24iOmZhbHNlfQ=="))

    var styleMinion = "width:100%;height:100% !important;background: url(https://iamcdn.net/img/block2.jpg) center center no-repeat;background-size:contain;position:absolute"
</script>

decoding the base64 data gives:

{
    "status": true,
    "md5_id": 10357725,
    "sources": ["sd", "hd"],
    "source_breaks": [],
    "domain": "47yaohuhin.buzz",
    "use_p2p": true,
    "use_down": false
}

Shell script:

curl -sL "https://playhydrax.com/?v=TZTopeD-V" |\
    grep "ADS(" |\
    sed -r 's/.*"(.*)".*/\1/' |\
    base64 -di
mrwunderbar666 commented 3 years ago

@KartikSoneji Thanks that did the trick!

That is now much easier to implement.

I just noticed that some of the domains have issues with SSL. No matter how I connect to some of them, some throw a handshake error:

HTTPSConnectionPool(host='www.x0g2gwwwr7.xyz', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1123)')))

I wonder, how the browser resolves that, because in curl the option --insecure does not help for handshake errors

Thanhcaro commented 3 years ago

@KartikSoneji @mrwunderbar666 Thanks that did the trick!

That is now much easier to implement.

I just noticed that some of the domains have issues with SSL. No matter how I connect to some of them, some throw a handshake error:

HTTPSConnectionPool(host='www.x0g2gwwwr7.xyz', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1123)')))

I wonder, how the browser resolves that, because in curl the option --insecure does not help for handshake errors

You are so talent. Can you tell me the way to get hydrax link to download?

kukhanh29 commented 3 years ago

@KartikSoneji @mrwunderbar666 Thanks that did the trick! That is now much easier to implement. I just noticed that some of the domains have issues with SSL. No matter how I connect to some of them, some throw a handshake error:

HTTPSConnectionPool(host='www.x0g2gwwwr7.xyz', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1123)')))

I wonder, how the browser resolves that, because in curl the option --insecure does not help for handshake errors

You are so talent. Can you tell me the way to get hydrax link to download?

A fool does not read the comments above.

kukhanh29 commented 3 years ago

wow. seems it was blocked by encryption js

tienduynguyen318 commented 3 years ago

it seems that hydrax.net has become abyss.to. Can anyone still download the files with the curl? If you can, please share the result.

Thanks

lavara123 commented 3 years ago

@KartikSoneji Do you know about abyss.to? how to do

KartikSoneji commented 3 years ago

@lavara123 If you find a website that uses abyss.to, then I'll be happy to update the script.

lavara123 commented 3 years ago

@lavara123 If you find a website that uses abyss.to, then I'll be happy to update the script.

Thanks a lot, I'm looking forward to it

curl 'https://embed2.megaxfer.ru/embed2/01bded490f99430277078b8ad5ae9af8' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36' \
  -H 'referer: https://www.2embed.ru/' \
  --compressed

curl output:

    <iframe src="https://playhydrax.com/?v=2FKgbNJtB&sub=https://cc.1clickcdn.ru/8d/3b/8d3b7506346ba4ef945e5274488b094e/eng-6.vtt" width="100%" height="100%" scrolling="no" frameborder="0"
            allowFullScreen="true" webkitallowfullscreen="true"
            mozallowfullscreen="true"></iframe>

and next link : https://geoip.redirect-ads.com/?v=2FKgbNJtB

KartikSoneji commented 3 years ago

Looks like they are using CloudFlare, this is not going to be as easy.

lavara123 commented 3 years ago

Looks like they are using CloudFlare, this is not going to be as easy.

wget -d --header="User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" https://playhydrax.com/?v=2FKgbNJtB
crypto-0 commented 2 years ago

@KartikSoneji @lavara123 did you ever figure out how to bypass the cloudflare I have been stuck for days in figuring out that section I don’t know what checks are being performed

KartikSoneji commented 2 years ago

Hi @crypto-0 Before, I was able to bypass Cloudflare's captcha with these options, but I haven't tested it recently.

curl -L --user-agent "Mozilla/5.0" --cipher AES256-SHA256 --tls-max 1.2 "<url>"

In nodejs, you have to set the cipher explictly:

const https = require("https");

async function fetch(url, options){
    return new Promise((resolve, reject) => {
        let req = https.request(url, options, res => {
            let data = [];
            res.on("data", chunk => data.push(chunk));
            res.on("end", e =>
                resolve({
                    headers: res.headers,
                    status: res.statusCode,
                    text: async () => Buffer.concat(data).toString()
                })
            );
        });

        req.on("error", reject);

        req.end();
    });
}

// use
let response = await fetch(url, {
    headers: { "User-Agent": "Mozilla/5.0" },
    ciphers: "AES256-SHA256"
});

Hope that helps.

kukhanh29 commented 2 years ago

Hi @crypto-0 Before, I was able to bypass Cloudflare's captcha with these options, but I haven't tested it recently.

curl -L --user-agent "Mozilla/5.0" --cipher AES256-SHA256 --tls-max 1.2 "<url>"

In nodejs, you have to set the cipher explictly:

const https = require("https");

async function fetch(url, options){
    return new Promise((resolve, reject) => {
        let req = https.request(url, options, res => {
            let data = [];
            res.on("data", chunk => data.push(chunk));
            res.on("end", e =>
                resolve({
                    headers: res.headers,
                    status: res.statusCode,
                    text: async () => Buffer.concat(data).toString()
                })
            );
        });

        req.on("error", reject);

        req.end();
    });
}

// use
let response = await fetch(url, {
    headers: { "User-Agent": "Mozilla/5.0" },
    ciphers: "AES256-SHA256"
});

Hope that helps.

Thank for share. how to download stream ? Can you do it ?

Nicryc commented 2 years ago

Any chance someone found a working solution to this issue?

metzger100 commented 1 year ago

@KartikSoneji Sorry for Pinging you, but I am searching for method to download content from abyss.to. To give an example Movie: https://topstreamfilm.org/player.html?v=jxx2_qNnl. I searched for a long time now in the Internet for a working method but I didn't find any. The Adress of the Movie is freeplayervideo.com/?v=jxx2_qNnl but I have no Idea how to get a working download from it. I am relativley new to Web Reverse Engineering and I know the methods to download Files directly with rightclickenabler or Monitoring the Networktraffic to get a .m3u8-File to download the Movie with ffmpeg. Would be awesome if you could help me understanding what abyss.to does and how I can get the Download for movies hosted by abyss.to. I checked the Docs of JW-Player but that didn't help me a lot. Greetings Metzger.

PatrickL546 commented 4 months ago

I made a repo since it's too long How-to-download-hydrax-abyss.to