ngomile / anigrab

Anime downloader that is fast and efficient
The Unlicense
124 stars 22 forks source link

AnimePahe started providing fake kwik links in the api response #10

Open newaccforhc opened 3 years ago

newaccforhc commented 3 years ago

name: "🐞 Bug report" about: Report a bug title: "[Bug] AnimePahe started providing fake kwik links in the api response" labels: "bug"


Describe the bug

AnimePahe started providing fake kwik links in the api response (those links go to 404). So, downloads from Animepahe aren't working. Now you need to go through adfly link before getting the real kwik link. ## Command You Entered ``` anigrab -u https://animepahe.com/anime/1e1d8732-a053-24d1-72cb-c8c6a895e3cb ``` ### Expected behavior Should give direct download links. ### Actual behavior Threw an error. ### Other details Maybe some adfly bypassing method can help to resolve this. On a positive note, they removed captcha to open kwik links now. So, cheers!
ngomile commented 3 years ago

I'm not sure how to handle this one but it might be that some header is required or something. I've checked but I honestly don't know, I believe the extractor is still fine but as you've stated, it's being given a fake link.

newaccforhc commented 3 years ago

@ngomile Oh.. No.. it has nothing to do with headers.. It's just you get a totally different link if you go through adfly or shst link..

Maybe doing something to bypass adfly or shst links would work.. There are many packages/modules to do that on github but, most of them are not maintained now.. So, idk if they still work or not.. But, you can check.

uali6981 commented 3 years ago

I got this error on "animePahe"

Bypassing captcha please wait... (node:6004) UnhandledPromiseRejectionWarning: TypeError: response.body.match is not a function or its return value is not iterable at module.exports.extract (C:\Users\------\AppData\Roaming\npm\node_modules\anigrab\src\extractors\kwik.js:51:44) (node:6004) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1) (node:6004) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

is this issue on my system or with code?

newaccforhc commented 3 years ago

@ngomile looks like you can refer here: https://github.com/KevCui/animepahe-dl/issues/8

uali6981 commented 3 years ago

@newaccforhc my bad that tool in m3u8. So I did not get enough downloading speed on that software

newaccforhc commented 3 years ago

Oh no.. I'm not talking about m3u8. Sorry for not mentioning before, What I wanted to say is that, you can get the real kwik links by referring to that tool. That tool is using stream links which have /e/ in them. If you replace it with /f/ then you get the kwik download links.

ngomile commented 3 years ago

So I was doing a little experimenting trying to get this working again like it was, I haven't looked deeply into the link @newaccforhc provided. I'll do that soon though seeing as how what I came up with is inefficient and uses way too much data and CPU. Essentially, I tried to use puppeteer-core to get the link from adfly. Starts too many Chrome processes and is too slow. For the curious souls the code looked like this:

async function getRealKwikLink(url) {
    const BLOCKED_RESOURCES_REG = /ads|inpagepush|css|facebook|analytics|cloud|png|hcaptcha|gif/;
    let kwikURL = '';
    let browser;

    try {
        browser = await puppeteer.launch({
            executablePath: browserExecutablePath,
        });
        const page = await browser.newPage();
        page.setRequestInterception(true);
        page.on('request', request => {
            if (BLOCKED_RESOURCES_REG.test(request.url())) {
                request.abort();
            } else {
                request.continue();
            }
        });

        await page.goto(url, { waitUntil: 'networkidle2' });
        await page.waitForSelector('#skip_bu2tton', { visible: true });
        kwikURL = await page.evaluate(
            '(() => document.querySelector("#skip_bu2tton").getAttribute("href"))()'
        );
    } catch (error) {
        console.error(error);
    } finally {
        await browser.close();
    }

    return kwikURL;
}

But it actually worked as shown below after entering this command anigrab beastars -u -s animepahe:

Capture

Just way too slow, this will be sorted out eventually... maybe.

IshigamiYuu commented 3 years ago

Screenshot (14) help it keeps going like this

ngomile commented 3 years ago

@IshigamiYuu animepahe isn't working at the moment.

justfoolingaround commented 3 years ago

The "kwik" links aren't fake. Kwik has a "DRM" system (really not a DRM but since you're having issues with it, we're going to act like it is one).

I'm not familiar with Node so I'll just drop you the method to get valid stream url(s) from AnimePahe.

First, make an API call to get the "kwik" URL. Then, make a GET request to that URL with "https://animepahe.com/" as the referer (referer in headers). This now allows you to access the site for further extraction. Now, you need to extract the m3u8 URL. There should be some tools in Node that allows JS evaluation but for a faster approach, use regex, here's the algorithm to extract "kwik" m3u8.

To stream/download the m3u8 is also quite troublesome due to the above mentioned "DRM" system, hence, you need to use the "kwik" url (not https://kwik.cx but the url from which you extracted the m3u8) as the referer in the download header.

Then, you should be able to stream/download from AnimePahe.

This is also the reason why web browsers are failing to stream even from the official AnimePahe site.

Good luck!