Cloudflare enabled on API endpoints

cerinoligutom commented 2 years ago

Try this url on an http client: https://nhentai.net/api/gallery/177013

Zekfad commented 2 years ago

That's site issue for sure. The only thing we can do is to pass cookies into agent. I can try to add some example solutions (e.g. try, catch cf, get token, retry with token). That's really tricky, because CF made to prevent scrapping/boting/etc/, which is what we're doing.

Chaoray commented 2 years ago

sometimes api works but most time it doesn't. I hope api can be work again cuz this is the most comfortable nhentai api in my mind <3

Zekfad commented 2 years ago

Current workaround: You'll need tough-cookie and http-cookie-agent for this.

// index.mjs
import { API, TagTypes, } from 'nhentai-api';
import { CookieJar } from 'tough-cookie';

import _httpCookie from 'http-cookie-agent/http';
const { HttpsCookieAgent: CookieAgent } = _httpCookie;

const jar = new CookieJar();
const agent = new CookieAgent({ cookies: { jar, }, });

jar.setCookie('cf_clearance=4iV5fuQ0.prnWCqcZ.kqLFV6hp.QQ5y_lRkEw97N3ZA-1653929202-0-150', 'https://nhentai.net/');

const api = new API({ agent, });

api.getBook(177013).then(book => {
    console.log(book.title.pretty);
});

To get cf_clearance cookie:

Get module User-Agent:
- Run node -p "(new (require('nhentai-api').API)()).getBook(-1).catch(e => console.log(e.httpResponse.req.getHeaders()['user-agent']));"
Open https://nhentai.net/api/galleries/search in browser.
Open DevTools and set User Agent to module's:
- Open bottom panel if it's not present (ESC).
- Under three dots (⋮) open Network conditions.
- Set custom User Agent to the module's:
  
  Screenshot
Without closing DevTools reload page by pressing to URL and pressing enter (needed to refresh headers)
Wait for Cloudflare to complete.
Read cookie from Network tab and request (make sure you're watching at request with your User-Agent, on screenshot you can see it as the last header):

Screenshot
![image](https://user-images.githubusercontent.com/8970959/171037214-d4767f8d-4d8f-40d3-bd94-84b6c47cc4e8.png)
Place cookie to code.

Chaoray commented 2 years ago

Current workaround: You'll need tough-cookie and http-cookie-agent for this.
// index.mjs
import { API, TagTypes, } from 'nhentai-api';
import { CookieJar } from 'tough-cookie';

import _httpCookie from 'http-cookie-agent/http';
const { HttpsCookieAgent: CookieAgent } = _httpCookie;

const jar = new CookieJar();
const agent = new CookieAgent({ cookies: { jar, }, });

jar.setCookie('cf_clearance=4iV5fuQ0.prnWCqcZ.kqLFV6hp.QQ5y_lRkEw97N3ZA-1653929202-0-150', 'https://nhentai.net/');

const api = new API({ agent, });

api.getBook(177013).then(book => {
  console.log(book.title.pretty);
});
To get cf_clearance cookie:

Get module User-Agent:

Run node -p "(new (require('nhentai-api').API)()).getBook(-1).catch(e => console.log(e.httpResponse.req.getHeaders()['user-agent']));"

Open https://nhentai.net/api/galleries/search in browser.

Open DevTools and set User Agent to module's:

Open bottom panel if it's not present (ESC).

Under three dots (⋮) open Network conditions.

Set custom User Agent to the module's:

Screenshot

Without closing DevTools reload page by pressing to URL and pressing enter (needed to refresh headers)

Wait for Cloudflare to complete.

Read cookie from Network tab and request (make sure you're watching at request with your User-Agent, on screenshot you can see it as the last header):

Screenshot

Place cookie to code.

Is this permanent or need to be refreshed once in a while? 🤔

Zekfad commented 2 years ago

@Chaoray, sadly, it's temporary and you'll need to refresh it, We're working on a more automated solution, when it's done (or when maybe site admins disable CF on API) I'll tell you more info.

Chaoray commented 2 years ago

OK, I think it is enough for personal usage. I will wait for furthur infomation 😊

Tenandrobilgi commented 2 years ago

Having the same problem. Tried the solution above but still doesn't work.

Zekfad commented 2 years ago

Make sure you got right token for right User Agents, check if UA Header is the one using by your node.

sinkaroid commented 2 years ago

There is at least has 2 workarounds:

Like mentioned above, You need some tricky with cf_clearance stuff. It is works outside this module, I've implement that raw request through phin. But it seems cookie have an expire time, so you need to set it again.
Use their IP. After doing some reversing on nhentai.net, Looks like i get the real ip behind cloudflare https://github.com/sinkaroid/jandapress/blob/master/src/utils/options.ts#L7..L9

@Zekfad one question, So I can't arbitrary set? When i fill it with your module nhentai-api-client/3.4.3 Node.js/16.9.1 It's works

But when i just arbitrary ie: mymodule/1.0.0 Node.js/16.9.1 returns 503 aka still cloudflare protection. It's working as expected? I can't using own module to mock them Screenshot_556

import p from "phin";
import { CookieJar } from "tough-cookie";
import { HttpsCookieAgent } from "http-cookie-agent/http";

const jar = new CookieJar();
jar.setCookie("some cookie", "https://nhentai.net/");

async function test() {
  const res = await p({
    url: "https://nhentai.net/api/galleries/search?query=futa",
    core: {
      agent: new HttpsCookieAgent({ cookies: { jar, }, }),
    },
    "headers": {
      "User-Agent": "jandapress/1.0.5 Node.js/16.9.1" // just worked with "nhentai-api-client", cannot use my own even i've already to set it
    },
  });
  console.log(res.statusCode);
}

test();

Zekfad commented 2 years ago

@sinkaroid token is linked to the user agent, so you have to get new one for each UA you use.

sinkaroid commented 2 years ago

Thanks it's working now

Chaoray commented 2 years ago

There is at least has 2 workarounds:

Like mentioned above, You need some tricky with cf_clearance stuff. It is works outside this module, I've implement that raw request through phin. But it seems cookie have an expire time, so you need to set it again.

Use their IP. After doing some reversing on nhentai.net, Looks like i get the real ip behind cloudflare https://github.com/sinkaroid/jandapress/blob/master/src/utils/options.ts#L7..L9

@Zekfad one question, So I can't arbitrary set? When i fill it with your module nhentai-api-client/3.4.3 Node.js/16.9.1 It's works

But when i just arbitrary ie: mymodule/1.0.0 Node.js/16.9.1 returns 503 aka still cloudflare protection. It's working as expected? I can't using own module to mock them
import p from "phin";
import { CookieJar } from "tough-cookie";
import { HttpsCookieAgent } from "http-cookie-agent/http";

const jar = new CookieJar();
jar.setCookie("some cookie", "https://nhentai.net/");

async function test() {
  const res = await p({
    url: "https://nhentai.net/api/galleries/search?query=futa",
    core: {
      agent: new HttpsCookieAgent({ cookies: { jar, }, }),
    },
    "headers": {
      "User-Agent": "jandapress/1.0.5 Node.js/16.9.1" // just worked with "nhentai-api-client", cannot use my own even i've already to set it
    },
  });
  console.log(res.statusCode);
}

test();

How to use their IP?
Send requests directly to these IP?

Edit: I still got 503 after I checked all steps and settings. Is this won't work on host like heroku? But It worked on my pc when testing..... I have no idea

Zekfad commented 2 years ago

Is this won't work on host like heroku?

Indeed it won't. I mean, token is IP bound. If you have your server you can of course start proxy, and complete challenge though proxy, to get valid token for given IP. But that's not an option for heroku I guess. You can try method from https://github.com/Zekfad/nhentai-api/issues/25#issuecomment-1147142230 But I'd be against it, because it will give them a reason to change IPs of their servers if they'll get too much traffic.

Chaoray commented 2 years ago

Is this won't work on host like heroku?

Indeed it won't. I mean, token is IP bound. If you have your server you can of course start proxy, and complete challenge though proxy, to get valid token for given IP. But that's not an option for heroku I guess. You can try method from #25 (comment) But I'd be against it, because it will give them a reason to change IPs of their servers if they'll get too much traffic.

Ok¯\_(ツ)_/¯ Now the problem is how to apply ip to nhentai-api :D

const api = new API({
    hosts: {
        api: '35.186.156.165'
    },
    ssl: false
});

Like this?

Tenandrobilgi commented 2 years ago

Is this won't work on host like heroku?

Indeed it won't. I mean, token is IP bound. If you have your server you can of course start proxy, and complete challenge though proxy, to get valid token for given IP. But that's not an option for heroku I guess. You can try method from #25 (comment) But I'd be against it, because it will give them a reason to change IPs of their servers if they'll get too much traffic.

Oh, so that's why it wasn't working. But how would I go on about using their IP to get through Cloudflare?

Zekfad commented 2 years ago

@Chaoray, just IP, don't specify URI. Also, disable SSL. @Tenandrobilgi, you can use proxy I think, if it's not against the TOS, Chromium based browsers has command-line for that.

Chaoray commented 2 years ago

Using site IPs is perfect but getRandomBook() lost its function. :( sad

Chaoray commented 2 years ago

Cloudflare blocked direct ip access. And, error logs:

APIError: connect ETIMEDOUT 104.21.66.123:80
    at Function.absorb (/app/node_modules/nhentai-api/dist/cjs/bundle.cjs:471:66)
    at ClientRequest.<anonymous> (/app/node_modules/nhentai-api/dist/cjs/bundle.cjs:523:647)
    at ClientRequest.emit (node:events:527:28)
    at Socket.socketErrorListener (node:_http_client:454:9)
    at Socket.emit (node:events:527:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  originalError: Error: connect ETIMEDOUT 104.21.66.123:80
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16) {
    errno: -110,
    code: 'ETIMEDOUT',
    syscall: 'connect',
    address: '104.21.66.123',
    port: 80
  },
  httpResponse: null
}

Hary1723 commented 2 years ago

Cloudflare blocked direct ip access. And, error logs:

APIError: connect ETIMEDOUT 104.21.66.123:80
    at Function.absorb (/app/node_modules/nhentai-api/dist/cjs/bundle.cjs:471:66)
    at ClientRequest.<anonymous> (/app/node_modules/nhentai-api/dist/cjs/bundle.cjs:523:647)
    at ClientRequest.emit (node:events:527:28)
    at Socket.socketErrorListener (node:_http_client:454:9)
    at Socket.emit (node:events:527:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  originalError: Error: connect ETIMEDOUT 104.21.66.123:80
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16) {
    errno: -110,
    code: 'ETIMEDOUT',
    syscall: 'connect',
    address: '104.21.66.123',
    port: 80
  },
  httpResponse: null
}

~~What is that IP address?~~ Nevermind

Hary1723 commented 2 years ago

@Zekfad How would I use IP instead of the default URL? I tried from this but kept getting errors

const api = new API({
    hosts: {
        api: '35.186.156.165'
    },
    ssl: false
});

Zekfad commented 2 years ago

I guess it's no longer possible to bypass it this way, as both IPs I know doesn't work.

Hary1723 commented 2 years ago

The IP http://138.2.77.198:3002 works on the browser though. at least to me

haine4 commented 2 years ago

Current workaround: You'll need tough-cookie and http-cookie-agent for this.

// index.mjs
import { API, TagTypes, } from 'nhentai-api';
import { CookieJar } from 'tough-cookie';

import _httpCookie from 'http-cookie-agent/http';
const { HttpsCookieAgent: CookieAgent } = _httpCookie;

const jar = new CookieJar();
const agent = new CookieAgent({ cookies: { jar, }, });

jar.setCookie('cf_clearance=4iV5fuQ0.prnWCqcZ.kqLFV6hp.QQ5y_lRkEw97N3ZA-1653929202-0-150', 'https://nhentai.net/');

const api = new API({ agent, });

api.getBook(177013).then(book => {
  console.log(book.title.pretty);
});

To get cf_clearance cookie:

* Get module User-Agent:

  * Run `node -p "(new (require('nhentai-api').API)()).getBook(-1).catch(e => console.log(e.httpResponse.req.getHeaders()['user-agent']));"`

* Open https://nhentai.net/api/galleries/search in browser.

* Open DevTools and set User Agent to module's:

  * Open bottom panel if it's not present (ESC).
  * Under three dots (`⋮`) open **Network conditions**.
  * Set custom User Agent to the module's:

    Screenshot
    ![image](https://user-images.githubusercontent.com/8970959/171036425-34d04def-c785-49fa-9ea7-ee46257778be.png)

* Without closing DevTools reload page by pressing to URL and pressing enter (needed to refresh headers)

* Wait for Cloudflare to complete.

* Read cookie from **Network** tab and request (make sure you're watching at request with your User-Agent, on screenshot you can see it as the last header):

  Screenshot
  ![image](https://user-images.githubusercontent.com/8970959/171037214-d4767f8d-4d8f-40d3-bd94-84b6c47cc4e8.png)

* Place cookie to code.

Hey It's working on my local, but how to use this with repl / railway / or heroku?

haine4 commented 2 years ago

The IP http://138.2.77.198:3002 works on the browser though. at least to me

It's the nhentai address? how did you know?

Hary1723 commented 2 years ago

Hey It's working on my local, but how to use this with repl / railway / or heroku?

Same question here, hence I thought using the IP might work.

It's the nhentai address? how did you know?

https://github.com/sinkaroid/jandapress/blob/master/src/utils/options.ts#L10

Uryaaa commented 2 years ago

The IP http://138.2.77.198:

Now how to use this with nhentai-api

Chaoray commented 2 years ago

No, you can't. Nhentai blocked direct ip access, which was the previous method we were using before it was blocked.
Or you can do one thing that coverts a book number to thumbnail urls and image urls and basically just let them load from user client.

lewisakura commented 1 year ago

Nhentai blocked direct ip access

@Chaoray this is incorrect. The direct IP access error comes from the fact that the IP the person attempted to use was a Cloudflare network IP, which naturally disallows direct access (otherwise how would they know what you're trying to connect to?). If you navigate to the IP and port, with http (not https!), you can in fact get it to load just fine. It looks broken because all of the CSS isn't there, but other than that it is fully functional, API included.

With a bit of patching, you can get the library to work fine with the IP address:

diff --git a/node_modules/nhentai-api/dist/cjs/bundle.cjs b/node_modules/nhentai-api/dist/cjs/bundle.cjs
index 46ad4e5..840d3b3 100644
--- a/node_modules/nhentai-api/dist/cjs/bundle.cjs
+++ b/node_modules/nhentai-api/dist/cjs/bundle.cjs
@@ -507,7 +507,7 @@ class API{
    * Applies provided options on top of defaults.
    * @param {?nHentaiOptions} [options={}] Options to apply.
    */
-constructor(options={}){_defineProperty(this,"hosts",void 0),_defineProperty(this,"ssl",void 0),_defineProperty(this,"agent",void 0);let params=function processOptions({hosts:{api:api="nhentai.net",images:images="i.nhentai.net",thumbs:thumbs="t.nhentai.net"}={},ssl:ssl=!0,agent:agent=null}={}){return agent||(agent=ssl?https.Agent:http.Agent),"Function"===agent.constructor.name&&(agent=new agent),{hosts:{api:api,images:images,thumbs:thumbs},ssl:ssl,agent:agent}}(options);Object.assign(this,params)}
+constructor(options={}){_defineProperty(this,"hosts",void 0),_defineProperty(this,"ssl",void 0),_defineProperty(this,"agent",void 0);let params=function processOptions({hosts:{api:api="138.2.77.198",images:images="i.nhentai.net",thumbs:thumbs="t.nhentai.net"}={},ssl:ssl=!0,agent:agent=null}={}){return agent||(agent=ssl?https.Agent:http.Agent),"Function"===agent.constructor.name&&(agent=new agent),{hosts:{api:api,images:images,thumbs:thumbs},ssl:ssl,agent:agent}}(options);Object.assign(this,params)}
 /**
    * Get http(s) module depending on `options.ssl`.
    * @type {https|http}
@@ -518,7 +518,7 @@ constructor(options={}){_defineProperty(this,"hosts",void 0),_defineProperty(thi
    * @param {string} options.host Host.
    * @param {string} options.path Path.
    * @returns {Promise<object>} Parsed JSON.
-   */request(options){let{net:net,agent:agent}=this;return new Promise(((resolve,reject)=>{Object.assign(options,{agent:agent,headers:{"User-Agent":`nhentai-api-client/3.4.3 Node.js/${process.versions.node}`}}),net.get(options,(_response=>{const
+   */request(options){let net=(options.host==="138.2.77.198"?http__default.default:this.net),agent=(options.host==="138.2.77.198"?new http.Agent():this.agent);return new Promise(((resolve,reject)=>{Object.assign(options,{agent:agent,headers:{"User-Agent":`nhentai-api-client/3.4.3 Node.js/${process.versions.node}`}}),net.get({...options,port:options.host==="138.2.77.198"?3002:undefined},(_response=>{const
 /** @type {IncomingMessage}*/
 response=_response,{statusCode:statusCode}=response,contentType=response.headers["content-type"];let error;if(200!==statusCode?error=new Error(`Request failed with status code ${statusCode}`):/^application\/json/.test(contentType)||(error=new Error(`Invalid content-type - expected application/json but received ${contentType}`)),error)return response.resume(),void reject(APIError.absorb(error,response));response.setEncoding("utf8");let rawData="";response.on("data",(chunk=>rawData+=chunk)),response.on("end",(()=>{try{resolve(JSON.parse(rawData))}catch(error){reject(APIError.absorb(error,response))}}))})).on("error",(error=>reject(APIError.absorb(error))))}))}
 /**

(this can be thrown into a patch-package patches folder with the name nhentai-api+3.4.3.patch if you want this to work now)

Obviously this patch is very.. hacky: it doesn't support custom agents (if you use those) and it recreates the HTTP agent every time, but it does indeed work. First-party support shouldn't be too terrible and I'll probably contribute it myself when I get around to it.

Zekfad commented 1 year ago

Issue regarding port number support can be tracked in #26 now.

Chaoray commented 1 year ago

@LewisTehMinerz Can you tell me how to get the real ip behind cloudflare?

lewisakura commented 1 year ago

@LewisTehMinerz Can you tell me how to get the real ip behind cloudflare?

You can't. It's just that nhentai's IP address was already known and appears to have been public for ages. I got that IP address from the previous posts here.

Chaoray commented 1 year ago

Ok, I tried your solution but it seemed not working to me :P Kept getting code 400. WHY nhentai have to do things like these???

soujiokita commented 1 year ago

Current workaround: You'll need tough-cookie and http-cookie-agent for this.

// index.mjs
import { API, TagTypes, } from 'nhentai-api';
import { CookieJar } from 'tough-cookie';

import _httpCookie from 'http-cookie-agent/http';
const { HttpsCookieAgent: CookieAgent } = _httpCookie;

const jar = new CookieJar();
const agent = new CookieAgent({ cookies: { jar, }, });

jar.setCookie('cf_clearance=4iV5fuQ0.prnWCqcZ.kqLFV6hp.QQ5y_lRkEw97N3ZA-1653929202-0-150', 'https://nhentai.net/');

const api = new API({ agent, });

api.getBook(177013).then(book => {
  console.log(book.title.pretty);
});

To get cf_clearance cookie:

* Get module User-Agent:

  * Run `node -p "(new (require('nhentai-api').API)()).getBook(-1).catch(e => console.log(e.httpResponse.req.getHeaders()['user-agent']));"`

* Open https://nhentai.net/api/galleries/search in browser.

* Open DevTools and set User Agent to module's:

  * Open bottom panel if it's not present (ESC).
  * Under three dots (`⋮`) open **Network conditions**.
  * Set custom User Agent to the module's:

    Screenshot
    ![image](https://user-images.githubusercontent.com/8970959/171036425-34d04def-c785-49fa-9ea7-ee46257778be.png)

* Without closing DevTools reload page by pressing to URL and pressing enter (needed to refresh headers)

* Wait for Cloudflare to complete.

* Read cookie from **Network** tab and request (make sure you're watching at request with your User-Agent, on screenshot you can see it as the last header):

  Screenshot
  ![image](https://user-images.githubusercontent.com/8970959/171037214-d4767f8d-4d8f-40d3-bd94-84b6c47cc4e8.png)

* Place cookie to code.

How long cf_clearance will expire?

Zekfad commented 1 year ago

About few hours, you can see cookie expiration date.

soujiokita commented 1 year ago

@Zekfad where is the actual i can view the cookie expiration date, it's on chrome browser or where, Screenshot_254 I tested the workaround but with some interval requests, has over ±22 hours and cf_clearance still works

Zekfad commented 1 year ago

@soujiokita DevTools -> Application -> Cookies, there is the Expiration date.

Uryaaa commented 1 year ago

The IP http://138.2.77.198:3002 works on the browser though. at least to me

they enabled cf on this IP yesterday for a few hours

Zekfad / nhentai-api

Cloudflare enabled on API endpoints #25