Sometimes it fails to bypass cloudflare

[NabiKAZ]

In the case of Chrome and Edge, sometimes (and not for always) it does not work for the particular site I was focusing on. But I did not encounter any problems with Firefox and Safari. This is the main sample site: https://pegaxy.io/ You can specifically focus on this address: https://api.pegaxy.io/my/info And I used it on Docker. You can send consecutive requests and test. By the way, I already asked about the details of what I want to do here: https://stackoverflow.com/q/71529199/1407491

Here is an example of a log that works:

SHOW LOG

``` # docker run --rm lwthiker/curl-impersonate:0.3-chrome curl_chrome99 'https://api.pegaxy.io/my/info' \ > -H 'authority: api.pegaxy.io' \ > -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"' \ > -H 'accept: application/json' \ > -H 'sec-ch-ua-platform: "Windows"' \ > -H 'sec-ch-ua-mobile: ?0' \ > -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36' \ > -H 'sec-ch-ua-platform: "Windows"' \ > -H 'origin: https://play.pegaxy.io' \ > -H 'sec-fetch-site: same-site' \ > -H 'sec-fetch-mode: cors' \ > -H 'sec-fetch-dest: empty' \ > -H 'referer: https://play.pegaxy.io/marketplace' \ > -H 'accept-language: en-US,en;q=0.9,fa;q=0.8,de;q=0.7' \ > --compressed -s -vv * Trying 172.67.10.157:443... * Connected to api.pegaxy.io (172.67.10.157) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-CHACHA20-POLY1305,ECDHE-RSA-CHACHA20-POLY1305,ECDHE-RSA-AES128-SHA,ECDHE-RSA-AES256-SHA,AES128-GCM-SHA256,AES256-GCM-SHA384,AES128-SHA,AES256-SHA * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: none * ALPS, offering h2 } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.2 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [19 bytes data] * TLSv1.3 (IN), TLS handshake, Unknown (25): { [3156 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [80 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.pegaxy.io * start date: Mar 3 05:22:24 2022 GMT * expire date: Jun 1 05:22:23 2022 GMT * subjectAltName: host "api.pegaxy.io" matched cert's "*.pegaxy.io" * issuer: C=US; O=Let's Encrypt; CN=E1 * SSL certificate verify ok. * Using HTTP2, server supports multiplexing * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 } [5 bytes data] * Using Stream ID: 1 (easy handle 0x7f0ad96c1a90) } [5 bytes data] > GET /my/info HTTP/2 > Host: api.pegaxy.io > sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99" > sec-ch-ua-mobile: ?0 > sec-ch-ua-platform: "Windows" > upgrade-insecure-requests: 1 > user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 > accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 > sec-fetch-site: none > sec-fetch-mode: navigate > sec-fetch-user: ?1 > sec-fetch-dest: document > accept-encoding: gzip, deflate, br > accept-language: en-US,en;q=0.9 > authority: api.pegaxy.io > sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99" > accept: application/json > sec-ch-ua-mobile: ?0 > user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 > sec-ch-ua-platform: "Windows" > origin: https://play.pegaxy.io > sec-fetch-site: same-site > sec-fetch-mode: cors > sec-fetch-dest: empty > referer: https://play.pegaxy.io/marketplace > accept-language: en-US,en;q=0.9,fa;q=0.8,de;q=0.7 > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [222 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [222 bytes data] * old SSL session ID is stale, removing { [5 bytes data] * Connection state changed (MAX_CONCURRENT_STREAMS == 256)! } [5 bytes data] < HTTP/2 200 < date: Mon, 21 Mar 2022 23:22:10 GMT < content-type: application/json; charset=utf-8 < vary: Accept-Encoding < vary: Origin < access-control-allow-origin: https://play.pegaxy.io < etag: W/"29-bVHj0ypH/h4ZX9esOoZbwspQiQY" < x-frame-options: SAMEORIGIN < x-xss-protection: 1; mode=block < x-content-type-options: nosniff < referrer-policy: no-referrer-when-downgrade < content-security-policy: default-src 'self' http: https: data: blob: 'unsafe-inline' < strict-transport-security: max-age=31536000; includeSubDomains < cf-cache-status: DYNAMIC < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" < set-cookie: __cf_bm=5alOaylVbD3I6nERRUHj3COnrAUKDaLA0KIZt9bE8ww-1647904930-0-AWqZ1lWXsTPZlGfyqlMjEtpAUJK8Qjbj8rDlMh2vHuuLEIWEH9vFrwcc/lCm4WBFkT/MMs68cv04GOiBTvbTALw=; path=/; expires=Mon, 21-Mar-22 23:52:10 GMT; domain=.pegaxy.io; HttpOnly; Secure; SameSite=None < server: cloudflare < cf-ray: 6efa6d987ed09b71-FRA < content-encoding: br < { [45 bytes data] * Connection #0 to host api.pegaxy.io left intact {"status":false,"error":"USER_NOT_FOUND"} ```

And here is a sample log for when it does not work:

SHOW LOG

``` # docker run --rm lwthiker/curl-impersonate:0.3-chrome curl_chrome99 'https://api.pegaxy.io/my/info' \ > -H 'authority: api.pegaxy.io' \ > -H 'sec-ch-ua-mobile: ?0' \ > -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"' \ > -H 'accept: application/json' \ > -H 'sec-ch-ua-mobile: ?0' \ > -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36' \ > -H 'sec-ch-ua-platform: "Windows"' \ > -H 'origin: https://play.pegaxy.io' \ > -H 'sec-fetch-site: same-site' \ > -H 'sec-fetch-mode: cors' \ > -H 'sec-fetch-dest: empty' \ > -H 'referer: https://play.pegaxy.io/marketplace' \ > -H 'accept-language: en-US,en;q=0.9,fa;q=0.8,de;q=0.7' \ > --compressed -s -vv * Trying 172.67.10.157:443... * Connected to api.pegaxy.io (172.67.10.157) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-CHACHA20-POLY1305,ECDHE-RSA-CHACHA20-POLY1305,ECDHE-RSA-AES128-SHA,ECDHE-RSA-AES256-SHA,AES128-GCM-SHA256,AES256-GCM-SHA384,AES128-SHA,AES256-SHA * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: none * ALPS, offering h2 } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.2 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [19 bytes data] * TLSv1.3 (IN), TLS handshake, Unknown (25): { [3156 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [78 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.pegaxy.io * start date: Mar 3 05:22:24 2022 GMT * expire date: Jun 1 05:22:23 2022 GMT * subjectAltName: host "api.pegaxy.io" matched cert's "*.pegaxy.io" * issuer: C=US; O=Let's Encrypt; CN=E1 * SSL certificate verify ok. * Using HTTP2, server supports multiplexing * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 } [5 bytes data] * Using Stream ID: 1 (easy handle 0x7f396fbbda90) } [5 bytes data] > GET /my/info HTTP/2 > Host: api.pegaxy.io > sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99" > sec-ch-ua-mobile: ?0 > sec-ch-ua-platform: "Windows" > upgrade-insecure-requests: 1 > user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 > accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 > sec-fetch-site: none > sec-fetch-mode: navigate > sec-fetch-user: ?1 > sec-fetch-dest: document > accept-encoding: gzip, deflate, br > accept-language: en-US,en;q=0.9 > authority: api.pegaxy.io > sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99" > accept: application/json > sec-ch-ua-mobile: ?0 > user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 > sec-ch-ua-platform: "Windows" > origin: https://play.pegaxy.io > sec-fetch-site: same-site > sec-fetch-mode: cors > sec-fetch-dest: empty > referer: https://play.pegaxy.io/marketplace > accept-language: en-US,en;q=0.9,fa;q=0.8,de;q=0.7 > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [222 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [222 bytes data] * old SSL session ID is stale, removing { [5 bytes data] * Connection state changed (MAX_CONCURRENT_STREAMS == 256)! } [5 bytes data] < HTTP/2 403 < date: Mon, 21 Mar 2022 17:14:41 GMT < content-type: text/html; charset=UTF-8 < cache-control: max-age=15 < expires: Mon, 21 Mar 2022 17:14:56 GMT < x-frame-options: SAMEORIGIN < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" < set-cookie: __cf_bm=nEnD.QUz3L43TqTxUdnxUWV7R3svGpN9CQ8MU3thu88-1647882881-0-AT/Vw/Y/DoqdLAESxkrplf95mmnU269etAJ8DpG5l//9sJ3+zDd8fC5iTyhD5x7trGkAsWonR5ErB3lSN+RuLvg=; path=/; expires=Mon, 21-Mar-22 17:44:41 GMT; domain=.pegaxy.io; HttpOnly; Secure; SameSite=None < vary: Accept-Encoding < server: cloudflare < cf-ray: 6ef8534949b69b3f-FRA < content-encoding: br < { [922 bytes data] * Connection #0 to host api.pegaxy.io left intact

Please enable cookies.

Sorry, you have been blocked

You are unable to access pegaxy.io

Why have I been blocked?

This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.

What can I do to resolve this?

You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.

Cloudflare Ray ID: 6ef8534949b69b3f • Your IP: xx.xx.xx.xx • Performance & security by Cloudflare

```

Thanks for great project.

[lwthiker]

Hi,

When you use the wrapper scripts (e.g. curl_chrome99) all the default Chrome headers are already added for you. You can look at the script: it is a simple wrapper around the curl-impersonate binary that sets the ciphers, HTTP headers and a few other options. They way you have used the script causes the headers to be added twice, which might be the reason that Cloudflare blocks you.

Try simply running:

curl_chrome99 'https://api.pegaxy.io/my/info'

If you need to change any of the hard-coded headers, the way to go should be modifying the script.

Let me know if that works so I know whether I can close this issue.

[NabiKAZ]

Thanks. No problem right now. If the problem is repeated, I will leave the message again.

[ElPapi89]

A.Z.,

Im trying to acess pegaxy api and its impossibl. i tried add headers e sessio, but nothing. can u help me? i saw ur post about the that.

import requests
import json

url = "https://api.pegaxy.io/rent/0?rentMode=SHARE_PROFIT"

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' ,
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' ,
'Accept-Language': 'en-US,en;q=0.5' ,
'Connection': 'keep-alive' ,
'Upgrade-Insecure-Requests': '1' ,
'Sec-Fetch-Dest': 'document' ,
'Sec-Fetch-Mode': 'navigate' ,
'Sec-Fetch-Site': 'none' ,
'Sec-Fetch-User': '?1', 
'Pragma': 'no-cache' ,
'Cache-Control': 'no-cache'
}

response = requests.get(url, headers=headers)
response.raise_for_status()  # raises exception when not a 2xx response
if response.status_code != 204:
   return response.json()
response

Im getting this error

HTTPError: 403 Client Error: Forbidden for url: https://api.pegaxy.io/rent/0?rentMode=SHARE_PROFIT

Thanks

[NabiKAZ]

@ElPapi89 I do not see anything of this project and curl_chrome99 command in your code. If you plan to run it in Python. You must first compile this project or launch Docker for it and make sure that the following command on the command line gives the correct result:

curl_chrome99 'https://api.pegaxy.io/rent/0?rentMode=SHARE_PROFIT' or docker run --rm lwthiker/curl-impersonate:0.4-chrome curl_chrome99 https://api.pegaxy.io/rent/0?rentMode=SHARE_PROFIT

Then, with the help of shell command methods, call it in Python. For example, something like this:

import subprocess
p = subprocess.Popen("curl_chrome99 https://api.pegaxy.io/rent/0?rentMode=SHARE_PROFIT", stdout=subprocess.PIPE, shell=True)
print(p.communicate())

Or other methods mentioned in this article, for example: https://www.the-analytics.club/python-shell-commands