search

zfcsoftware / cf-clearance-scraper

This package provides an api that returns cookies (cf-clearance) that you can request on a website protected by Cloudflare WAF (corporate or normal) without being blocked by WAF.
MIT License
91 stars 7 forks source link

It does not work with proxy with auth. #7

Closed cod888 closed 1 week ago

cod888 commented 1 week ago

I tried to use proxies with an auth username/password, and it displays 'HTTP ERROR 407' error when opening a URL. Proxies are working btw.

body:

{
  "url": "https://nopecha.com/demo/cloudflare",
  "mode": "waf",
  "proxy": {
    "host": "someip",
    "port": "someport",
    "username": "someusername",
    "password": "somepassword"
  }
}
zfcsoftware commented 1 week ago

Proxies with auth information were used while coding the library. It has been tested many times. There is probably a problem with your proxies. Can you register with Webshare and try with the test proxy information it provides?

cod888 commented 1 week ago

I use the proxies (oculusproxies) in my current project, and they are working, so I'm using them here. However, I tried others in my pool, and some of them were working. I don't understand why not all of them are working here. Also, did you see such behavior when a page is not loaded fully (all CSS, images, etc) with your tool? I tried opening a page with the same proxy through your tool and directly in a browser, and the result page looks different.

zfcsoftware commented 1 week ago

I use the proxies (oculusproxies) in my current project, and they are working, so I'm using them here. However, I tried others in my pool, and some of them were working. I don't understand why not all of them are working here. Also, did you see such behavior when a page is not loaded fully (all CSS, images, etc) with your tool? I tried opening a page with the same proxy through your tool and directly in a browser, and the result page looks different.

There may be a momentary situation related to your proxies, puppeteer's local proxy support is used. Images, fonts, css etc. are blocked so that they do not consume resources. You can try using different proxies, this problem is not library related.

cod888 commented 1 week ago

Understood about resources. Thanks for your explanation. Looks like not all proxies are working with this solution. sad

cod888 commented 1 week ago

there was an issue with my proxies. the lib is cool. 👍