search

milahu / aiohttp_chromium

aiohttp-like interface to chromium. based on selenium_driverless to bypass cloudflare
MIT License
46 stars 7 forks source link

What is the motivation behind this? #3

Closed yb-cs closed 9 months ago

yb-cs commented 9 months ago

What exactly are you trying to achieve by this project? are you trying to bypass the cloudflare non-interactive challenge or what exactly?

milahu commented 9 months ago

are you trying to bypass the cloudflare non-interactive challenge

yes. my goal is to make the http client look like a normal browser to bypass blocking based on browser fingerprint

this problem is already solved by selenium_driverless but i dont like the API of selenium_driverless especially file downloads have too much boilerplate code see also readme#why and https://github.com/kaliiiiiiiiii/Selenium-Driverless/issues/140

my code is too high-level to upstream it to selenium_driverless because it is based on CDP events like Network.responseReceived so this could interfere with user-defined CDP event listeners

i need this for my opensubtitles-scraper when i send requests with aiohttp then i get blocked even with a VIP account

yb-cs commented 9 months ago

Can you tell me more about the specific request you are trying to do?

yb-cs commented 9 months ago

Because you might not even need a browser for doing this....

yb-cs commented 9 months ago

Also look at this if you insist on doing this approach https://github.com/makindotcc/FlarelessHeadlessChrome

I have not tested it but looks useful in your case.

milahu commented 9 months ago

https://github.com/makindotcc/FlarelessHeadlessChrome

binary-patching chromium sounds fun ^^ i guess they use chromedriver, but chromedriver is slower than CDP

Migrating Selenium system tests to Cuprite

we migrated our test suite to the Selenium / Webdriver wrapper around Chrome browser around ~2018.

a new ruby testing driver approach is being developed. It is called Cuprite, it runs the Ferrum library under the hood which, in turn, is an API that directly instruments the Chrome browser using the Chrome DevTools Protocol (CDP)

We are indeed very happy about the migration to Cuprite. Our tests are much faster, the API to handle them is simpler

ferrum via awesome-chrome-devtools

Ferrum connects to the browser by CDP protocol and there's no Selenium/WebDriver/ChromeDriver dependency. The emphasis was made on a raw CDP protocol because Chrome allows you to do so many things that are barely supported by WebDriver because it should have consistent design with other browsers.

im not sure what your goal is here : P aiohttp_chromium works for me, and i wont change a running system...

yb-cs commented 9 months ago

im not sure what your goal is here : P

Well... just trying to help out and figure what exactly you need.

milahu commented 9 months ago

mmkay. closing