Akamai Detecting when trying to Login

[smartahsan]

Hi,

A website I use your chromedriver for has recently changed there Akamai protection. Now when I try to login they block my login request. Is there anyway to fix this or is this something you could look into?

[daybyter]

Same here. What irritates me: when you open a browser window by script, but then type the URL by hand, the site is still blocked.

If you open the browser by hand, the site works here.

So it must be something that is passed to browser when it is opened.

[smartahsan]

Ya, im not sure what of fix would work I have tried user agents but to no avail. Akamai is really stepping up there game.

[daybyter]

My problem is cloudflare. But same issue here.

[daybyter]

I checked the data further and maybe I have an idea. Need to do further tests.

[ultrafunkamsterdam]

If you use v2, you can even close the driver while cloudflare is in it's 5s detection loop and will instantly be back to "human" mode.

Does someone have an akamai protected example url for me to analyze?

[daybyter]

I only have a cf site.. http://tutti.ch

[ultrafunkamsterdam]

I only have a cf site.. http://tutti.ch

I need akamai example. I got my own cf bot-fight-mode-enabled site to test with.

[rsbotmoo]

Currently having the same issue for Cloudflare. Worked until 3-4 days ago.

[LandyLane]

Also no longer working on Incapsula. Im wondering if its due to a chrome update? On version 90+ atm.

[david-jng]

https://www.nike.com uses the newest Akamai version.

[ultrafunkamsterdam]

Live from bathroom so sorry for quality

I think you are trying to access some login page, DIRECTLY, which akamai successfully blocks because no previous cookies. Try to enter main website before the login URLs and so on.

[david-jng]

No, cookies are not the point. I successfully figured out that I am able to login if I start Chrome from within the command line with a fresh chrome profile. The same procedure using Selenium returns an error. Same problem with Puppeteer and Playwright. I guess those browser automation tools change something in the browser which get's detected by Akamai.

Edit: Opening the page is not blocked by Akamai. It's the login / registration.

[ultrafunkamsterdam]

What you describe is exactly what v2 does....

[ultrafunkamsterdam]

I fully agree browser behavior changes when debug/automation is connected although google says it doesn't 🤣

[david-jng]

Yeah it definitely changes something within the browser. Akamai knows it - we (or I? ;)) don't...

[ultrafunkamsterdam]

I did not have problem with akamai. I can't determine your workflows if you dont post full examples with code and errors of what you are doing

[rsbotmoo]

@ultrafunkamsterdam I am having issues with Incapsula, worked until 5 days ago +- Test URL: https://secure.runescape.com/m=account-creation/create_account?theme=oldschool

Result without Selenium:

Result with Selenium (also with V2)

[smartahsan]

I did not have problem with akamai. I can't determine your workflows if you dont post full examples with code and errors of what you are doing

ok so heres an example https://www.nike.com/ca/launch Try logging into an account through this page with the driver. They will block your login. This has just recently happened after heightened Akamai security.

[ultrafunkamsterdam]

Yeah it definitely changes something within the browser. Akamai knows it - we (or I? ;)) don't...

It has all to do with timings.

Well if i wasnt too busy answering all those issues which are raised by people who have to either need to learn python, read selenium docs, search google, and people having individual problems (and not sending any code, errors etc), for some random site they think they can abuse for free coupons, game cash and whatnot other cheapass stuff, then we'd never know.

luckiliy i still have cleartext botprotect.io code which i definitely need to read and test. It has the "Secret" thing. But it's just A FREAKING LOT.

And regarding the difference in browser when automating or not. it's even instant. when disconnecting driver it proceeds easily and breakout out of the cf-loop, after which we can reconnect.

https://user-images.githubusercontent.com/21027969/116285727-b5124480-a78e-11eb-811b-4a0f06bf74dc.mp4

[aleixrodriala]

Yeah it definitely changes something within the browser. Akamai knows it - we (or I? ;)) don't...

It has all to do with timings.

Well if i wasnt too busy answering all those issues which are raised by people who have to either need to learn python, read selenium docs, search google, and people having individual problems (and not sending any code, errors etc), for some random site they think they can abuse for free coupons, game cash and whatnot other cheapass stuff, then we'd never know.

luckiliy i still have cleartext botprotect.io code which i definitely need to read and test. It has the "Secret" thing. But it's just A FREAKING LOT.

And regarding the difference in browser when automating or not. it's even instant. when disconnecting driver it proceeds easily and breakout out of the cf-loop, after which we can reconnect.

final_608806008c74730104d39fc8_252044.2.mp4

Do you have Discord or similar mate? I would love to check that botprotect code and try to help, I have been working on bot detection also for quite a while.

Cheers for your work

[daybyter]

From my subjective pov it seems to me that at least the first check takes longer with webdriver control.

Is there a way to actually measure this? Maybe compare the timestamps of requests between human and webdriver control?

If there is actually a difference, what causes it? Additional calculations in the challenge? Additional requests?

The server does not know the performance of the client, so the timing cannot be the only criteria, since you cannot compare it to anything? Or could you do 2 calculations? One that is affected by the webdriver and one that is not, so you can compare the two?

Pardon my naive questions that won't be very helpful, I guess... :-(

[aleixrodriala]

After hours of research, for some reason this works:

use chrome with this extra arg: "--disable-popup-blocking"

driver = uc.Chrome()
driver.execute_script("window.open('https://coinfaucet.eu');")
driver.switch_to.window(driver.window_handles[1])
sleep(5)
driver.save_screenshot('lol.png')

I have also tried to open a new tab and then use the driver.get but only works like 70% of the times, but still works some times.

It's weird because as @ultrafunkamsterdam showed on his video, they should still be able to detect that the remote debugger is attached, but maybe fails because they are checking the other tab, but if that was the case, is it that when you do driver.get the debugger gets attached to the new window?

PD: I have been doing some more tests and if you try to get the source page or do anything in the page before passing the CF challenge on the new tab, the challenge won't pass, so could be that debugger it's not attached to new tab untill you do something with it, interesting at least, maybe someone can follow up.

[daybyter]

Once you passed the challenge you should have 2 cookies set in your session?

Could you add those cookies to your webdriver session before fetching the website?

[ultrafunkamsterdam]

Once you passed the challenge you should have 2 cookies set in your session?

Could you add those cookies to your webdriver session before fetching the website?

Yes that should be possible using some clever javascripting, or --still working on that-- being able reusing the same profile folder using options.add_argument('--user-data-dir=path/to/folder')

EDIT: cloudflare stores this information per session (browser window). so the above will not solve that particular problem.

After hours of research, for some reason this works:

use chrome with this extra arg: "--disable-popup-blocking"

driver = uc.Chrome()
driver.execute_script("window.open('https://coinfaucet.eu');")
driver.switch_to.window(driver.window_handles[1])
sleep(5)
driver.save_screenshot('lol.png')

I have also tried to open a new tab and then use the driver.get but only works like 70% of the times, but still works some times.

It's weird because as @ultrafunkamsterdam showed on his video, they should still be able to detect that the remote debugger is attached, but maybe fails because they are checking the other tab, but if that was the case, is it that when you do driver.get the debugger gets attached to the new window?

PD: I have been doing some more tests and if you try to get the source page or do anything in the page before passing the CF challenge on the new tab, the challenge won't pass, so could be that debugger it's not attached to new tab untill you do something with it, interesting at least, maybe someone can follow up.

👍

Actually that is kind of how v2 already does it :) you could have saved some work. below snippet copy pastable as is

import undetected_chromedriver.v2 as uc
driver = uc.Chrome()
with driver:
    driver.get('https://coinfaucet.eu')
driver.save_screenshot('lol.png')

[aleixrodriala]

Once you passed the challenge you should have 2 cookies set in your session? Could you add those cookies to your webdriver session before fetching the website?

Yes that should be possible using some clever javascripting, or --still working on that-- being able reusing the same profile folder using options.add_argument('--user-data-dir=path/to/folder')

EDIT: cloudflare stores this information per session (browser window). so the above will not solve that particular problem.

After hours of research, for some reason this works: use chrome with this extra arg: "--disable-popup-blocking"

driver = uc.Chrome()
driver.execute_script("window.open('https://coinfaucet.eu');")
driver.switch_to.window(driver.window_handles[1])
sleep(5)
driver.save_screenshot('lol.png')

I have also tried to open a new tab and then use the driver.get but only works like 70% of the times, but still works some times. It's weird because as @ultrafunkamsterdam showed on his video, they should still be able to detect that the remote debugger is attached, but maybe fails because they are checking the other tab, but if that was the case, is it that when you do driver.get the debugger gets attached to the new window? PD: I have been doing some more tests and if you try to get the source page or do anything in the page before passing the CF challenge on the new tab, the challenge won't pass, so could be that debugger it's not attached to new tab untill you do something with it, interesting at least, maybe someone can follow up.

👍

Actually that is kind of how v2 already does it :) you could have saved some work. below snippet copy pastable as is

import undetected_chromedriver.v2 as uc
driver = uc.Chrome()
with driver:
    driver.get('https://coinfaucet.eu')
driver.save_screenshot('lol.png')

Yes I know that v2 disables the service, but I found interesting that opening a new tab without having to close or restart any session works.

And also what was interesting was that if I requested page_source or done something on the new tab, then the CF would fail, my idea was not to solve CF but find why they are detecting it. Maybe there is a way to check what changes on a new tab when requesting page_source or doing anything with selenium.

[daybyter]

You showed, that disconnecting the webdriver during the forwarding loop is enough to make the page work.

So in theory there must be a difference between the request before the disconnect and the following request, when the webdriver is already disconnected. Could a compare between those 2 requests help?

It could be the timing also, but I guess that would be very unreliable, since the PC of the client might be slowed down by some application or whatever.

So I guess th data must be different?

[david-jng]

Since about 2 days I'm facing a login error on Nike.

Code:

import undetected_chromedriver.v2 as uc
chrome_options = uc.ChromeOptions()
driver = uc.Chrome(options=chrome_options)

Manually navigate to Nike and try to login. The following error occurs.

My IP is not banned, because I am able to login manually without starting Chrome using undetected_chromedriver.

Example login page: https://www.nike.com/launch?s=upcoming

[david-jng]

@ultrafunkamsterdam Could you give us a short update about this issue? It's also happening in V3.

[githeping]

Yeah it definitely changes something within the browser. Akamai knows it - we (or I? ;)) don't...

It has all to do with timings.

Well if i wasnt too busy answering all those issues which are raised by people who have to either need to learn python, read selenium docs, search google, and people having individual problems (and not sending any code, errors etc), for some random site they think they can abuse for free coupons, game cash and whatnot other cheapass stuff, then we'd never know.

luckiliy i still have cleartext botprotect.io code which i definitely need to read and test. It has the "Secret" thing. But it's just A FREAKING LOT.

And regarding the difference in browser when automating or not. it's even instant. when disconnecting driver it proceeds easily and breakout out of the cf-loop, after which we can reconnect.

final_608806008c74730104d39fc8_252044.2.mp4

It can not reconnect？？？

[5326987143287]

I heard that akamai SRC was leaked, not sure if thats true.