Rakuten.com detects headless chrome 7 out of 10 times

[someotheraccname]

Hi,

rakuten.com detects headless Chrome even with the evasions of this repo applied. Example: https://www.rakuten.com/category/443/?p=2

The first access sometimes works and sometimes doesnt. If you try to access a few more pages (parameter p=X) after that, access is almost always blocked. It makes no difference if cookies/local storage are deleted between those accesses.

If run with flag headless: false access is always granted.

I created a small test project to illustrate this problem https://github.com/Rsmwe/Headless-detected-demo

Hope this helps. Thanks.

[antoinevastel]

Hi,

Thanks for the information. I visited the website with a vanilla Chrome, and a headless Chrome without evasion technique. Here is what I obtained:

Normal chrome

ap:true
bt:{"charging":true,"chargingTime":0,"dischargingTime":"Infinity","level":1,"onchargingchange":null,"onchargingtimechange":null,"ondischargingtimechange":null,"onlevelchange":null}
fonts:33,57,65
fh:e83989f0163c5647fb6328102daf3586f7975b84
timing:{"1":95,"2":534,"3":636,"profile":{"bp":1,"sr":15,"dp":1,"lt":0,"ps":0,"cv":19,"fp":0,"sp":1,"br":0,"ieps":1,"av":0,"z1":54,"jsv":1,"nav":0,"z2":2,"z3":2,"fonts":14},"main":622,"compute":94,"send":650}
bp:1038350511,-1979380391,1738406762,749224105,-23072613
sr:{"inner":[664,851],"outer":[1707,942],"screen":[0,0],"pageOffset":[0,0],"avail":[1706,941],"size":[1706,960],"client":[649,837],"colorDepth":24,"pixelDepth":24}
dp:{"XDomainRequest":0,"createPopup":0,"removeEventListener":1,"globalStorage":0,"openDatabase":1,"indexedDB":1,"attachEvent":0,"ActiveXObject":0,"dispatchEvent":1,"addBehavior":0,"addEventListener":1,"detachEvent":0,"fireEvent":0,"MutationObserver":1,"HTMLMenuItemElement":0,"Int8Array":1,"postMessage":1,"querySelector":1,"getElementsByClassName":1,"images":1,"compatMode":"CSS1Compat","documentMode":0,"all":1,"now":1,"contextMenu":0}
lt:1517913304130+1
ps:true,true
cv:53bc4dcc1d03d1bd1182bcbc71f7626b392ae39c
fp:false
sp:false
br:Chrome
ieps:false
av:false
z:{"a":1565723222,"b":1,"c":0}
zh:
jsv:1.7
nav:{"userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36","appName":"Netscape","appCodeName":"Mozilla","appVersion":"5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36","appMinorVersion":0,"product":"Gecko","productSub":"20030107","vendor":"Google Inc.","vendorSub":"","buildID":0,"platform":"Linux x86_64","oscpu":0,"hardwareConcurrency":4,"language":"en-US","languages":["fr-FR","fr","en-US","en"],"systemLanguage":0,"userLanguage":0,"doNotTrack":null,"msDoNotTrack":0,"cookieEnabled":true,"geolocation":1,"vibrate":1,"maxTouchPoints":0,"webdriver":0,"plugins":["Chrome PDF Plugin","Chrome PDF Viewer","Native Client","Widevine Content Decryption Module"]}
t:f1c0916bdb540f805665eb9be44333424ee13e04
u:68b42305f9845ab49d1fb0c1c3079808
fc:true

Chrome headless

ap:true
bt:{"charging":true,"chargingTime":0,"dischargingTime":"Infinity","level":1,"onchargingchange":null,"onchargingtimechange":null,"ondischargingtimechange":null,"onlevelchange":null}
fonts:33,57,65
fh:e83989f0163c5647fb6328102daf3586f7975b84
timing:{"1":61,"2":323,"3":542,"4":679,"profile":{"bp":0,"sr":1,"dp":0,"lt":0,"ps":0,"cv":20,"fp":0,"sp":1,"br":0,"ieps":0,"av":1,"z1":35,"jsv":1,"nav":0,"z2":2,"z3":2,"z4":1,"fonts":16},"main":626,"compute":61,"send":696}
bp:
sr:{"inner":[1920,1080],"outer":[1920,1080],"screen":[0,0],"pageOffset":[0,0],"avail":[1920,1080],"size":[1920,1080],"client":[1905,1080],"colorDepth":24,"pixelDepth":24}
dp:{"XDomainRequest":0,"createPopup":0,"removeEventListener":1,"globalStorage":0,"openDatabase":1,"indexedDB":1,"attachEvent":0,"ActiveXObject":0,"dispatchEvent":1,"addBehavior":0,"addEventListener":1,"detachEvent":0,"fireEvent":0,"MutationObserver":1,"HTMLMenuItemElement":0,"Int8Array":1,"postMessage":1,"querySelector":1,"getElementsByClassName":1,"images":1,"compatMode":"CSS1Compat","documentMode":0,"all":1,"now":1,"contextMenu":0}
lt:1517913573363+1
ps:true,true
cv:b0f4b8fb75ab6b8f88a41e310c7a8285010da36b
fp:false
sp:false
br:
ieps:false
av:false
z:{"a":1565723222,"b":1,"c":0}
zh:
jsv:1.7
nav:{"userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36","appName":"Netscape","appCodeName":"Mozilla","appVersion":"5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36","appMinorVersion":0,"product":"Gecko","productSub":"20030107","vendor":"Google Inc.","vendorSub":"","buildID":0,"platform":"Linux x86_64","oscpu":0,"hardwareConcurrency":4,"language":"en-US","languages":[],"systemLanguage":0,"userLanguage":0,"doNotTrack":null,"msDoNotTrack":0,"cookieEnabled":true,"geolocation":1,"vibrate":1,"maxTouchPoints":1,"webdriver":true,"plugins":[]}
t:f1c0916bdb540f805665eb9be44333424ee13e04
u:68b42305f9845ab49d1fb0c1c3079808
fc:true

Differences

There are differences in the following attributes:

Timing

-### Normal Chrome
+### Headless Chrome
-timing:{"1":95,"2":534,"3":636,"profile":{"bp":1,"sr":15,"dp":1,"lt":0,"ps":0,"cv":19,"fp":0,"sp":1,"br":0,"ieps":1,"av":0,"z1":54,"jsv":1,"nav":0,"z2":2,"z3":2,"fonts":14},"main":622,"compute":94,"send":650}
+timing:{"1":61,"2":323,"3":542,"4":679,"profile":{"bp":0,"sr":1,"dp":0,"lt":0,"ps":0,"cv":20,"fp":0,"sp":1,"br":0,"ieps":0,"av":1,"z1":35,"jsv":1,"nav":0,"z2":2,"z3":2,"z4":1,"fonts":16},"main":626,"compute":61,"send":696}

Nav

-### Normal Chrome
+### Headless Chrome

-nav:{"userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36","appName":"Netscape","appCodeName":"Mozilla","appVersion":"5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36","appMinorVersion":0,"product":"Gecko","productSub":"20030107","vendor":"Google Inc.","vendorSub":"","buildID":0,"platform":"Linux x86_64","oscpu":0,"hardwareConcurrency":4,"language":"en-US","languages":["fr-FR","fr","en-US","en"],"systemLanguage":0,"userLanguage":0,"doNotTrack":null,"msDoNotTrack":0,"cookieEnabled":true,"geolocation":1,"vibrate":1,"maxTouchPoints":0,"webdriver":0,"plugins":["Chrome PDF Plugin","Chrome PDF Viewer","Native Client","Widevine Content Decryption Module"]}
+nav:{"userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36","appName":"Netscape","appCodeName":"Mozilla","appVersion":"5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36","appMinorVersion":0,"product":"Gecko","productSub":"20030107","vendor":"Google Inc.","vendorSub":"","buildID":0,"platform":"Linux x86_64","oscpu":0,"hardwareConcurrency":4,"language":"en-US","languages":[],"systemLanguage":0,"userLanguage":0,"doNotTrack":null,"msDoNotTrack":0,"cookieEnabled":true,"geolocation":1,"vibrate":1,"maxTouchPoints":1,"webdriver":true,"plugins":[]}

Cv

This seems to be a canvas that looks like this:

Normal Chrome

cv:53bc4dcc1d03d1bd1182bcbc71f7626b392ae39c

Headless Chrome

cv:b0f4b8fb75ab6b8f88a41e310c7a8285010da36b

Lt:

Normal Chrome

lt:1517913304130+1

Headless Chrome

lt:1517913573363+1

Br:

Normal Chrome

br:Chrome

Headless Chrome

br:

[someotheraccname]

Hi,

where did you find the canvas element ?

I assume the attributes you mentioned are generated here:

https://www.rakuten.com/akam/10/6f922ba4 (http://jsnice.org/ helps to deobfuscate)
https://www.rakuten.com/akam/10/pixel_6f922ba4

Generally speaking, could there be some kind of hmac request signing going on?

[antoinevastel]

I deobfuscated the script a little bit and ran it with few breaking points to look at the different variables. Indeed I obtained the attribute from the request sent in your second link. Concerning your last question, I don't have much time this week, but I'll look at it with more attention next week.

[antoinevastel]

I gave a look at the attributes and launched your script. There are few differences compared to when I launched directly Chrome headless without Pupeteer last time. Here are few remarks: With your modifications, in the attribute sr you have the subattribute outer, which is less than inner. This is the kind of things that can be easily detected. Moreover, you have no battery object with Pupeteer. Finally, the attribute u is different whereas it was not the case before. However, it seems to be equal to a variable hardcoded in the script, which is maybe generated dynamically server side : var last = "68b42305...c1c3079808";

If you want to execute the code locally, you need to create a global variable, i.e. in window, called bazadebezolkohpepadr. Otherwise the script won't execute until the end.

[zfLQ2qx2]

@antoinevastel Post your deobfuscated script in a gist? Looks like the urls above are no longer valid, I'm curious what they were doing to detect headless.

[antoinevastel]

I don't think I can post the deobfuscated script because of copyright. However, I can show the result of the script when it is executed with the modified version of Chrome headless that applies evasion techniques. The script collects the object presented in the image below. I'll try to explain what these attributes means.

[zfLQ2qx2]

@antoinevastel If you find the script again, maybe post the updated url? Looks like they are just scoring the presence/absence of features, trying to make a strong enough signal. Screen size is a def giveaway, max touch points 1, no plugins w/o mention of a mobile browser, I think the whole canvas test is to verify that the browser is actually rasterizing an image, that may be the point of the timing also - I’m guessing a headless browser will probably skip a lot of rasterizing until it has to produce a snapshot.

Did you find out for sure what is tripping it with mocking enabled?

The other day I found someone who was counting the number of times the mouse moved less then a hundred pixels within 500ms, and assuming it was a headless browser if it was less then N.

[someotheraccname]

Hi,

the headless detection is still deployed, the script just moved to:

https://www.rakuten.com/akam/10/5929aead https://www.rakuten.com/akam/10/pixel_5929aead

If the location changes again, you can look it up via Chrome->Sources.

_Edit: Location changed again to:_

https://www.rakuten.com/akam/10/3ae02e24 https://www.rakuten.com/akam/10/pixel_3ae02e24

MD5: 8E1AFD48FD0CD2C2B0D8D527B498BB06

[zfLQ2qx2]

I found someone who has already done a bit of work decoding this script:

https://gist.github.com/ttilberg/c23f39318f5efacdd3ec45f3e1b19ad4

[zfLQ2qx2]

It looks to me as this script is just collecting data about the presence of browsing features in very convoluted and round-about ways, I think the actual decision about rather to allow the client or not is happening someplace else, so we won't know what their trigger is.

[cyphercodes]

Were you able to solve this? @Rsmwe @zfLQ2qx2

[crimsonsoccer55]

Any updates on getting around this?

[MannyPamintuanWorkAccount]

If you want to execute the code locally, you need to create a global variable, i.e. in window, called bazadebezolkohpepadr. Otherwise the script won't execute until the end.

Would anyone kindly provide additional context as to what the variable bazadebezolkohpepadr represents? I'm seeing it in a lot of HTML on github, and with some online searching have not concluded what it is and its source.

cc: @antoinevastel

[Buggem]

If you want to execute the code locally, you need to create a global variable, i.e. in window, called bazadebezolkohpepadr. Otherwise the script won't execute until the end.

Would anyone kindly provide additional context as to what the variable bazadebezolkohpepadr represents? I'm seeing it in a lot of HTML on github, and with some online searching have not concluded what it is and its source.

cc: @antoinevastel

To this day, the meaning is unknown... I just learnt about bazadebezolkohpepadr and I have no idea what the hell it is