[ToDo] Bypass poe.trades cloudflare protection

[Eruyome]

It should be possible to bypass this, here is a reference: https://autohotkey.com/board/topic/111944-winhttprequest-and-ddos-protection/

And here is a script that I made to test/develop this, it's just the current function to make the winhttp request to poe.trade: https://dl.dropboxusercontent.com/u/13620316/bypassCloudFlare.ahk

Any help appreciated.

Ideas:

• Reverse engineer the cloudflare javscript to create the GET request used to delegate the browser to poe.trade, if it's necessary to execure javascript
• Embed webbrowser in a hidden GUI (Reference), find a way to get the actual search url without having to bypass cloudflare or open poe.trade in this browser and automate filling the form, then sending it and download the page contents (no idea if actually possible).

[DoctorVanGogh]

I would strongly advise against trying to circumvent any kind of ddos protection. Ultimately you're picking a fight with someone who's been in a cat and mouse game for years (and mostly winning).

Suggestions: Try to get xyz to offer some formalized interface for automated queries. Could probably be integrated into the personalized shop managemant system available at http://currency.poe.trade/shop

Something like an api key per shop account and you would set this key in the local script settings.

[ghost]

Poe.trade really needs to just release an API that outputs search results as json so you can use that data in an app.

[DoctorVanGogh]

That's the parsing/output side - let's try to get the access/input side working again first ;)

[s1monxyz]

Have you tried to reach out to Xyz on his reddit account? I'm sure he'll help you make it work.

[rbarbosatoro]

So the problem is that the "Security" they have just make imposible to manually send an HTTPRequest with AHK. As tank said, you will need to pass a fully writed form via POST, the problem is that Cloudflare redirects you before you can access the source of that webpage.

I can only theorize about the solution and that is to stop the redirect, something that is nearly as imposible without even visualizing the page.

If they have this protection "ON" now, is because they were suffering a DDOS attack , i presume. It seems that we will need to wait.

[dpmack]

Bypassing the DDOS protection isn't actually that big of a deal unless the trade macro is the thing causing the DDOS (from legit users). Which would mean that poe.trade will not work in the long run cause whats the point of a trade system if it only works when ppl don't use it widely.

The DDOS protection is really simple, when the user goes to the website, they are redirected, some javascript is run to confirm the browser is real, then a cookie is set so that the check does not need to run every time.

Thus all the trade macro must do is pretend to be a browser that is already cleared, if the macro makes the request using the same user-agent and cookies it is indistinguishable from a real browser.

I do not know how often the cookie changes, but so far my cookie has worked for atleast an hour.

For user friendliness we are best off making the script provide a pop up asking for cookie and user-agent when a failure is detected (502 i think). We can also provide a bookmarklet that would also get the cookie and user agent without requiring ppl to user dev console. (Though some ppl might be uncomfortable running javascript like that).

[thirdy]

How about employing an actual browser like http://electron.atom.io/

I know it's out of scope of this project. But I just thought that might help.

[dpmack]

If all else fails we can get ppl to open up their dev consoles to fix this (the cookies are httponly so javascript cannot get them).

I am currently learning AHK and comobj stuff because I believe this can be worked around with an non visible IE instance and and a pinvoke call. When I its close to working but not quite yet.

[Jiinnx]

i mean for who want a dirty solution the one with cookies works, but cannot add here since it is my personal cookies and i do believe everyone should do their own :)

anyway it looks something like this in the macro:

HttpObj.SetRequestHeader("Cookie",": cfduid=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; qca=xxxxxxxxxxxxxxxxxxxxxxxxxxx; userid=xxxxxxxxxxxxxxxxxxxxxxx; live_notify_sound=1; live_notify_browser=0; live_frequency=0; __uis=xxxxxxxxxxxxxxxxxxxxxxxxx; league=Breach; cf_clearance=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; OX_plg=swf|shk|pm; OX_sd=3; mb_uid2=xxxxxxxxxxxxxxxxxxxxxxx; _ga=GA1.2.601433097.1479816558; _gat=1")

[thirdy]

Here's a step-by-step workaround:

1. In your browser. Open a new tab.

2. Open the Chrome Developer Console by pressing ctrl+shift+j.

3. Click on the Network tab. This allows us to view the actual traffic in/out of our browser.

4. In the address bar, enter poe.trade and press enter.

5. You should now be able to see the initial request for poe.trade and copy your user-agent. Make sure you copy it properly, triple clicking or highlighting should do it. See this screenshot: Copy this into your notepad.

6. Next, get the Cloudflare cookies via the Application tab > Cookies > poe.trade. See this screenshot:

7. You'll need two cookies. __cfduid and cf_clearance (cf probably means cloudflare). Copy these into your notepad. Make sure you copy it properly, triple clicking or highlighting should do it.

8. Next, edit this file: POE-TradeMacro\trade_data\TradeMacro.ahk. (You can use notepad, though it's worth getting notepad++).

9. On line 1036, you'll see:

   HttpObj.SetRequestHeader("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36")`

   • Replace it with your own user-agent.

10. Add a new line after line 1042. Add this (of course, replace the values):

    HttpObj.SetRequestHeader("Cookie","__cfduid=d03c6f347eae223a8d6e0a54a07e3b21f1480833467; cf_clearance=c513ae763bcd95e2a7cacd69855ad5180a70fd32-1482347774-31536000")

    • Note the spaces after the semi-colon. It won't work if you don't have it. This will add a cookie as expected by Cloudflare. See screenshot:

11. Save the your changes. Exit the macro (if your running it). Start the macro. Here's me testing it:

Cheers! Shameless plug: checkout my new project called Wreaclast Online

Update1, /u/twiz__ posted a fix to make it work for currencies as well:

Copy the UserAgent and Cookie lines from the above section down below, to line 1088. Picture here:

[dpmack]

I have done a bunch of learning and I think I have a programmatic solution but its really far reaching.

To make things work you must get a useragent + cookie pair that match (probably IP too but that is free).

You can use comobj to make IE navigate to poe.trade, and IE passes the test since its a real browser, IE can be invisible for this.

comobj also allow us to snag its user agent easily, however the cookies we need are httponly (for a reason) and the comobj cannot see them.

There is a function in wininet.dll ... InternetGetCookieEx which allows the getting of httponly cookies, but with a catch, cookies from websites in 'protected' mode are hidden.

The cookie belongs to cloudflare (not poe.trade) meaning you can 'trust' cloudflare.com in IE which will allow that function to get the cookie now.

It is possible to pragmatically trust a website but that is done in registry and I think that's a bit dirty, even if we trust then untrust as soon as we have what we want, I am not sure about editing a user's registry.

We don't have to go all the way with these options either, we could ask users to trust cloudflare themselves, or less have to manually get useragent/cookie everytime it expires (with a GUI).

I think trusting cloudflare is safe but someone who knows more of what IE actually does with that trust stuff would be best to weigh in.

Another option is using an IE comobj to get the html of the page then parsing it from there, but that means users who don't like IE will have it running and loading all those ads and garbage just to do basic screenscraping.

[Dixeet]

dpmack, exactly what I have done too and its a fucking pain in the a**. If only those cookies was not httpOnly. Another poke that i made was to fill directly the form http://pastebin.com/wAExWeDx you can see a simply relative search done but it will be the same difficulty to dev.

[dpmack]

Those cookies are like that so we can't do this, and they work pretty well.

I don't expect this DDOS to go away (its probably ppl trying to force GGG make their own trading).

poe.trade cannot allow this script access in any special way since a DDOS could use exactly the same thing as whatever we get

So I think the only ways we solve this is by legit passing cloudflare's test, which is by using a real browser.

[Dixeet]

The other solution that we thought was to make nodejs a requirement.

With nodejs, it should be easy to execute the javascript sent by cloudflare, then submit the form and get the response headers with setcookies in it

[dpmack]

easy to execute but I have not looked into what the javascript actually does to confirm a valid user

it may do complex checks to confirm its running in a browser, and technically then we'd be working against the DDOS protection (meaning they can attempt to break our workaround)

while it can be argued getting the key and using it else were is working against, atleast we aren't trying to fool the detection script, and the key is to confirm there is an actual user, which is what we'd be doing

[Jiinnx]

Honestly its how they configured CF because they can select other options to protect against DDOS than cookies exchange (i work for a competitor of CF and we have 7 different methods for it based on different scenarios). Only thing is that probably default its with cookies exchange and its the easier to configure.

[Dixeet]

@dpmack this is a simple dynamic js script that we could execute in node to do the same thing i think. The only restriction is it has to be execute in js because there stuff like this var xVnYEFU={"QXc":+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]))}; @Jiinnx Yes you are right But if i'm correct Eruyome had an anwer from poe.trade dev and he should implement something more convenient to use his website

[anonymous2ch]

Or you could have just used https://github.com/Anorov/cloudflare-scrape or https://github.com/codemanki/cloudscraper as a client

[Dgc2002]

Just to verify that cloudflare-scrape is fairly straight forward:

But yes, ideally poe.trade makes some configuration changes that take this issue into account.

[dpmack]

well i was trying to figure a way to solve this problem without requiring people to install extra software, i consider installing node or python better option than editing registry of course

if we aren't opposed to bundling an exe with the script we could use py2exe to package the python solution, then get it to give the ahk script the tokens it needs

but, someone said something about poe.trade helping out with this problem and i know nothing about what that entails

[xyzz]

The "helping out" part was about providing a way to open searches on poe.trade (the ctrl+q hotkey), so simply by allowing GET requests to /search instead of having to POST and extract the URL.

I don't see any way to allow trade macro access the API without making it trivial for attackers to bypass the protection (short of buying a separate server specifically for the trade macro and allowing direct access to it).

[Eruyome]

Ok guys, now that I'm awake again I'll join this discussion. For now I'll try to release a hotfix using thirdys step by step workaround. Probably with some file to write the information to, some popup if it's not filled out.

Then we'll have to see how to proceed, I like the idea of using cloudflare-scape or it's nodejs port, have no python or py2exe experience though in case this is not easy and straightforward. I don't like requiring a python installation, I think they can be a bit tricky sometimes, especially if you have or need different versions. Nodejs should be more user friendly.

Regarding xyz's help, being able to make GET request to /search would easily allow the "open on poe.trade" functionailty again, it would also allow opening the page in an ahk-internal IE. If it's possible to download the loaded page contents all the other problems would be solved, too. So we shouldn't ignore this option, I'm not saying it would be the best/easiest solution though.

[anonymous2ch]

NodeJS is installed with latest nVidia drivers. And there is NodejsPortable (http://gareth.flowers/nodejs-portable/) which essentially is everything we need to launch scrapper.

[Eruyome]

So, I tried implementing the fix explained by @thirdy on the dev branch https://github.com/PoE-TradeMacro/POE-TradeMacro/tree/development, but I can't get it to work on my end. Please tell me if this works for anyone and if not help me find the problem. If you find it you can simply create a pull request.

EDIT: I also tested https://github.com/Anorov/cloudflare-scrape, retrieving the cookies/user agent works, retrieving the page contents works (using a search url), but downloading the contents using the (trimmed down) curl example fails (only downloads the cloudflare page).

[Nephera]

Can someone post a quick tutorial (preferably with pictures) specifically for currency.poe.trade?

[Eruyome]

The step by step tutorial above also tells you what to do to fix currency search. @pgl33 Did you use my version from the dev branch or your self edited release version? If the latter is the case, could you zip the folder and upload it somewhere, I want to see if there's any mistake I made. Not that it's likely considering the small changes to be made... anyway I can't get it to work for me.

[Eruyome]

https://camo.githubusercontent.com/9c96ed605fef3dbee706fc145a2d1c0941c4b1aa/687474703a2f2f692e696d6775722e636f6d2f62717338616b622e706e67

last screenshot above, same file

[Eruyome]

I just released a hotfix based on the latest release (it seems I broke something else in the dev version). It is still not the most user-friendly solution but gives some instructions and doesn't require script-editing.

[Dagda47]

Thanks. The updated version does it for me.

[Eruyome]

Next step is to automate getting the useragent/cookies, for example with https://github.com/Anorov/cloudflare-scrape or https://github.com/codemanki/cloudscraper, possible also using http://www.py2exe.org/ or http://gareth.flowers/nodejs-portable/.

[Eruyome]

I'm not sure, can you try that without a VPN somewhere? And did you doubl-/triple-check that you used the right useragent/cookie values?

[nooksXS]

currency search is working fine, but item search wont work

even with the new version 1.3.0 only currency search is working

[Eruyome]

Have you tried to self-fix it according to thirdy's tutorial? Got any success with that?

[nooksXS]

i tried thirdy's tutorial before your hotfix, same problem with item seach not working

[emrearapoglu]

I think I found the problem, when you copy "cf_clearance" cookie, it doesn't copy all of it. Only copies first part before the "-".

I was trying the self-fix method when I realized my cf-clearance key was too short. Now it works with or without VPN, using 1.3.0 version. Sorry! 😅

[engelswut]

I had the same problem while copying the cf_clearance key, the column was too small so it was not the full key copied. I got the column wider and copied the whole key and et voila, everything works !

[nooksXS]

UPDATE

i change the default browser from Chrome Version 55.0.2883.87 to Firefox 50.1.0 and use the firefox web console to find the user agent, cfduid and cf_clearance und used these information

now it works for currency and item search

[Eruyome]

The development branch is now up to date, too. So please use it if you consider any development on the script.

[Damaged2]

have same problem with cf_clearance So i use it "Edit This cookie" addon chrome

[OzoneH3]

Hi, for convenience until this isue is solved I modified the cfscrape python script a little and created an exe with cxfreeze:

Download Link: get_cookie_data.rar

Running the file get_cookie_data.exe (takes about 6 sec) will create a cookie_data.txt with the format

useragent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36
cfduid=dcaf7ac5f55c26aaef8eb8fdb264787bb1482413958
cfclearance=33c6514d113eca470dc90a58b7c12552f8ebf386-1482413963-604800

Just copy the file over, done.

The script from which I created the exe is inside: get_cookie_data.py

Edit: Um automatisch beim TradeMacro Scriptstart die cookies abzurufen die Zeile 885 in TradeMacroInit.ahk von:

FileRead, cookieFile, %A_ScriptDir%\cookie_data.txt

nach:

Run, %A_ScriptDir%\trade_data\get_cookie_data\get_cookie_data.exe
FileRead, cookieFile, %A_ScriptDir%\trade_data\get_cookie_data\cookie_data.txt

ändern. Der get_cookie_data ordner muss sich dazu im trade_data ordner befinden.

[Eruyome]

@OzoneH3 That's probably all we need to get this working without user-interaction. I'm pretty sure though, that there will be people who don't like to trust some .exe file. Can you give me a quick overview on how to create this exe with cxfreeze? I could include the source files and a manual on how to compile it yourself.

[OzoneH3]

@Eruyome Sure, thats why I've added the script as well.

Install Instructions:

• Install Python 3.5
• Change directory to script folder
• In cmd do: pip install cfscrape

CMD only:

• Run script directly with python get_cookie_data.py

Create EXE:

• Get cxfreeze and install (Make sure it's the same as your python version): https://pypi.python.org/pypi/cx_Freeze
• Create exe with: python -path to cxfreeze script- get_cookie_data.py
• cxfreeze script should be located when using the installer in: C:\Users\\AppData\Local\Programs\Python\Python35-32\Scripts\cxfreeze
• That creates a dist folder where the exe will be located

PS: Here is a pastebin to the script if you don't want to download anything at all: get_cookie_data.py

[CMterio]

so is there any way i can fix prophecies ? thank you for good working.

[Eruyome]

@OzoneH3 thx, will take a look at it. @CMterio what do you mean, fix prophecies?

[CMterio]

@Eruyome idk, everytime i try to check a prophecy with control+q or control+d he give me a wrong information which is the last item or currency i check before.

EDIT: Unique Belt Information in tooltip. Control+d

everything else is fine.

[Eruyome]

That's because it doesn't work on prophecies. https://github.com/PoE-TradeMacro/POE-TradeMacro/issues/33

@OzoneH3 So I followed the instructions you gave, everything seemed to go right (installations), used cxfreeze to create the dist folder with the .py script but when I execute the .exe file I get this error: the application was unable to start correctly 0xc000007b Any idea what could have went wrong?

EDIT: I get this when using cxfreeze, no idea if I need those http://pastebin.com/uSzWwsNw, probably need some of them...

[OzoneH3]

@Eruyome Hmm, weird, first time I've used cxfreeze so not really. A quick google search indicates it could be a dll problem. Missing runntime or wrong version 32bit <-> 64bit.

Maybe take a look at this: stackoverflow

PS: I'm running a 64bit windows but have the 32bit python and cxfreeze version installed.

[Eruyome]

Ok, that could be the case. That aside, it doesn't seem like you can create a single, stand-alone .exe with cxfreeze, right? Would be nice to have that instead of a dist folder with 1000 files, unless you only need a small part of them...

EDIT: cxfreeze's description states "stand-alone executables" so it should be possible, it's not the case with the files you gave me though.

[OzoneH3]

Yeah, it's kind of a mess. I'm not sure if py2exe creates a standalone because it's only working up to python version 2.7 and I didn't wan't to install an older version thant the one I already had, in case it breaks something here.

If people are installing python and cfscrape on their own, the cmd variant would be enough anyway. Creating the exe is mainly for distributing to people who don't want to install anything.

In that case just run the python script on ahk start. Updating the python script would be also way easier if anything changes in the future.

[Eruyome]

I can't really expect everyone to install python though, especially because it can be a mess (multiple installed versions, path entries etc). I think nodejs is a bit more user-friendly, at least in my experience, I have python 2.7 and 3.5 installed. Since my knowledge about this stuff is a bit limited I've run into more than enough problems^^