[ToDo] Bypass poe.trades cloudflare protection

[Eruyome]

It should be possible to bypass this, here is a reference: https://autohotkey.com/board/topic/111944-winhttprequest-and-ddos-protection/

And here is a script that I made to test/develop this, it's just the current function to make the winhttp request to poe.trade: https://dl.dropboxusercontent.com/u/13620316/bypassCloudFlare.ahk

Any help appreciated.

Ideas:

• Reverse engineer the cloudflare javscript to create the GET request used to delegate the browser to poe.trade, if it's necessary to execure javascript
• Embed webbrowser in a hidden GUI (Reference), find a way to get the actual search url without having to bypass cloudflare or open poe.trade in this browser and automate filling the form, then sending it and download the page contents (no idea if actually possible).

[TheCoderJT]

Followed everything to a tee and didn't work :( not sure why i will probably just wait until there is a bettter way out there..

[Eruyome]

Some people seem to have the problem that they don't copy the entire cfclearance string, are you sure you got that right? It's easier to see here:

[TheCoderJT]

yes i copied everything, unless im doing something wrong, i followed the instructions even triple checked by saving the file and reloading the macro and tried to ctrl d an item and still nothing shows....

Like when you add the new line with this code - HttpObj.SetRequestHeader("Cookie","__cfduid=d03c6f347eae223a8d6e0a54a07e3b21f1480833467; cf_clearance=c513ae763bcd95e2a7cacd69855ad5180a70fd32-1482347774-31536000")

I enter in my cfduid and clearance i got from ctrl shift j - i also added the first code as well and entered in my useragent and saved the file and reloaded the macro and tried to ctrl d a gem and no results

Im going to try one more time...will let you know my results

[TheCoderJT]

Okay so after my struggles finally got it to work :) i think what i did wrong was this - On the developers window think i didn't copy the whole clearance code down....

I also found out that this only works if you use the newest release so before changing the useragent. cfduid and cf_clearance rewrite your poe.macros folder with the newest files on the release page..

[DoctorVanGogh]

Well, if installing python is considered too much of a hassle, and a binary exe redistribution is a nono (I agree with both statements), here's an idea:

Use a .NET component/program. Everything since Vista has at least .NET 2.0 preinstalled, which includes a compiler. Distribute the source code, invoke the compiler from the 'init_thingemagic.ahk' script, call the generated exe as necessary - Profit!

Here's a hacked together C# source code to open a (minimized) browser window to poe.trade and read the stored cookies afterward:

using System;
using System.Runtime.InteropServices;
using System.Text;
using System.Windows.Forms;
using ConsoleApplication3;

namespace ConsoleApplication1 {
    class Program {

        [DllImport("wininet.dll", CharSet = CharSet.Auto, SetLastError = true)]
        static extern bool InternetGetCookieEx(string pchURL, string pchCookieName, StringBuilder pchCookieData, ref System.UInt32 pcchCookieData, int dwFlags, IntPtr lpReserved);

        public const int INTERNET_COOKIE_HTTPONLY = 0x00002000;

        private static bool _completed;

        [STAThread]
        static void Main(string[] args) {
            var miniBrowser = new BrowserWindow(new Uri("http://poe.trade"));

            Application.Run(miniBrowser);

            Console.Write($"{miniBrowser.Agent}{Environment.NewLine}{miniBrowser.Cookies}");           

            Console.ReadKey();      // REMOVEME
        }

        public static string GetCookieString(Uri uri) {
            var url = uri.ToString();

            // Determine the size of the cookie      
            UInt32 datasize = 256 * 1024;
            StringBuilder cookieData = new StringBuilder(Convert.ToInt32(datasize));                
            if (!InternetGetCookieEx(url, null, cookieData, ref datasize, INTERNET_COOKIE_HTTPONLY, IntPtr.Zero)) {       
                if (datasize < 0)
                    return null;
                // Allocate stringbuilder large enough to hold the cookie    
                cookieData = new StringBuilder(Convert.ToInt32(datasize));
                if (!InternetGetCookieEx(url, null, cookieData, ref datasize, INTERNET_COOKIE_HTTPONLY, IntPtr.Zero))
                    return null;
            }
            return cookieData.ToString();
        }

    }
}

class BrowserWindow : Form {
    private readonly Uri _uri;

    private WebBrowser _wb;
    private string _agent;
    private string _cookies;

    public string Cookies => _cookies;
    public string Agent => _agent;

    public BrowserWindow(Uri uri) {
        _uri = uri;
        ShowInTaskbar = false;
        WindowState = FormWindowState.Minimized;
    }

    protected override void OnLoad(EventArgs e) {
        base.OnLoad(e);
        _wb = new WebBrowser {
                            AllowNavigation = true,
                            ScriptErrorsSuppressed = true
                        };

        // get user agent
        string js = @"<script type='text/javascript'>function getUserAgent(){document.write(navigator.userAgent)}</script>";
        _wb.Url = new Uri("about:blank");
        _wb.Document.Write(js);
        _wb.Document.InvokeScript("getUserAgent");
        _agent = _wb.DocumentText.Substring(js.Length);

        // send browser to targeted uri
        _wb.DocumentCompleted += wb_DocumentCompleted;
        _wb.Navigate(_uri);
    }

    void wb_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e) {
        if (e.Url == _uri) {
            _cookies = Program.GetCookieString(_uri);
            this.Close();
        }
    }
}

Console output will be something like:

Vector smash protection is enabled.
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; Win64; x64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729)
__cfduid=**************************; cf_clearance=***************; [potentially other stuff]

Seems installed browser plugins can write stuff (See 'Vector smash protection is enabled' line), so a production version couldn't simple pipe the user agent & cookies to the console. Instead you pass in filenames where to write the desired data.

Also I think this source code needs a C#6 compiler. If this is a way to go forward I can easily clean this up so even a basic .NET 2.0 compiler will run things.

[dcazrael]

In reply to the hotfix that went out. User agent is on line 1042 in the file I downloaded.

HttpObj.SetRequestHeader("Cookie","__cfduid=" cfduid "; cf_clearance=" cfClearance)

The additional " are confusing for new users and can cause errors. Would suggest to write the line like this HttpObj.SetRequestHeader("Cookie","__cfduid=cfduid; cf_clearance=cfClearance")

[Dixeet]

Ok guys, I think i manage to make it working without any other requirements. It still need to be implemented in the current script but I'm to tired to do this tonight. Here is the code if someone wants to try it : http://pastebin.com/WNERKTy9 The request is longer than before I think but still acceptable I guess

[thisworldisanillusionexile]

Entered my data for the three variables on the cookie data file, and now this.

Pls fix.

[buddyp450]

I followed all the steps and all searches are turning up blank.. did I miss something obvious?

[TFSN001]

can someone help me dont work at all!

[tissuee]

after plugging in the user-agent, cf and cfclearance

[Eruyome]

@dcazrael The latest version doesn't require any script editing, maybe this wasn't clear enough. @thisworldisanillusionexile can you check if \temp\currentLeagues.json is downloaded properly at script start? Did this ever happen with an older relase for you? @buddyp450 Make sure you copied the correct values, cfclearance has the following format xxxxx-xxx-xxx (just longer), many people only copy part of this. @TFSN001 @tissuee Have you edited the script in any way (not required in latest release) and does this happen for you in any older release?

@Rodrive I will certainly take a look at your solution. @DoctorVanGogh Aside from Rodrive's solution (which I haven't checked out yet), yours seems to be pretty good and solves some concerns many people could have (binary exe redistribution). I'm not entirely opposed to distribute such an exe as long as I can provide the source code and a manual on how to compile this exe yourself. Of course your solution is much more preferable. Therefore it would be really appreciated if you can clean your script up and make sure it works everywhere, I have no experience writing and compiling C# programs, or even invoking this compiler.

Some more thoughts on compiling OzoneH3's python script with cxfreeze/pyinstaller/py2exe into a sinlge stand-alone exe: I couldn't get any of those to work on my machine so far, but those problems should be solvable. Not neccessarily easy for someone inexperienced. That's one more concern regarding the ability for everyone to self compile it instead of using the distributed exe.

[Eruyome]

@Rodrive no matter if I implement your solution into the macro or just test it in a stand-alone script, it only downloads the cloudflare html, am I missing something here?

[OzoneH3]

Rodrives solution kind of works. It needs to have a sleep of 5000 the first time it runs thought to actually pass the cloudfare test. After that 50 ms is fine since IE saves the cookies. The speed is pretty bad thought since it renders the whole result page before returning the html body.

Best testet with wb.Visible:=0 to actually see whats happening

[Dixeet]

@OzoneH3 Yeah this is exactly what I was saying to Eruyome, I need to add a control to check if this is the first time we go on the website. For the speed, this is the big problem with this solution...

[Eruyome]

My current thoughts on this: Use DoctorVanGogh's solution (or a similiar one) and have Rodrives as a fallback (to be enabled in settings) in case someone has problems with the other solution. This fallback could be implemented later on.

[OzoneH3]

Modified DoctorVanGoghs script a little to compile with csc.exe .net 4.0. Only getting the cfduid though: http://pastebin.com/JtpseZVd

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Win64; x64; Trident/7.0; .NET
 CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
.0; .NET4.0C; .NET4.0E)__cfduid=d07af817a6f3811437738f2aaa44b83761482485005

[Eruyome]

I got this using your edited script:

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; Win64; x64; Trident/7.0; .NET
4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729; Info
Path.3; Tablet PC 2.0; Creative AutoUpdate v1.41.07)
__cfduid=d6a91f7d44cd54586ee3685624f60160c1482481576; 
cf_clearance=d46b2e6c3d7ef26ae74a0ddf1c01261e08df6678-1482483053-604800; 
_ga=GA1.2.847074042.1482483054; 
mb_uid2=1658909506903757439

[OzoneH3]

Did you clear your IE cookies beforehand? It seems like this is only getting the cookies from the cloudflare challenge site and dosen't really get past the test.

[Sidusol]

Can someone help me with this one.

In line 1036 and 1042. I have in this lanes smth like this:

HttpObj.SetRequestHeader("Host","poe.trade")

HttpObj.SetRequestHeader("User-Agent", UserAgent)

The cookie lane is much lower and it doesn't look like the one u said: HttpObj.SetRequestHeader("Cookie","__cfduid=" cfduid "; cf_clearance=" cfClearance)

I changed the lanes with user agent and with cookie to make it looks like yours, but the macro doesn't work...

[Eruyome]

@Sidusol don't edit the script, just cookie_data.txt

@OzoneH3 I did not, you're right... anyway I also edited the script to write the results to file. http://pastebin.com/nt38sDTc.... I really like this solution, we just have to get it past the challenge now.

[Sidusol]

ah ok... thx

Can u explain how this macro actually works. It searches the items on poe.trade, but it ignores the mods... so, it's not so good to trust about prices. It can show me 1 alter, but with mods it can cost 1 alchemy.

[OzoneH3]

@Eruyome Ok got it working and added your code as well: http://pastebin.com/YPt1XRwN

Waits 6 secs before reading the cookies

[tissuee]

@Eruyome I followed the guide below because when I first load the script, it said loading cookies value failed something like that.

(thirdy commented 2 days ago • edited Here's a step-by-step workaround:

In your browser. Open a new tab. Open the Chrome Developer Console by pressing ctrl+shift+j. Click on the Network tab. This allows us to view the ......)

[Eruyome]

@Sidusol please read https://github.com/POE-TradeMacro/POE-TradeMacro/wiki/FAQ, will do a better job than me explaining it here.

@OzoneH3 thx, will take a look at it and will start implementing it if everything works.

@tissuee that's ok, the popup also says to just edit cookie_data.txt (so basically following the guide only to a certain point.

[tissuee]

@Eruyome so actually I don't need to change the coding in the trademarco.adh?

[Eruyome]

@tissuee That's right.

[Eruyome]

@OzoneH3 Can we add a check to se if we are on the cloudflare page or not and just skip the 6s delay if not necessary? I'd like to get the cookies at every script start, but waiting 6s everytime shouldn't be neccessary.

[OzoneH3]

@Eruyome Sure, here you go. Also only writes cfduid and cf_clearance cookie into the file: http://pastebin.com/yKGQ7LFJ

[DoctorVanGogh]

Okay - I'll see if I can't get this working over the holidays ;)

Edit: I can also reproduce the issue with only getting the initial verification cookie - needs some simple checks to get correct results.

[Eruyome]

@DoctorVanGogh I don't think there's any need to do more, should be working now with Ozones latest changes and I can do the ahk parts.

[Eruyome]

https://github.com/PoE-TradeMacro/POE-TradeMacro/releases/tag/1.4.0-beta I implemented the c# script, please give me feedback if everything works.

[DoctorVanGogh]

Oooookay - I think the current solution is way too brittle*, but it'll work for now.

Some hints for the ahk parts: To find out which .NET version(s) are installed check the registry under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP. Enumerate all subkeys, and look for ones with a InstallPath value. Also check the Wow6432Node node to find 32bit versions installed in 64 bit systems - see https://stackoverflow.com/questions/6660512/how-to-get-csc-exe-path & https://support.microsoft.com/en-us/kb/318785. Then you can use that path and invoke csc.exe in that directory. I think the syntax should be:

[.NET InstallPath]\csc.exe /target:exe /out:[Script directory]\CookieHelper.exe [Script directory]\CookieHelper.cs

---

: Two problems: If cloudflare ever renames their cookies, or prolongs the wait timeout this will immediately stop working. So it'd be cleaner to dump all cookies, not look for specific ones with magic names (which might be correct right now* but there is no assurance that this will hold true tomorrow). Also the navigation detection is hacky... It won't work if cloudflare increases it's timeout to over 6 seconds, or for any other reason there's a delay in the network of more than a second (we're talking internet here - for someone, somewhere things won't work as expected). My way would have been to check if the current Document contains "PoE" or "Exile" after a navigation and assume this was the real page. From what I have observed the cloudflare protection is rather generic and makes no mention of those things ;)

[DoctorVanGogh]

Re 1.4.0-beta fix:

Possibly won't work on default installs of

• Vista
• Server 2008 (& Server 2008 R2)
• Windows 7
• Windows 8
• Windows 8.1
• Windows 10

None of those systems come with .NET 4.0 installed by default. They might have that version installed by some other component, but that's not guaranteed. For details see https://blogs.msdn.microsoft.com/astebner/2007/03/14/mailbag-what-version-of-the-net-framework-is-included-in-what-version-of-the-os/

[OzoneH3]

Sure won't win any beauty contests. It's just a quick fix to get things going. Feel free to expand upon it. Would it be possible to convert it to .net 2.0? I tried a little but since I don't really do much with C# I couldn't figure it out.

The script compiles with 3.0 upward which is present since Vista and upward by default. But like you said, so much other programs need .net 4.0 that requiring it won't hurt anybody.

[DoctorVanGogh]

3.0 may have been removed in Vista (Yes, someone out there will have done it). I've changed the code slightly so every compiler from 2.0 and up will work: http://pastebin.com/epg6cAE0 (No var, no object initializer).

And just because you have 4.0 installed (and a ton of other people too) doesn't mean everybody has. They might just have 4.6.2, they might just have 2.0 or they might have 3.5 SP1 (god what a mess). Hell, on my local dev machine I've got nearly every framework ever published by MS installed - but there's going to be someone out there who hasn't - so let's try to use the smallest possible requirement, which would be '2.0 or above'.

[Eruyome]

I'll let you guys debate about the c# script and optimize it as much as you can ;-) Worst-case scenario I simply check if .Net 4 is installed and make it an requirement, as far as I know you can install it on every Windows XP+.

I'll try to implement your suggestions for the ahk parts later.

[DoctorVanGogh]

Requiring .NET 4.0 is fine - hardcoding the release version number is not. If Microsoft pushes a security update or service pack for .NET 4, your v4.0.30319 won't work anymore. 😁

[OzoneH3]

The current Master branch works for me. Created the .exe and read the cookies correctly.

[Eruyome]

I will definitely improve that hardcoded part ;-)

Btw: I can't really use poe.trade with Chrome anymore, http://poe.trade works, every /search link not, it just shows the cloudflare page stuck in a loop. After 5 seconds it repeats with a changed Ray ID displayed. IE and Firefox work, Trademacro works, restarting Chrome, deleting poe.trade cookies and Chrome cache doesn't help either. Any ideas?

[DoctorVanGogh]

WAG: Your chome generated clearance might have been blacklisted since you posted it's value here publicly, and some nitwit used it to hammer cf's service 😁 Can you get a new IP? I'd bet cloudflare incorporates that into the cookies.

[Eruyome]

1. Those values change when you delete your cookies, they are newly generated.
2. poe.trade front-page works without even seeing the cloudflare page.

That's all I get (restartet router since then to get a new IP, no change): http://imgur.com/tSHe5rI http://imgur.com/TjYvxOp

[Dixeet]

The v1.4;0 is working for me too. I didnt install .Net 4 explicitly but is seems i have it. Maybe because of a game's/softwares which sometimes require to have .NET. I'm currently on windows 10

[Eruyome]

So, the generated cookies don't work anymore for me... if I use the useragent/cookies from firefox (manual copy) it works. Anyone experiencing the same?

Aside from that, I optimized getting the .NET install path, this is unrelated though, even the beta version that definitely worked doesn't anymore.

@DoctorVanGogh Not sure I got that part with the Wow6432Node right but I hope so.

EDIT: Ok, so deleting my IE's cookies fixed that.

[xerance]

Doesn't seem to work for me, while the "open up item on poe.trade" function works (opens up the site with the item i'm trying to check), the ingame qucik search doesn't. http://puu.sh/sZ8qA/4489f3fb1b.jpg

Anyone know what step i might have messed up on? tried to do this 4 times, yet i still end up at the same error.

[Eruyome]

@xeratzy Are you using v1.3.0 or the beta (v1.4.0)?

[xerance]

1.3.0 there is a beta?

[Eruyome]

There's only one thing I can think of because I had this issue (described it just above), try deleting your Internet Explorers cookies. Oh and try using the beta. https://github.com/PoE-TradeMacro/POE-TradeMacro/releases/tag/1.4.0-beta

[xerance]

Even though I'm using chrome? edit: beta works perfect!

[hammypants]

Note that the automatic solution does not work for those that have lowered/disabled security settings for Internet Explorer. (That one should probably have if you are using other browsers.)