Can't get cf_clearance since 2.4 update

[cirf]

v2.4.0 Cookie file found. Retrieving cookies failed. Windows Version: 10.0.15063 (10.0.15063), 64bit. Compiling 'getCookieData' script successful. Net Framework used for compiling: v4 Internet Explorer: v11.0.15063.0

after the update today i get the error message on start cf_clearance missing. I even can not get this cookie parameter in firefox or internet explorer, only in chrome. If i set the chrome cokie manually i also get an error that poe.trade can not be accesed. If i run cURL manually i get the 200 Code like shown in the FAQ 2.8

i am quite clueless, any ideas?

[Eruyome]

There's a file temp\StartUpLog.txt in your TradeMacro folder, please show me that.

[cirf]

----------------------------------------------------------
2017-08-26T11:53Z
----------------------------------------------------------

Requesting leagues from api.pathofexile.com...

cURL command:
"P:\Tools\POE-TradeMacro\lib\curl.exe" "http://api.pathofexile.com/leagues?type=main" -ILks -H "Host:api.pathofexile.com" -H "Connection:keep-alive" -H "Cache-Control:max-age=0" -H "Content-type:application/x-www-form-urlencoded; charset=UTF-8" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "User-Agent:Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36" 

Answer:
HTTP/1.1 404 Not Found
Pragma: no-cache
Connection: close
Content-type: text/html; charset=utf-8
Content-Length: 5529

----------------------------------------------------------
2017-08-26T11:53Z
----------------------------------------------------------

Testing CloudFlare bypass, connecting to http://poe.trade...

cURL command:
"P:\Tools\POE-TradeMacro\lib\curl.exe" "http://poe.trade" -ILks -H "User-Agent:Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "Accept-Encoding:gzip, deflate" -H "Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4" -H "Connection:keep-alive" -H "Host:poe.trade" -H "Upgrade-Insecure-Requests:1" 

Answer:
HTTP/1.1 404 Not Found
Pragma: no-cache
Connection: close
Content-type: text/html; charset=utf-8
Content-Length: 5502```

[Eruyome]

You seem to have some issue that I haven't encountered before, the answers to both requests (GGG API and poe.trade) are the same, here are mine for reference, vastly different.

Requesting leagues from api.pathofexile.com...

cURL command:
"D:\Users\Eru\Documents\GitHub\POE-TradeMacro\lib\curl.exe" "http://api.pathofexile.com/leagues?type=main" -ILks -H "Host:api.pathofexile.com" -H "Connection:keep-alive" -H "Cache-Control:max-age=0" -H "Content-type:application/x-www-form-urlencoded; charset=UTF-8" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "User-Agent:Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36" 

Answer:
HTTP/1.1 302 Found
Server: nginx/1.8.1
Date: Sat, 26 Aug 2017 01:01:06 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Set-Cookie: POESESSID=713d8fdcf465fb2d05e538cb55831927; path=/; domain=.pathofexile.com; HttpOnly
Location: /login/?redir=%2Fleagues%3Ftype%3Dmain
X-Frame-Options: SAMEORIGIN

HTTP/1.1 404
Server: nginx/1.8.1
Date: Sat, 26 Aug 2017 01:01:07 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Set-Cookie: POESESSID=fa0210ddee69475477a5d56280aa6c9c; path=/; domain=.pathofexile.com; HttpOnly

----------------------------------------------------------
2017-08-26T01:01Z
----------------------------------------------------------

Testing CloudFlare bypass, connecting to http://poe.trade...

cURL command:
"D:\Users\Eru\Documents\GitHub\POE-TradeMacro\lib\curl.exe" "http://poe.trade" -ILks -H "User-Agent:Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "Accept-Encoding:gzip, deflate" -H "Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4" -H "Connection:keep-alive" -H "Host:poe.trade" -H "Upgrade-Insecure-Requests:1" 

Answer:
HTTP/1.1 200 OK
Date: Sat, 26 Aug 2017 01:01:08 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Set-Cookie: __cfduid=df7e5b31a1b36443645660e9169786e6f1503709268; expires=Sun, 26-Aug-18 01:01:08 GMT; path=/; domain=.poe.trade; HttpOnly
X-Frame-Options: DENY
Content-Encoding: gzip
Server: cloudflare-nginx
CF-RAY: 3942dbaea15815b3-FRA

[Eruyome]

Do you see any other information when executing the shown command in a windows cmdlet? Simply copy the entire line.

[cirf]

Interesting, setting the custom header via -H "Host:poe.trade" fucks it up

curl "http://poe.trade" -ILks -H "User-Agent:Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8" -H "Accept-Encoding:gzip, deflate" -H "Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4" -H "Connection:keep-alive" -H "Host:poe.trade" -H "Upgrade-Insecure-Requests:1"

HTTP/1.1 404 Not Found Pragma: no-cache Connection: close Content-type: text/html; charset=utf-8 Content-Length: 5502

without that parameter i get the 200

curl "http://poe.trade" -ILks -H "User-Agent:Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8" -H "Accept-Encoding:gzip, deflate" -H "Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4" -H "Connection:keep-alive" -H "Upgrade-Insecure-Requests:1"

HTTP/1.1 200 OK Date: Sat, 26 Aug 2017 15:49:50 GMT Content-Type: text/html; charset=utf-8 Connection: keep-alive Set-Cookie: __cfduid=daa1ced67baf722950bb3c36abba9e6aa1503762590; expires=Sun, 26-Aug-18 15:49:50 GMT; path=/; domain=.poe.trade; HttpOnly X-Frame-Options: DENY Content-Encoding: gzip Server: cloudflare-nginx CF-RAY: 3947f18134e126d2-FRA

Any idea on that?

[Eruyome]

I don't know why that header messes your request up...

Could you open your browser developer tools, network tab, maybe "preserve log" (if chrome) and take a look at a poe.trade search request? Look for the request headers, is the "Host" header there?

If that header isn't required for everyone to make this request work I guess I can not use it but I'm not sure about that. Also, what about the request to http://api.pathofexile.com/leagues?type=main?

[cirf]

after removing the host part i get a 200 from http://api.pathofexile.com/leagues?type=main?

HTTP/1.1 200 OK
Date: Sat, 26 Aug 2017 19:29:00 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Set-Cookie: __cfduid=da6c638b6cd3c4818237661d24871318f1503775739; expires=Sun, 26-Aug-18 19:28:59 GMT; path=/; domain=.poe.trade; HttpOnly
X-Frame-Options: DENY
Content-Encoding: gzip
Server: cloudflare-nginx
CF-RAY: 39493286f1f1275c-FRA

i checked 3 browsers, all showed that Host Part in the developer console, chrome (wich has the cf_clearance part), vivaldi (chromium based, wich hasn't) and firefox (also does not get this cf_clearance)

[byg0n3]

I can confirm that 'Host' header is not required/mandatory for poe.trade @cirf are you using any proxy?

[Eruyome]

I know it's not required in general but sometimes there are and were people with some weird curl issues so wanted to make sure that everything is there that could be needed. I guess I can try removing it and see if someone complains^^ What I don't get is why this issue occurs in the first place.

[cirf]

i have not configured any proxy. Checked the windows 10 settings, no proxy activated there.

[byg0n3]

I'd take one more guess... Can you execute nslookup poe.trade in cmdlet and show us the output?

[cirf]

I researched a bit for myself and found out that i had a blacklist in my router activated. Side effect was blocking direct access to ip adresses without a dns entry. As the macro was working all the time with this activated i did not thought that this could be the reason and also forgot about that setting. now after removing it it works again.

Trade macro and curl are working now!

I am not sure how this started to apear now after the update? Sorry for wasting your time and thanks for helping me!

[byg0n3]

Well, I was thinking almost about the same :) Some kind of weird dns/ip interaction on your side.

[byg0n3]

@tenslidetails чаще всего ошибка 404 в подобной ситуации (проблемы с заголовком Host) возникает из-за того, что сам заголовок изменяется чем-либо/кем-либо по пути и становится невалидным. Причиной может быть proxy, некорректная настройка роутера (неверные/неполные dns записи, layer7 proxy, chain, nat, forwarding) и т.п. Я погу посоветовать отправить запрос непосредственно с интерфейса роутера (если есть такая возможность и навыки) или отключить dns server на роутере (получение записей с роутера) и попытаться брать dns записи напрямую от провайдера или гугла (опять же, если есть возможность - воткнуть кабель от сразу в комп, минуя роутер). При любых манипуляциях с dns записями желательно скинуть на компе dns кэш: ipconfig /flushdns.

English TL;DR: the most common case here - malformed Host header. Reasons - proxy, incorrect router setup (wrong/incomplete dns record, layer7 proxy, chain, nat, forwarding) etc. Try requesting poe.trade directly from the router interface or disable dns server on the router and receive dns records directly from isp/google. The most easiest way - plug isp cord into pc, bypassing the router. And don't forget to clear dns cache on your pc: ipconfig /flushdns

[byg0n3]

На некоторых Dlink-ах за это вроде отвечает опция DNS Relay. Тут ничем помочь не могу, ищите мануалы к железке. И в свойствах подключения на компутере указываем сервера dns от првайдера или 8.8.8.8 и 8.8.4.4 для гугловых (Свойства -> TCP/IPv4 -> Свойства -> Общие)

[Eruyome]

Is this still unresolved?

[Eruyome]

I can implement some workaround, but for now you could edit lib\PoEScripts_Download and replace

    curl        := """" A_ScriptDir "\lib\curl.exe"" "  
    headers := ""
    cookies := ""
    For key, val in ioHdr {
        val := Trim(RegExReplace(val, "i)(.*?)\s*:\s*(.*)", "$1:$2"))   

        If (RegExMatch(val, "i)^Cookie:(.*)", cookie)) {
            cookies .= cookie1 " "      
        }
    }

with

    curl        := """" A_ScriptDir "\lib\curl.exe"" "  
    headers := ""
    cookies := ""
    For key, val in ioHdr {
        val := Trim(RegExReplace(val, "i)(.*?)\s*:\s*(.*)", "$1:$2"))
        If (not RegExMatch(val, "i)host:.*")) {
            headers .= "-H """ val """ "    
        }       

        If (RegExMatch(val, "i)^Cookie:(.*)", cookie)) {
            cookies .= cookie1 " "      
        }
    }

[dnastandsfor]

No cURL response when requesting poe.trade. A similar problem, the solution you proposed did not help.

[Eruyome]

I talked to @dnastandsfor already, since he's also russian and says poe.trade is blocked for him I'm not surprised that the curl request fails, but what I find weird is that the response is empty. I'd at least expect some kind of response. There were others where poe.trade is blocked but I believe for nobody else was TradeMacro just stuck at "Reading user-agent and cookies".

[dnastandsfor]

Да конечно, все сохранил и перепроверил 10 раз @tenslidetails

[byg0n3]

Is request totally blocked (like by ISP/office proxy/other stuff like that) or it just fails cos of the header?

[Eruyome]

It seems to be totally blocked. Anyway one other issue was that the getCookieData.exe got stuck/hung up. I added a timeout to kill it after 15 seconds. The better solution would be to change that script to have a timeout though^^

[dnastandsfor]

In general, I was helped only by the proxy connection. thanks @Eruyome for helping

[byg0n3]

@Eruyome maybe it's time to actually implement alternative search via poeapp? :) At least it can be used as a safe fallback for normal price checking with ctrl+d.

[dnastandsfor]

@dein0s poeapp and poe.trade, one ip, btw.

[byg0n3]

@dnastandsfor you're wrong with this. They both use CloudFlare (reverse proxy, ddns and all that stuff), and if their (CloudFlare) IP range is restricted for you for some reason - that might be a problem. But, besides that, I doubt that they actually use the same hosting provider.

[dnastandsfor]

@dein0s ok, did not know

[Eruyome]

Even if poeapp wouldn't also use Cloudflare (I'm assuming there are all teh same issues as with poe.trade), it wouldn't be that easy. Implementing is might not even be that much work but I talked to the dev and atm he wouldn't even be able to handle the requests. At least there would be more requests without any ad revenue.