PoE-TradeMacro / POE-TradeMacro

Price checking script for Path of Exile.
https://poe-trademacro.github.io/
GNU General Public License v3.0
936 stars 188 forks source link

Extremely slow initialization and CTRL + D searches #738

Closed joprut closed 6 years ago

joprut commented 6 years ago

AHK version: v1.1.29.01 Windows version: 10 Home 10.0.16299 Build 16299

All was going well, I've been using Trade Macro for at least a year and never had a problem. Now it takes Forever to initialize. from merging scripts to Paying Respects to the cynical Brit there is a noticeable slowdown and from then on it is very long until it disapears. when the script is loaded, when i press CTRL+C it returns a result instantly but when i press CTRL+D it takes forever.(It still brings up results though). I've reinstalled AHK and trade macro, Cleared Cookies on all browsers and it still continues to take that long.

Poe.trade is fast and responsive on browser.

internet connection : 150 Mbps Download ,15 Mbps Upload

Update : Friends who live in the same area as me do not have this problem. Update: Right clicks on the icon take a long time top open menu and when I open, it causes the program to stop responding for a good few seconds. Update: poe.trade is the one i use for everything except rare item pricing which is poeprices.info. rare item pricing works just fine. Update: it started yesterday as well (june 6 2018) Help me resolve that issue please :)

codyst78 commented 6 years ago

I am having the EXACT same issue! Just started yesterday and I have used this script for over a year!

aRTy42 commented 6 years ago

Which site are you requesting with Ctrl+D and what kind of item are we talking about? poeprices.info with the machine learning for rares for example? poe.ninja for currency? poe.trade for everything?

(Ctrl+C is a local thing, it does not query online resources, so that one should never be delayed)

joprut commented 6 years ago

Updated initial comment for further informaition :)

Kutharsus commented 6 years ago

I'm having the same issue as OP.

aRTy42 commented 6 years ago

This technical aspect is something for Eruyome. I can give you the advice I gave codyst78 in discord though: Try Ctrl+Q (to bring up the result in the browser) until either the issue resolves itself (if it is actually due to the connection) or the problem is found.

joprut commented 6 years ago

Thank you very much for the help everyone :)

Zeronthe commented 6 years ago

I also have this issue, but it takes an eternity for it to pull anything up in Ctrl+Q as well. It's like the entire script is hanging for no reason. Poe.trade is fast, same with my internet and other associated things.

It just started happening yesterday(6/6.) Also takes an eternity to merge/initialize, going back a few versions doesn't seem to resolve, niether does doing a clean install of the script via deleting stuff in docs/temp/appdata/etc. Restarting network equipment also does nothing.

I'm on spectrum connecting from florida, but changing DNS servers doesn't seem to resolve this so it's likely not a routing issue.

siisjuu commented 6 years ago

I've had this kind of issue since like 2 days ago. I even made a post on AHK reddit because I thought the problem was on my end (None of my friends had this issue, post can be found here: www.reddit.com/r/AutoHotkey/comments/8p9p0l/autohotkey_not_responding_when_running_a_script/).

Internet connection is 150/50Mbps.

It takes like 2-3 minutes to even start the script. Trying to price uniques, prophecies, div cards, maps etc. takes an eternity. The machine learning pricing works almost instantly. CTRL + Q works fine for now. Poe.trade opens instantly.

I made a small unlisted video that shows the problem (www.youtube.com/watch?v=l64uZmm1vDE).

Sapporun commented 6 years ago

I got the same issue and my connection is fine. The launch of poe-trademacro is incredibly slow and the ctrl d searches takes ~2 minutes

Eruyome commented 6 years ago

https://github.com/PoE-TradeMacro/POE-TradeMacro/tree/dev

Test this version, it shouldn't solve any issue but it writes a file temp\timings.txt which logs all requests with url and some times/durations.

Example:

http://currency.poe.trade/
Downloading process duration: 407ms 
Data request duration     : 235ms 
Header request duration   : 172ms 
Validating result duration  : 0ms 

Also, there's the file temp\StartUpLog.txt, I don't think there's anything to see regarding this issue there, but who knows. I'm also experiencing slower requests to poe.trade as usual, but not by much.

Sapporun commented 6 years ago

only poe.trade seem to be slow according to the timings.txt

http://poe.trade Downloading process duration: 60469ms Data request duration : 187ms Header request duration : 157ms Validating result duration : 0ms

the rest looks like this

http://poe.ninja/api/Data/GetCurrencyOverview?league=tmpstandard Downloading process duration: 312ms Data request duration : 156ms Header request duration : 156ms Validating result duration : 0ms

Update: added files timings.txt StartupLog.txt

Zeronthe commented 6 years ago

Here are my startuplog and timings.txt. script initialization took a long time, same with a ctrl d search. ctrl q seemed to work fine with this version. My internet is as fast as it ever was, timings.txt StartupLog.txt

Here's my speedtest result to show it's clearly not my internet speed. Changing DNS seems to have no effect either.

siisjuu commented 6 years ago

StartupLog.txt timings.txt

Here's my logs. poe.trade and currency.poe.trade seems to be around 60000ms, while others are pretty normal.

Kutharsus commented 6 years ago

Tested again this morning at a different location (work) and everything seems to be working correctly again. I will update my timings at home later today, but maybe its fixed now?

I did however install AHK for the first time on this machine, along with the /dev version you provided so that may have something to do with my timings.

http://poe.ninja/api/Data/GetCurrencyOverview?league=eventhardcore Downloading process duration: 297ms Data request duration : 78ms Header request duration : 78ms Validating result duration : 31ms

http://poe.trade/search Downloading process duration: 1109ms Data request duration : 625ms Header request duration : 484ms Validating result duration : 0ms

StartupLog.txt timings.txt

Updated with timingDetail log - Working location timings.txt timingsDetail.txt

timingDetail - Non-working location timings.txt timingsDetail.txt

Eruyome commented 6 years ago

Download the same dev version again, this time show me temp\timingsDetail.txt.

Apparently the actual request (curl request + reading result into variable) is pretty fast for all of you while the entire download process takes 60s.

So I randomly added timestamps to the logfile, hopefully to see where the delay occurs.

Sapporun commented 6 years ago

timingsDetail.txt

Eruyome commented 6 years ago

That's odd...

            t4_1 := A_TickCount
            dataStart := A_TickCount            
            html    := StdOutStream(curl """" url """" commandData)
            dataEnd := A_TickCount          
            t5_1 := A_TickCount

t4_1 is triggered after less than 1ms from function start. The request downloading the data comes after that and takes 187ms (Data request duration). t5_1 is triggered after 30 seconds from function start.

Anyway, please repeat (download and run).

Eruyome commented 6 years ago

I can see from your logfiles that the poe.trade request gets repeated an additional time. The first time it fails (as in the response validates to "invalid"), this whole process takes 60s, the second request succeeds in a few ms. Every "request" actually makes 2 requests, one for the data and one for the request headers to check the http response code and such. Every request has a 30s timeout, which explains the logged timestamps, there are two 30s jumps.

The curl command from your StartUpLog file suggests the following: Normally there should be a bunch of headers, including the HOST header. This host header is known to cause the request to fail which leads to a second request without this header, which succeeds. This is an issue that only a few people seem to have and so far I don't know how to fix that.

I could try and completely remove the host header from all requests, at least for all requests to trade sites like poe.trade which are being used frequently. I don't know if removing this header will cause issues for other uses though. It may work for all of them which would greatly help people like you.

The real question is why curl failes for some users when using this request and how to fix it:

curl.exe "http://poe.trade" -ILks -H "User-Agent:Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "Accept-Encoding:gzip, deflate" -H "Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4" -H "Connection:keep-alive" -H "Host:poe.trade" -H "Upgrade-Insecure-Requests:1" --max-time 30 
Sapporun commented 6 years ago

timingsDetail.txt

Eruyome commented 6 years ago

Try again, I directly removed said host headers from the request.

Sapporun commented 6 years ago

It started very fast and is working fine now, will update with files StartupLog.txt timings.txt timingsDetail.txt

Hopefully it does not cause any issues for other people, anyway, it works!

Eruyome commented 6 years ago

Do me a favour and execute this in a windows console/terminal:

"C:\Users\Tandfen\Desktop\POE-TradeMacro-dev\POE-TradeMacro-dev\lib\curl.exe" "http://poe.trade" -ILks -H "User-Agent:Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" -H "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "Accept-Encoding:gzip, deflate" -H "Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4" -H "Connection:keep-alive" -H "Host:poe.trade" -H "Upgrade-Insecure-Requests:1"

and show me the result.

Eruyome commented 6 years ago

I now removed all host headers from all requests, it's probably redundant and should still work for everyone. We'll have to see if it causes any issues.

Sapporun commented 6 years ago

I tried the execution and got no result at all, the pathing is correct though No result

Zeronthe commented 6 years ago

Looks like this new version is working for me as well. Odd that it would just magically stop working for some users at a specific time, but at least this fixes it.

Eruyome commented 6 years ago

As I expected, thx.

machogriz commented 6 years ago

Just wanted to chime in. I was having the same issue. I just downloaded and used the dev version and it fixed it for me as well. Thanks for this!

joprut commented 6 years ago

Seems to be working fine for me too! Thank you for the help. I was sleeping, that's why I wasn't there to help.

Eruyome commented 6 years ago

Will make a release soon. Anyway, initialization is still somewhat slow because GGGs API is too slow.

Kutharsus commented 6 years ago

👍 Thanks Eruyome!! You never know how much you use this application until it stops working. I wonder how anyone functions without it :)

I was checking what updates came out on the 5th that could have possibly caused the issue and I came across the Adobe flash player security update. I don't think it has anything to do with the problem, but in the acknowledgments section, I saw this "Jihui Lu of Tencent KeenLab and willJ of Tencent PC Manager" IT WAS THEM!! Lol Link to the security bulletin - https://helpx.adobe.com/security/products/flash-player/apsb18-19.html

ost3g0 commented 6 years ago

Having the same issue as OP. 2 days ago this script has stopped working for any search related tasks. Also takes FOREVER to start up. Is there anything I can do / test ?

EDIT:

I downloaded that dev version above, worked and fixed my issues!

Eruyome commented 6 years ago

This dev version is already released as a new stable version.