Querying Speed

[Austinb]

Seems to be some speed issues when doing queries. If queries finish faster than the current setting for reading from the socket the section is still hung until the read timeout passes (currently 800000 us). Need to speed this up to match the results from v1 without off-lining servers that are slow in responding.

[alissonlinneker]

He does not advance to the next connection without first passing through earlier.

How do I perform all connections in the array at once?

Tip: http://www.php.net/manual/pt_BR/ref.pcntl.php

Another example:

$host="localhost"; if ($fp = @fsockopen($host, 80, $errno, $errstr, 2)) { $out = "POST /port.php HTTP/1.1\r\n"; $out .= "Host: $host\r\n"; $out .= "Connection: Close\r\n\r\n"; $out .= "ip=123.456.789&porta=20"; fwrite($fp, $out); fclose($fp); }

[Austinb]

alissonlinneker,

GameQ has to send the requests in a linear fashion, that is each server's information is sent out in turn, but the listen side is meant to receive the responses in any order and be as fast as possible. I am still working on some tweaks on the listen side to make it faster. Also, for specific servers that require a challenge each server will have to wait for a challenge response before being able to continue. So for example A2S is probably slower compared to other protocols because it requires a challenge before you can send any other packets to the server.

That being said, at some point GameQ will become too slow for your needs. If you have a very large list of servers and need the responses in the timely manner you will have to develop some kind of system to spawn off parallel processes and/or do work in batches. That is beyond the scope of this project as GameQ just provides the ability to query many different servers and types of servers using one set of classes. GameQ should behave properly if you run it in parallel process as each set of connections is maintained internally. If you find this is not the case please open up tickets and I will gladly fix any issues.

If you end up developing some kind of parallel/spawning system in PHP and want to share it please let me know I will gladly link to it as an example other people can use, comment on and/or improve upon.

[alissonlinneker]

Hello ! :D

I got a solution.

See attached file "test.php".

See result:

root@server:/var/www/teste# php t.php [Thread #1]: The port 71 is close. [Thread #2]: The port 72 is close. [Thread #0]: The port 70 is close. [Thread #3]: The port 73 is close. [Thread #10]: The port 80 is open. [Thread #7]: The port 77 is close. [Thread #9]: The port 79 is close. [Thread #4]: The port 74 is close. [Thread #5]: The port 75 is close. [Thread #8]: The port 78 is close. [Thread #6]: The port 76 is close. [Thread #11]: The port 81 is close. [Thread #14]: The port 84 is close. [Thread #12]: The port 82 is close. [Thread #15]: The port 85 is close. [Thread #16]: The port 86 is close. [Thread #17]: The port 87 is close. [Thread #13]: The port 83 is close. [Thread #18]: The port 88 is close. [Thread #19]: The port 89 is close. [Thread #19]: The port 90 is close.

In my test, he swept 20 ports in less than 3 seconds.

Implement the gameQ please.

Install PHP on your PCNL. ./configure --enable-pcntl

You need the function "pcntl_fork."

You can only run on CLI or CGI.

Tnks :D

2012/5/23 Austin Bischoff < reply@reply.github.com

alissonlinneker,

GameQ has to send the requests in a linear fashion, that is each server's information is sent out in turn, but the listen side is meant to receive the responses in any order and be as fast as possible. I am still working on some tweaks on the listen side to make it faster.

That being said, at some point GameQ will become too slow for your needs. If you have a very large list of servers and need the responses in the timely manner you will have to develop some kind of system to spawn off parallel processes and/or do work in batches. That is beyond the scope of this project as GameQ just provides the ability to query many different servers and types or servers using one set of classes. GameQ should behave properly if you run it in parallel process as each set of connections is maintained internally. If you find this is not the case please open up tickets and I will gladly fix any issues.

If you end up developing some kind of parallel/spawning system in PHP and want to share it please let me know I will gladly link to it as an example other people can use, comment on and/or improve upon.+

---

Reply to this email directly or view it on GitHub: https://github.com/Austinb/GameQ/issues/24#issuecomment-5876530

[Austinb]

Hello,

I do not see an attachment. Do you have this on gist, pastie or something similar?

[alissonlinneker]

Sorry.

I'm sending it back.

See please.

2012/5/24 Austin Bischoff < reply@reply.github.com

I do not see an attachment. Do you have this on pastie or something similar?

[alissonlinneker]

See: www.gameservs.com.br/Thread.rar

[alissonlinneker]

What's new?

[alissonlinneker]

News?

[Austinb]

I havent had anytime to really work on this with other work in the way. I will try to carve out some time to bring in the branch I have been working on to speed up the calls.

[SamuelGold]

We hope you will find a bit time, problem still actual.

[Austinb]

I just pushed up a quick change to allow you to change the speed at which stream_select waits. This will allow you to set it how high or low you want. The loop still operates under the maximum timeout set. Using the original code from v1 produced too many instances of data being absent from time to time. I would rather it be accurate and a bit slower than missing data and a bit faster. As always better patches are welcome.

[Austinb]

On a side note: I am NOT working on any kind of offloading or batch based processing for GameQ. As I said before it is beyond the scope of this project. Feel free to make a branch and add a pull req and I will gladly look at it.

[SamuelGold]

Thank you, it works!

[Austinb]

Please provide any feedback on the branch. Once I am ok with how it is operating and people are using it I will merge it into the v2 master.

[SamuelGold]

I think, before merge, need increase default value 'stream_timeout' to 150000, because some servers can located far away from script. In my case ping from script to servers about 1-2ms, but someone can have 100 or more.

[Austinb]

As for the default I would like people to test with different settings/servers/whatever to see if you can find some kind of floor at which point the value of stream_timeout affects the actual data response (i.e. at some low point server A does not always return all the data).

I plan on making some changes/tweaks before it is merged. Need more testing and a lot more feedback from people using the program.

[SamuelGold]

I use it for monitoring about 200 servers, works fine. But, as I wrote before, all servers a very near mon script.

[zeagon]

This patch is really good Austin. 50000 may be a bit low for timeout when querying many different server locations though. I would recommend a default of 100k - 200k, this would require quite some tests

Even in my really bad setup the speed increased several seconds.

My server is in USA 50k setup: US based servers seems fine and return data as expected. EU based servers, germany, france, netherlands are not reached. 100k setup: Bosnia, Russia, Ukrain, Germany still down 150k: Most of Europe, china, Australia seems ok, need more tests. Bosnia, Ukrain unstable, Russia, Japan not reached. 200k: Bosnia unstable, Unkrain ok, Russia, Japan not reached 250k: Bosnia stable, Russia, Japan not reached 300k: Russia, Japan not reached ( still a speed increase ) 500k: Russia, Japan not reached ( nomore speed increase ) something might be wrong with these servers

Overall, at this point of tests i would recommend 200k as default. 90% of servers around will properbly answer while still having a noticable speed improvement I called GameQ 9 times on 1 page( complete init - while not conform, good for testing ~ 2 second faster), so i would guess at 200k a normal included GameQ will have a very good speed improvement

[Austinb]

Yea I am looking for a good default value that most users will find acceptable. With the setting being changeable users can tweak it how they want if they feel it is not fast enough or having trouble reaching some servers.

I was thinking some where around 200-400ms as the default (original was 800ms before the patch) so that you can reach most places in the world by default and tweak it down if you are doing everything locally or only in one region of the world (i.e. only European servers).

I am going to leave this open for another week or so depending on feedback and then I will merge it in. Thanks for all the feedback thus far.

Just fyi while testing some of the protocols when migrating to v2 I did notice servers in ru/jp not responding. I always assumed this was some kind of network setup or firewall restriction.

[SamuelGold]

What about idea make a 2 requests? First with 100k. If server not answer, then we make second with 800k as origin.

[SamuelGold]

Ideally, we need measure ping to server and put it in database :D Once in a day remeasure this value. I think it give the best result.

[Austinb]

That is an interesting idea but would require a lot of internal reworking to give it that logic. Right now the sockets_listen method doesnt care if the server responds or not, its job is to listen for the data coming back and put it somewhere and nothing else. Its something that would be possible but would obviously take more time to make work since it adds another level of complexity to the whole open socket -> send data -> listen for data -> do something with response/no response work flow.

As far as ping, not all of the protocols respond with ping so in those cases you would have to do something manually such as use a shell to ping the server. That is a good idea as well. The program is already built to allow server level overrides in the settings when you pass the server information to addServer(s)() but it is not yet used anywhere. Also they do not pass to the sockets side either.

If you are interested in these being added please feel free to make new issues for each of these and any others for that matter and I can look into them as time allows.

[SamuelGold]

"Ping" I mean not ICMP request. I mean like: $starttime=microtime(true); //here we send a status packet to server //recieve answer $ping=microtime(true)-$starttime; $ping+=30000; //here save ping for this server to database

In my case I enough 50k, just want give some Ideas.

[firefly2442]

This does not work for me. I tried with the default 50K and with 150K and neither worked. I get:

The server did not respond within the specified time.

When querying only two servers. When I switch back to the "v2" branch, everything is fine.

[SamuelGold]

Try set up 800k.

[firefly2442]

Ahh, my apologies, it was something else.

[alissonlinneker]

PHP Multi-thread? :D

[Austinb]

"On a side note: I am NOT working on any kind of offloading or batch based processing for GameQ. As I said before it is beyond the scope of this project. Feel free to make a branch and add a pull req and I will gladly look at it."

[Ruok2bu]

How many servers should be batched when dealing with hundreds of server queries every few minutes?

[Austinb]

To be honest I have not used the script to query massive numbers of servers. I would do a trial by fire, meaning try say 20. If that works then bump it up until the time it takes to receive all the responses is too long for your needs/wants.

I would be curious to know how many work in a batch on average. Also dont forget you can tweak some of the settings in the patch attached to this issue so if you have all your servers say in the US you can probably tweak down the timeout to 200ms to make it faster.

[Ruok2bu]

You said earlier that the script runs the queries in a linear fashion. If i set up multiple queries that run at the same time, would they interfere with each other (if run from the same installation of gameq)?

[Austinb]

Yes it depends on the query. For example with Source servers you have to auth before you can query for anything other than the basic info. So this query has to be sent, read, applied to the other query packets and then those packets are sent back out and read in at any order.

In all cases the packets for each server are sent out in a linear fashion meaning that for each server in the list the packets are sent as the list is iterated over. When listening for the returning info its is done so that the script is looking for any response from all the packets sent out.

When running parallel calls or processes GameQ should behave fine as all the connections and such are handled internally so there should be no overlap (assuming you do not use the same instance of $gameq for each "child"). If you find otherwise I will be glad to fix any issues.

[Austinb]

Is anyone having any issues with this patch? If not I will make some tweaks and merge it into v2 master. Any testers please let me know of any issues or ideas in regards to this path.

Will merge it asap, have not had a lot of time to devote to this project as of late.

Thanks

[zeagon]

Had this run a while now with 350k, and as of now, nothing has been reported back from my users

[SamuelGold]

Patch ok, in my case I use 40k now, but all servers in one rack. My system uses many single requests, so it works more slow that gameq v1 :( I think good idea will make 2 requests ways: with timeout like in v2, and without it for single requests.

[Austinb]

So making a single server call va multiple calls is slower? Is it across all games or only specific ones? More info the better please.

[SamuelGold]

Yes, at the end of single req we have stream_timeout, 40k in my case. Code for debug:

    protected function sockets_listen()
    {
        global $do_echo_debug;
        // Set the loop to active
        $loop_active = TRUE;

        // To store the responses
        $responses = array();

        // To store the sockets
        $sockets = array();
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 54111 " . microtime() . "\n";
        // Loop and pull out all the actual sockets we need to listen on
        foreach($this->sockets AS $socket_id => $socket_data)
        {
            // Append the actual socket we are listening to
            $sockets[$socket_id] = $socket_data['socket'];
        }
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 54112 " . microtime() . "\n";
        // Init some variables
        $read = $sockets;

        // This is when it should stop
        $time_stop = microtime(TRUE) + $this->timeout;
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 54113 " . microtime() . "\n";
        // Let's loop until we break something.
        while ($loop_active && microtime(TRUE) < $time_stop)
        {
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 54114 " . microtime() . "\n";
            // Now lets listen for some streams, but do not cross the streams!
            $streams = stream_select($read, $write, $except, 0, $this->stream_timeout);// тут тормозит
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 54115 " . microtime() . "\n";
            // We had error or no streams left, kill the loop
            if($streams === FALSE || ($streams <= 0))
            {
                $loop_active = FALSE;
                break;
            }
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 54116 " . microtime() . "\n";
            // Loop the sockets that received data back
            foreach($read AS $socket)
            {
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 54117 " . microtime() . "\n";
                // See if we have a response
                if(($response = stream_socket_recvfrom($socket, 8192)) === FALSE)
                {
                    continue; // No response yet so lets continue.
                }
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 54118 " . microtime() . "\n";
                // Check to see if the response is empty, if so we are done
                // @todo: Verify that this does not affect other protocols, added for Minequery
                // Initial testing showed this change did not affect any of the other protocols
                if(strlen($response) == 0)
                {
                    // End the while loop
                    $loop_active = FALSE;
                    break;
                }
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 54119 " . microtime() . "\n";
                // Add the response we got back
                $responses[(int) $socket][] = $response;
            }

            // Because stream_select modifies read we need to reset it each
            // time to the original array of sockets
            $read = $sockets;
        }
if ($do_echo_debug==1) echo "ganeQ-GameQ.php 541110 " . microtime() . "\n";
        // Free up some memory
        unset($streams, $read, $write, $except, $sockets, $time_stop, $response);

        return $responses;
    }

Result for CS 1.6:

ganeQ-GameQ.php 54111 0.36596700 1352967157 ganeQ-GameQ.php 54112 0.36598000 1352967157 ganeQ-GameQ.php 54113 0.36599600 1352967157 ganeQ-GameQ.php 54114 0.36600500 1352967157 ganeQ-GameQ.php 54115 0.36602700 1352967157 ganeQ-GameQ.php 54116 0.36603500 1352967157 ganeQ-GameQ.php 54117 0.36604200 1352967157 ganeQ-GameQ.php 54118 0.36605500 1352967157 ganeQ-GameQ.php 54119 0.36606300 1352967157 ganeQ-GameQ.php 54114 0.36607300 1352967157 ganeQ-GameQ.php 54115 0.36608900 1352967157 ganeQ-GameQ.php 54116 0.36609700 1352967157 ganeQ-GameQ.php 54117 0.36610300 1352967157 ganeQ-GameQ.php 54118 0.36611200 1352967157 ganeQ-GameQ.php 54119 0.36612000 1352967157 ganeQ-GameQ.php 54114 0.36612800 1352967157 ganeQ-GameQ.php 54115 0.36614300 1352967157 ganeQ-GameQ.php 54116 0.36615000 1352967157 ganeQ-GameQ.php 54117 0.36615600 1352967157 ganeQ-GameQ.php 54118 0.36616500 1352967157 ganeQ-GameQ.php 54119 0.36617200 1352967157 ganeQ-GameQ.php 54114 0.36618000 1352967157 ganeQ-GameQ.php 54115 0.36965500 1352967157 ganeQ-GameQ.php 54116 0.36968800 1352967157 ganeQ-GameQ.php 54117 0.36969600 1352967157 ganeQ-GameQ.php 54118 0.36971100 1352967157 ganeQ-GameQ.php 54119 0.36972000 1352967157 ganeQ-GameQ.php 54114 0.36973300 1352967157 ganeQ-GameQ.php 54115 0.36976400 1352967157 ganeQ-GameQ.php 54116 0.36977400 1352967157 ganeQ-GameQ.php 54117 0.36978000 1352967157 ganeQ-GameQ.php 54118 0.36979500 1352967157 ganeQ-GameQ.php 54119 0.36980300 1352967157 ganeQ-GameQ.php 54114 0.36981200 1352967157 ganeQ-GameQ.php 54115 0.36982800 1352967157 ganeQ-GameQ.php 54116 0.36983600 1352967157 ganeQ-GameQ.php 54117 0.36984200 1352967157 ganeQ-GameQ.php 54118 0.36985500 1352967157 ganeQ-GameQ.php 54119 0.36986200 1352967157 ganeQ-GameQ.php 54114 0.36987000 1352967157 ganeQ-GameQ.php 54115 0.40996500 1352967157 ganeQ-GameQ.php 541110 0.41001800 1352967157

Result for Gamespy 3:

ganeQ-GameQ.php 54111 0.24597000 1352967923 ganeQ-GameQ.php 54112 0.24598800 1352967923 ganeQ-GameQ.php 54113 0.24599700 1352967923 ganeQ-GameQ.php 54114 0.24600200 1352967923 ganeQ-GameQ.php 54115 0.24625100 1352967923 ganeQ-GameQ.php 54116 0.24626500 1352967923 ganeQ-GameQ.php 54117 0.24627200 1352967923 ganeQ-GameQ.php 54118 0.24628900 1352967923 ganeQ-GameQ.php 54119 0.24629800 1352967923 ganeQ-GameQ.php 54114 0.24630900 1352967923 ganeQ-GameQ.php 54115 0.28647000 1352967923 ganeQ-GameQ.php 541110 0.28652200 1352967923

Result for GTA-SAMP:

ganeQ-GameQ.php 54111 0.18061400 1352968036 ganeQ-GameQ.php 54112 0.18063200 1352968036 ganeQ-GameQ.php 54113 0.18065000 1352968036 ganeQ-GameQ.php 54114 0.18065900 1352968036 ganeQ-GameQ.php 54115 0.18161400 1352968036 ganeQ-GameQ.php 54116 0.18163900 1352968036 ganeQ-GameQ.php 54117 0.18164400 1352968036 ganeQ-GameQ.php 54118 0.18165100 1352968036 ganeQ-GameQ.php 54119 0.18165600 1352968036 ganeQ-GameQ.php 54117 0.18166100 1352968036 ganeQ-GameQ.php 54118 0.18166500 1352968036 ganeQ-GameQ.php 54119 0.18166800 1352968036 ganeQ-GameQ.php 54114 0.18167400 1352968036 ganeQ-GameQ.php 54115 0.18182200 1352968036 ganeQ-GameQ.php 54116 0.18183600 1352968036 ganeQ-GameQ.php 54117 0.18185200 1352968036 ganeQ-GameQ.php 54118 0.18187300 1352968036 ganeQ-GameQ.php 54119 0.18188200 1352968036 ganeQ-GameQ.php 54114 0.18190000 1352968036 ganeQ-GameQ.php 54115 0.22199500 1352968036 ganeQ-GameQ.php 541110 0.22204500 1352968036

[Austinb]

I think you are confusing the different timeout values. With this patch there are now 2 different timeouts being used:

1. The big overall timeout (known as $this->timeout) is to keep servers that don't respond or bad ip:port combinations from hanging the script while waiting for a completed response from a server that is never going to respond. Basically this timeout is used as a fail safe in the while() loop for the stream_select(). This was changed from v1 and I will explain in my next response.
2. Is the new one with this patch known as $this->stream_timeout. This timeout value only affects the amount of time stream_select() will wait before releasing the script to continue on. So this value affects the time being spent between 54114 and 54115 in your testing. Making this value smaller uses more cpu of the machine the script is being run on.

More information:

The stream_select() timeout affects only how long it holds the script while listening for responses on the various open sockets that were created when the query packets were sent out to the server(s). Once this timeout is reached the script moves on and if there is data sitting in $read then work is done on that data and loops back but if no data the loop continues back around and the stream_select() listens for more data for the same amount of time as before. This continues until all of the data is received from all of the sockets stream_select() is listening on. Without the $this->timeout this loop could go on for a long time (i.e forever) if servers did not respond in a timely manner. Obviously preventing this behavior is a must.

Please note that $this->timeout has no bearing on the actual speed of the while() loop, it is $this->stream_timeout that affects the speed of the actual loop. If $this->stream_timeout is set to 40000 (us) and the server being queried responds within 40kus of the request being sent then the loop should only go around once then break out of the loop. Stream_select() will return immediately if there is no more data to be read on the sockets. The timeout only works if there is pending data on the sockets to be read. But if it takes say 80kus for the data to come back 2 or more loops will happen because stream_select() has not received all of the data back on the sockets. So the longer it takes for the server(s) responses to be received the longer the loop is running. In most cases the response time can not be affected.

So if you are only pinging servers within the same rack then setting the value of $this->stream_timeout low makes sense because in theory all of the servers should respond within a short amount of time (hence the patch). But this is not always the case. Query responses are still affected by the server's load the game server is running on. If the server is under high demand it is possible the responses could be slower than if the server only had a few players on it. So if a response takes 100kus normally it could be double or more than that if the server is under heavy load.

GameQ was never built to be a real-time querying system. Something like that requires direct access into the the game server's information by some other means. The query protocols for most game servers are not built for real-time information gathering.

[Austinb]

Here is the explanation for the change in the while() loop from v1 to v2.

The responses were not always reliable over the internet querying the same servers with the way the timeout was being calculated. The old way iteratively reduced the $tv_usec part of the stream_select() function until the value of $this->timeout was reached. Basically it used a big timeout value for the first pause on stream_select then reduced it from there making the pauses shorter and shorter until it ran out of time. What I found when going back through this code was that the same server did not always respond in time with the way the select was done in v1. The change I made in v2 was to make the responses more reliable by setting the timeout to a static value which was then changed to be a specific option in this patch. What was perceived as a faster call in v1 was the script actually bailing out early in some cases before the server had sent back its response fully.

Use https://github.com/Austinb/GameQ/blob/v1/GameQ/Communicate.php#L161 as reference for the next sections.

So let's say you set your timeout to 2 seconds for the timeout in v1. On the first loop the timeout of stream_select() would more or less be 2000000us. So for the first loop stream_select() would hold for 2 seconds waiting for data then move on and allow the script to process. Now when the loop comes back around to use stream_select() the timeout value of $t is either very small or negative. So if it is really small stream_select() only holds for a short bit then releases. The read does its thing and when it loops back round it ends because now $t is surely <= 0. What if the server's response takes 2.5 seconds to complete? You are now missing response data. What if the response takes 1 second? You have just waited an extra second you could have been using for something else like more socket reading for other servers.

This is where the problem lied in the v1 socket_listen code. The responses were not always reliable at any short timeout value especially when querying servers over the internet. So say you increase this value to 5 seconds. Now you probably have data from every server you query within that first loop but a lot of them probably would have finished faster. This is why I changed the usage of $this->timeout to be strictly a fail-safe and set the stream_select() to some static value. The value of timeout had way too much impact on the reliability of the server's response being complete.

All that being said I am always looking for ways to improve the script so if you have ideas post up an issue or fork and send a pull request.

[Austinb]

Merged in the patch. Any subsequent problems should be added as new issues.