Engine should use CPU time and not Wall Time

[GoogleCodeExporter]

Wall Time is strongly affected by other process in the same machine and bots 
which use a lot of time might have a problem with that.
CPU Time should be slightly better.

http://nadeausoftware.com/articles/2008/03/java_tip_how_get_cpu_and_user_time_be
nchmarking#TimingasinglethreadedtaskusingCPUsystemandusertime

Please find attached a patch.

Original issue reported on code.google.com by Maxim...@gmail.com on 27 Aug 2010 at 1:14

Attachments:

• Engine.patch

[GoogleCodeExporter]

Maxime: I love when people attach a patch with a change request. You truly rock.

I strongly support using CPU time instead of wall time. If we can make it work, 
it would be great. I have a few questions:

(1) Are you sure that this patch does limit the bots according to CPU time? 
It's not clear to me that's what's going on. There are two processes involved 
here: the engine and the bot. Whose CPU time is actually being measured in the 
patch, the engine's or the bot's? We have to make sure that we're not limiting 
the bot to 1 second of the engine's CPU time, as that would produce seemingly 
arbitrary time limits.

(2) There are two engines in the codebase. One is the Java engine that is used 
by the contestants to test their code. The other is located at 
planet_wars/backend/engine.py and is used server-side to run the tournaments 
amongst all the submitted bots. Time control is not very important in the Java 
engine, it is more important in the Python engine. I don't know it it's 
possible to accurately measure the CPU time being consumed by a Python 
subprocess. I will have to look into this.

If somebody (probably Maxime) can produce a proof-of-concept Python script that 
accurately measures the CPU time being consumed by a Python subprocess and cuts 
it off after one second, then I think we can go ahead and switch to CPU time 
instead of wall time.

Original comment by cameron.jp@gmail.com on 29 Aug 2010 at 5:42

[GoogleCodeExporter]

I have added some experimental Python code under 
experimental/python_subprocess_cpu_time. master.py starts up client.py as a 
subprocess. After 1 second of the master's CPU time, the master shuts down the 
subprocess then terminates. The trick here is to change this so that the master 
shuts down after 1 second of the client process' CPU time WITHOUT modifying 
client.py.

Can it be done? Probably. I just don't know how yet. There is probably some way 
to query how much CPU time has been used by a process based on its PID. Within 
master.py, you can get the client's PID using "process.pid". If there is no 
Python API function for doing this, there is probably a shell command for it. 
You can use the subprocess module to launch that shell command with the 
appropriate arguments, then parse the response. It's dirty, but it just might 
work.

Original comment by cameron.jp@gmail.com on 29 Aug 2010 at 7:38

[GoogleCodeExporter]

I just thought of a potential problem with using CPU time to limit the bots. 
Someone could attack the contest by submitting a bot that does this:

while (1) {
  sleep(999);
}

AFAIK this bot would run pretty much forever while using a negligible amount of 
CPU time. In order to guard against this we would have to keep the wall-time 
limit in addition to the CPU time limit. However, we could loosen the wall-time 
limit.

Original comment by cameron.jp@gmail.com on 29 Aug 2010 at 7:43

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

May I make a small suggestion?
The current code in experimental/python_subprocess_cpu_time/master.py relies on 
forking, which is expensive. The code attached to this post does the same but 
without any forking at all. It relies on two libraries: twisted (For the async 
process management) and psutil (Used to manage the processes and get the cpu 
time).

It spawns the process and then polls the cpu time (user+kernel time) every 0.1 
seconds. This is preferable to a "while time.clock()" type loop because it is a 
lot more efficient, and the current code can manage any number of processes.

You can grab psutil here: http://code.google.com/p/psutil/ and twisted here: 
http://twistedmatrix.com/ - or just use pip/easy_install.

Another advantage to this is you can use psutil to poll the total memory the 
program is using, and perhaps kill it if it is too excessive.

Original comment by sharpbla...@gmail.com on 14 Sep 2010 at 11:56

Attachments:

• server.py

[GoogleCodeExporter]

sharpblade1: that looks pretty awesome. I see you're quite active on the 
project in the last day or so. Keep it up! I have to fix other issues ATM, but 
I will get back to this one, promise!

Original comment by cameron.jp@gmail.com on 14 Sep 2010 at 11:47

[GoogleCodeExporter]

My concern with CPU time instead of wall-time is that I want to be able to 
craft my bot so that it opportunistically does more work if more time is 
available, or at the very least to have some way to interrupt work if it is 
taking too long for some reason. 

How would I be able to tell how much time I have used/remaining?

Original comment by hage...@gmail.com on 28 Sep 2010 at 11:34

[GoogleCodeExporter]

would it be possible to make sure each bot gets a separate core and then wall 
time can be used to limit them fairly? 

Original comment by ben...@gmail.com on 10 Oct 2010 at 10:45

[GoogleCodeExporter]

Please fix this defect. Because of it bots lose battles with timeout.

My bot finishes the turn in 0.8 sec and that's still not enough!
When this was set to 0.9 I lost %90 of battles with timeout.
With 0.8 sec I lose 10% of battles with timeout.
I use gettimeofday() function to measure time.

Original comment by buratin....@gmail.com on 29 Oct 2010 at 7:16