Proposition: the external agent project structure

[GoogleCodeExporter]

I appended an illustration of the structure change I have in mind. From my 
understanding of this project, here are the pros, cons and aspects sorted 
by devastation:

#1- The AIModule.dll is loaded into Starcraft, is given full memory access 
with no way to detect cheating. This lowers BWAPI's credibility as a 
competition platform. Since only "legal" data is copied to the shared 
memory, no undetected cheating is possible from an AIModule.exe.

#2- Debugging is currently not possible, but even if it will be, the 
AIModule.dll will be debugged in the starcraft memory space. The 
AIModule.exe can obviously be debugged and stepped-through quickly and 
independently.

#3- A contra. Dynamic information classes like UnitImpl will be hidden in 
BWAPI.dll. Unit will have part of that functionality exported from the 
BWAPIAgent.dll. Essentially code will have to be splitted. This means no 
gradual transition from one structure to another, but a major revision.

#4- Since AIModule.exe's runtime is independent of Broodwars, you could 
close AIModule.exe, recompile and start again without even pausing the 
game in starcraft. Also, menu controling AIs become more managable.

#5- Starcraft multithreads and is forcing BWAPI to do the same, including 
synchronisation and instruction queueing, instead of simple calls to the 
draw functions (ideal case). There is no drawback in instruction caching 
over the bridge. In fact, if implemented properly performance will improve.

#6- A contra. AIModule's event handlers are usually called directly, but 
will now have to be queued over the bridge. Considering that the event 
handlers are not called very often, I just noticed that this is not much 
of a performance issue =D

In case you accept, I would like to join the team.

beeep ;)

Original issue reported on code.google.com by goo...@teabix.com on 23 Nov 2009 at 5:50

Attachments:

• plan.png

[GoogleCodeExporter]

This restructuring would be a lot of work, but I agree that BWAPI in its current
state has no way to detect cheating. BWAPI is open source so people can 
literally
copy and paste offsets and structures in from BWAPI code and get direct access 
to all
the data, so I agree that this redesign would be a big improvement.

Also a lot of people would like to have better debugging capabilities for their 
AIs,
so this would help them a lot too.

I don't think I have enough experience in pipes or shared memory to implement 
this by
myself, but if you're willing to take the brunt of the work, I'd be happy to do 
what
I can to help along the way.

Original comment by lowerlo...@gmail.com on 23 Nov 2009 at 11:15

• Changed state: Accepted

[GoogleCodeExporter]

Original comment by lowerlo...@gmail.com on 23 Nov 2009 at 11:16

[GoogleCodeExporter]

I know pipes, played long enough with shared memory, so after encapsulating 
each 
you'll have no problems building on that.

I currently plan making all the basic templates not interfering with BWAPI (so 
keep 
working like nothing happened ;) ), then while you're all asleep (our timezones 
seem 
to mismatch), weld it all together. Ofcourse the exact time of welding will be 
announced. Then help will be needed finetuning the whole deal and hunting down 
bugs.

Until then questions will be coming up, not really issues so where should I be 
posting those?

Thx for adding,
-Teabix

Original comment by goo...@teabix.com on 23 Nov 2009 at 11:49

[GoogleCodeExporter]

I'm usually in the IRC channel, QuakeNet #BWAPI, so feel free to IM me there, 
and I
try to check the BWAPI forum (
http://www.broodwarai.com/forums/index.php?showforum=46 ) and google group 
regularly.

Original comment by lowerlo...@gmail.com on 24 Nov 2009 at 12:47

[GoogleCodeExporter]

I'll post questionable decisions here, so you can prevent too much damage being 
done ;)

The Unit* (and Bullet* or whatsitcalled) pointers will be recycled once the 
unit is 
dead. The reason is to reduce the growth and size of the shared memory.

Now without adding any overhead, thus max performance, the interface will 
behave as 
follows:
Access to dead Unit*s will cause no access violation. Ofcourse, a correct 
programmed 
AI would not try to access a Unit* anyway, once it's onUnitHide'n or 
onUnitDestroy'ed. However buggy AIs could continue accessing it, and confuse 
that 
already dead unit with a new one, once the Unit* gets recycled.
Is this acceptable? (If not, I'll add a layer to ensure unchanged behaviour)

Original comment by goo...@teabix.com on 24 Nov 2009 at 10:55

[GoogleCodeExporter]

People are using a unit's pointer/address as a unique ID for the unit, so I 
don't
think we should change the behavior. However we may not need to store any info 
about
inaccessible (dead or invisible) units in shared memory - BWAPI_Agent.dll could 
check
to see if the unit is in shared memory, and if not return no information 
(0/null/false).

Original comment by lowerlo...@gmail.com on 24 Nov 2009 at 4:43

[GoogleCodeExporter]

The Unit* can still be used as an ID during the whole lifetime of the unit, it 
only 
recycles the IDs after death.

Original comment by goo...@teabix.com on 24 Nov 2009 at 6:30

[GoogleCodeExporter]

I learned the unit array has constant address and size, for the sake of 
savegames. I 
suspect the bullet array has constant address and size too, for the very same 
reasons. If you know how, please find out those properties, that would simplify 
the 
whole shared memory deal a LOT.

Original comment by goo...@teabix.com on 24 Nov 2009 at 11:41

[GoogleCodeExporter]

I'm not sure if this is a good idea or not. D:
If people need to modify their works because of a huge change in structure, it 
kinda
wouldn't go over so well.

Original comment by AHeinerm on 25 Nov 2009 at 1:56

[GoogleCodeExporter]

It's better to impove the structure now, people now will have to switch to the 
agent 
project structure anyway, than later, when they have full functioning AIs.

Just noticed, this is a fully closed layer => a simple class inheritance. Both 
funtionalities will be available, we'll just export both classes =).

Original comment by goo...@teabix.com on 25 Nov 2009 at 7:21

[GoogleCodeExporter]

I plan storing a command in each Unit structure, instead of in a separate array.
commands are flushed every frame, and per frame the AI can only issue 1 command 
to 
each unit. Warn me if i'm missing something here.

Original comment by goo...@teabix.com on 25 Nov 2009 at 9:00

[GoogleCodeExporter]

Players can issue more than one command in a single frame.

Original comment by AHeinerm on 25 Nov 2009 at 2:00

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I'm writing the commands in the Unit structure, not in the Player structure. So 
every Unit can have it's own command in a given frame.

Original comment by goo...@teabix.com on 25 Nov 2009 at 2:22

[GoogleCodeExporter]

One unit can have more than one command in the same frame as well.
Examples:
Move and Build Interceptor.
Train SCV and Set Rally Point.

Original comment by AHeinerm on 25 Nov 2009 at 3:22

[GoogleCodeExporter]

ofcourse! thx
it's so obvious now =D

Original comment by goo...@teabix.com on 25 Nov 2009 at 3:53

[GoogleCodeExporter]

Going forward much faster than expected, simply copying classes out of my own 
framework and adjusting them to meet BWAPI guidelines. This weekend I won't be 
home 
regularly, so this is the plan:

Monday the26, at 5am GMT time I'll start welding code.

Untill then there will be some issues, like this "marketing" issue about the 
names 
of the new dlls and classes:
- the former "BWAPI.dll" is no longer the main one. I'm not a good namer, best 
I 
could do is "BWAPI_Hub.dll" for the injectee and "BWAPI_Agent.dll" for the user 
loadee. Or maybe just "BWAPI.dll" for one of them?
- I heared something about the AI being able to navigate through menus and 
online. 
If that's planned, "AIModule" probably should not be called that anymore. 
Something 
like "Agent" would be more approptiare...

Original comment by goo...@teabix.com on 26 Nov 2009 at 12:57

[GoogleCodeExporter]

What is the reason for using pure virtual interfaces like BWAPI::Unit and 
hiding the 
implementations in BWAPI::UnitImpl?

Original comment by goo...@teabix.com on 26 Nov 2009 at 1:10

[GoogleCodeExporter]

I mean instead of using __declspec( dllexport )

Original comment by goo...@teabix.com on 26 Nov 2009 at 1:20

[GoogleCodeExporter]

Beats me. D:

Original comment by AHeinerm on 26 Nov 2009 at 2:04

[GoogleCodeExporter]

Ok I have a nasty habit of not telling the whole story ^^'
Here's what I intended to suggest:
the barrier between API and AI currently is the barrier between BWAPI.dll and 
AIModule.dll. Communication currently runs through virtual function tables.

After this change, the barrier will be the bridge. Any code that is on the 
Agent 
side of the bringe, hence ANY code within the BWAPI_Agent.dll is classified as 
"not 
trustworthy" and exists only so people do not have to directly work with pipes 
and 
shared memory.

Conclusion: how about we make the BWAPI_Agent.dll's code public. Just a bunch 
of .cpps and .hs. Users will just have to include those libraries into their 
builds, 
and no .dll will be needed, but also, no virtual tables will be needed, because 
every piece of code will run under same compiler, the user's compiler.
This way there will be no portability issues, we won't have to design portable 
code 
for BWAPI.dll, only for that little "static library".

This looks too perfect, please point out the problem that i'm missing here ;)

Original comment by goo...@teabix.com on 26 Nov 2009 at 2:20

[GoogleCodeExporter]

Doesn't seem like any problem. But portable code is in the same scenario now.
BWAPI.dll will still need portable code because some people want to compile it 
that way.

Original comment by AHeinerm on 26 Nov 2009 at 2:56

[GoogleCodeExporter]

1. @deathknight: Why do people need compiling the BWAPI.dll?

2. Talking about APIs, I'm not suggesting it, but there's another project 
solution 
available worth the time to think about: a conbination of
- C-style __stdcall functions and
- handles instead of classes
Just like the whole windows API actually. This one has another huge advantage: 
I've 
heared about many people programming their AIs in PHP, Java or other languages. 
All 
of them utilize proxy bots, but with a C-style API, you could use the 
BWAPI_Agent.dll in any language directly.

Original comment by goo...@teabix.com on 26 Nov 2009 at 6:15

[GoogleCodeExporter]

1. Maybe they want the latest revision? People do compile the SVN themselves, 
and
that's why portability was requested. I'm not doing all the porting for them 
though.

Original comment by AHeinerm on 26 Nov 2009 at 9:42

[GoogleCodeExporter]

1. was just debating the portability issue with a guy (appended).

About that concept, that seemed too perfect? I just found the catch: If the 
Agent 
side's code is all compiled into Agent.exe, any additional module will have to 
be 
compiled too, like BWAI or BATA. Technically this is not a bad thing, the 
bridge 
being the actual API and all the Agent code being just help to get along better 
and 
optimisation will just rock, but I don't know if the guys from those projects 
will 
be too happy about that.

Original comment by goo...@teabix.com on 26 Nov 2009 at 10:32

Attachments:

• chat.txt
• plan2.png

[GoogleCodeExporter]

"The Unit* can still be used as an ID during the whole lifetime of the unit, it 
only 
recycles the IDs after death."

Please don't change the functionality to this. AIs currently cannot determine
anything about enemy units when they are invisible. If they see a different 
unit with
the same Unit* as a old unit, they could infer that the old unit died at some 
point.
It shouldn't be difficult to make sure each unit that exists in a match has a
different ID.

"What is the reason for using pure virtual interfaces like BWAPI::Unit and 
hiding the 
implementations in BWAPI::UnitImpl?"

I designed this when I was first learning how to load and use DLLs so I went 
with
whatever I could get working at the time.

Also just as an idea, what if BWAPI.dll and the agent communicated purely via a
socket, rather than virtual memory? This would let people write AIs in any 
language
that supports socket communication. Its less efficient than shared memory, but 
people
are essentially already doing this with the Java, PHP, Python, and Haskell 
proxy bots
(though currently with incomplete protocols).

If we came up with an efficient standard socket protocol (like only transmitting
information for units that have changed in some way, and a compressed 
visibility,
hasCreep, and hasPower array) that everyone could use, it would make writing 
AIs in
other languages for the competition a more practical option.

On top of our socket protocol we could implement our "official" C++ API which 
would
work exactly as before so current C++ AIs would be unaffected.

Original comment by lowerlo...@gmail.com on 27 Nov 2009 at 1:23

[GoogleCodeExporter]

I was working on the .net port. Now I am working on the Parser universal
Client/Interop code generator. There are other efforts that are already nearly
feature complete, like the Python wrapper (using swig), but my generator could
generate code for any language much easier. It should also generate client side 
code
as well, even for any new version after release.
Basically I think adding a HasChanged() method that returns true if something 
has
changed for current unit in current frame could make the interface more 
efficient,
but I don't think it is of much use, because the volume of the data is not too 
high.
Please just don't make the header files more complicated by using defines, or 
other
obscure C++ features, and I think I should be finished within a week :)

Original comment by igalve...@gmail.com on 27 Nov 2009 at 7:39

[GoogleCodeExporter]

-"It shouldn't be difficult to make sure each unit that exists in a match has a
different ID."

I have a
class AIModule : public Agent (better names were appreciated)
If you derive from AIModule, each KnownUnit gets a Unit* wrapper, so the 
functionality keeps the current one. Deriving from Agent you'll have the pure 
interface.

-"Also just as an idea, what if BWAPI.dll and the agent communicated purely via 
a
socket"

I started the shared memory deal because of 1 reason: performance. There is 
just no 
way (100000 drawLines and 5000 drawTexts)/frame can be efficient over a socket. 
And 
heavy duty algorithms need all the spare time (mine did).

We should be able to add this feature afterwards. But socket timing compared 
with 
shared memory, the proxybot as separate project would be as efficient as a 
buildin 
feature. So I don't know if it's worth messing with the code, but we'll see.

Talking about portability:

The socket is not a purely portable solution as is. Code has to be written for 
decompression of all the data and ideally, presentation of it in target's 
language-
syntax. I'll just call that code "receiver" ok? In the end the effort of 
writing a 
socket receiver will be the same as writing a shared memory receiver. And that 
for 
every new language out there. The most portable solution is the C-style-API, 
which 
can be imported by almost any language with just a list of imported functions, 
and 
all you have to do is to write a nativeproxy.dll, with no major overhead.

I googled on the languages for support for socket alternatives.

PHP has pipe and shared memory support, without reverting to WinAPI. It has a 
Win32-
compatible api import, so importing native functions will be no problem.

Java, found an unportable solution for both pipes and shared memory, but hey 
we're 
all on windows here right? Java's native import uses the "native" keyword

Python I can't tell if its actually usable but I found code that opens pipes as 
filestreams, and a "mmap" for shared memory access. It also supports native 
function 
calls.

Haskell is majectic, and wasting that on sockets would be..a waste ;) I'm not 
experienced enough here to rate that pipe and shared memory code, but native 
function support is inluded.

Original comment by goo...@teabix.com on 27 Nov 2009 at 9:35

[GoogleCodeExporter]

@igalvelis: i don't quite get it, what interface are you porting, the current 
virtual table interface or the proxybot one?

@all: This chart didn't change for at least a day now, so it's very near the 
end-
solution. (appended). Ignore the Bullet tree, it's just a template.

Original comment by goo...@teabix.com on 27 Nov 2009 at 10:20

Attachments:

• plan3.png

[GoogleCodeExporter]

@teabix
I used to work on http://eis.ucsc.edu/StarProxyBot porting to .net (finished). 
As you
well know StarProxyBot is not feature complete.
So now I am working on a parser that would parse the BWAPI's exported .h files 
(parse
is not swig based). With that information it would generate a client and server 
code
similar to StarProxyBot's. Basically you could make it generate client code in 
any
language C#,Python,Haskell as long as you would update the generator.
At the moment the parser is not finished yet.

Original comment by igalve...@gmail.com on 27 Nov 2009 at 11:28

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

That socket issue has been nagging on me, what if localhost sockets are fast 
enough. 
I appended the benchmarks I used to make up my mind. Start the base, then the 
agent. 
Then the base will take 10 measurements of bytes/second bandwidth.
The Pipe benchmark is just there out of curiosity, it will be used for 
synchronisation and not for data transfer.

On this PC I get 50MB/s for socket transfer and 3000MB/s for memory transfer. 
The 
memory transfer is so fast (duh! =D) , that you can actually see the Agent not 
as 
separate module, but as part of BWAPI.dll.
The way I see it, if socket support gets implemented in a separate proxybot.exe 
file:
- it's debugging will be easier/possible
- no performance drawbacks
- easier implementing GUI showing incoming connections, status etc.

But if the question is to make proxybot part of the BWAPI project, i'd first 
ask all 
non C++ programmers out there, do they really have to be sockets? Native api is 
the 
better solution here.

Original comment by goo...@teabix.com on 27 Nov 2009 at 12:42

[GoogleCodeExporter]

forgot to append

Original comment by goo...@teabix.com on 27 Nov 2009 at 12:50

Attachments:

• Throughput.rar

[GoogleCodeExporter]

I don't think it would matter for the parser/generator what kind of code it 
outputs.
It can be memory transfers, or network sockets, unix or win32, http, com object 
or
any other kind of IPC. Whatever it is - it should be easy to adapt the 
generator. Of
course i should speak less about code that is not finished yet, and get to 
coding part.

Original comment by igalve...@gmail.com on 27 Nov 2009 at 1:19

[GoogleCodeExporter]

PS:It will take me a few days to finish my work, mainly because I have a 
deadline on
my work project now. Its 2 AM Friday, and I am still at work

Original comment by igalve...@gmail.com on 27 Nov 2009 at 11:56

[GoogleCodeExporter]

insight: The bridge extends BWAPI.dll to a second process, combines both 
processes 
into a single well-oiled machinery you might say. But the bridge itself is not 
an 
interface of BWAPI, it is merely a medium, like a network is for sockets. And 
just 
as it is for sockets, we have to provide functionality at the agent side, that 
wraps 
around the bridge protocoll and exposes a usable interface.

conclusion: This is a bridge issue, with everything about the protocoll and 
data 
ownership, for optimal machinery workflow. I fork the interface issue that got 
mixed 
up here to issue 142.

Original comment by goo...@teabix.com on 28 Nov 2009 at 12:59

[GoogleCodeExporter]

Why is Util an own project, instead of just an include file folder for the 
other 
projects?

Original comment by goo...@teabix.com on 28 Nov 2009 at 5:30

[GoogleCodeExporter]

I noticed people have problems understanding.
Appended timing table, hope It'll help. This is just a quick sketch so don't 
rely on 
accuracy.

Original comment by goo...@teabix.com on 29 Nov 2009 at 11:40

Attachments:

• plan4.png

[GoogleCodeExporter]

started. trying to put up a workable review as quick as it goes...

Original comment by goo...@teabix.com on 30 Nov 2009 at 10:26

• Changed state: Started
• Added labels: Component-Logic, Priority-Critical

[GoogleCodeExporter]

The whole Game obj... so much functionality, so confusing.

I make it work so that Agent can connect to the Hub, then commit. There you 
should 
get the idea of how the new SharedMemory based classes work, .. further details 
later when i commit...

Original comment by goo...@teabix.com on 30 Nov 2009 at 2:40

[GoogleCodeExporter]

I hope you don't mind I'm implementing issue 146 in the BWAgent namespace, 
(instead 
of Broodwar), so calls will be like BWAgent::drawDoodle(); Please tell if I 
should 
make it Broodwar::drawDoodle(), but then all other stuff, like Unit should be 
also 
in Broodwar manespace, Broodwar::Unit. Which also seems logical...

Original comment by goo...@teabix.com on 30 Nov 2009 at 2:47

[GoogleCodeExporter]

I commited the changes. The loading as it was has not changed in this review, 
but 
from now on, we should disassemble dll loading, and transfer the functionality 
to 
the bridge. Currently only connection works, so from now on there is still some 
way 
to go. 

There is also a new project "Interface Packer". It's goal is to generate 
low-fat .h 
files for the AIModule, to use with the .lib we provide. This project might get 
removed if we switch to __declspec(dllexport), but we might just keep it to 
generate 
cleaner code for publishing.

Any question please feel free to ask. In chat please write my nick, it's the 
only 
way I can hear someone wrote me, else I could not respond for hours.

Original comment by goo...@teabix.com on 1 Dec 2009 at 2:05

• Added labels: Priority-Medium
• Removed labels: Priority-Critical

[GoogleCodeExporter]

This is how it should look, when you scratched all together (appended)
The mismatch was created on purpose by changing the revision number and 
building 
only BWAgent.dll

Original comment by goo...@teabix.com on 1 Dec 2009 at 2:14

• Added labels: Milestone-Release1.0

Attachments:

• pipe_connection.png
• pipe_connection_mismatch.png

[GoogleCodeExporter]

Looks great. Can you add a project for example/test AIModule.exe?

Original comment by lowerlo...@gmail.com on 1 Dec 2009 at 7:15

[GoogleCodeExporter]

I'd love to start adding functionality to the bridge, but I'm not sure where to
start. If you could add functionality for something simple, like 
Game::getMouseX/Y,
that would be really helpful. I'll be on IRC in about 12 hours if you want to 
chat
and explain how the new system works.

Original comment by lowerlo...@gmail.com on 1 Dec 2009 at 7:57

[GoogleCodeExporter]

uhh, you thrive when I sleep =).
I added an ai project for testing. To be able to compile it:
- build BWAgent
- run Interface Packer.

Original comment by goo...@teabix.com on 1 Dec 2009 at 12:13

[GoogleCodeExporter]

strap yourselves in boys, 2.4 is the last working release in the old structure. 
Those who commit: to watch old code, create separate folder and checkout r1700, 
that 
still got loads. Since we're reimplementing the same functionality, we can just 
watch those functions in the old version and implement bridge equivalents

Original comment by goo...@teabix.com on 1 Dec 2009 at 10:58

• Added labels: Priority-Critical
• Removed labels: Priority-Medium

[GoogleCodeExporter]

If you are doing major changes to the BWAPI.dll interface it might be a good 
idea to
bump up the major version number.

Original comment by igalve...@gmail.com on 2 Dec 2009 at 1:01

[GoogleCodeExporter]

yeah definitely will make this bwapi beta 3.0 when its done

Original comment by lowerlo...@gmail.com on 2 Dec 2009 at 11:37

[GoogleCodeExporter]

lowerlogic
It seems you (or someone else?) have committed some methods that working over 
the
bridge. Would you like me to take a look at those methods and see if I can
auto-generate similar code for the rest of the methods for you? I am not a C++
expert, but given simple examples, maybe I could make something workable.

Original comment by igalve...@gmail.com on 4 Dec 2009 at 12:58