Support for other programming languages - e.g. Python

[pintman]

It would be great to have support for other programming languages to write snake bots. Python or a more general approach using a network protocol or STDIN/STDOUT would be a great option.

[cfr34k]

Hi pintman,

yes, that would be great indeed. The new back-end should make it relatively easy, but some work needs to be done nonetheless:

As a first step, the general possibility to choose a programming language needs to be implemented. From my perspective that could be done with a new database entry that specifies the language and is passed to the scripts in the docker4bots/ directory while building and running the user code.

Then a framework for each programming language needs to be added. The framework abstracts all the low-level communication such that the user only has to implement the init() and run() functions. So far we only have the C++ framework in docker4bots/spn_cpp_base/ which can be used as a reference.

---

Regarding your suggestion to use STDIN/STDOUT or a network socket for communication: Before I started writing the new framework I did some evaluation of different inter-process communication (IPC) mechanisms, including UNIX sockets, FIFOs and shared memory. It turned out that UNIX sockets and FIFOs become „slow“ when several hundred KiB are transferred, meaning that a transfer to another process and back can take several milliseconds. I blame the copy operations. I didn't test TCP sockets, but they are probably in the same range. Shared memory does not need additional copy operations (at least in C++) and is therefore very fast [1].

Spending several milliseconds just for pushing the data to the bot was not acceptable as we wanted to enforce 10 ms response time, so we chose shared memory for the data exchange and a UNIX socket for signalling (sending small packets is in the microsecond range).

Long story short, it would be best if any new framework could directly access the shared memory (should work for any language that can use mmap()). If that’s not possible, I'd suggest to write a small “proxy” (in C, maybe?) that reads the shared memory and passes the data to the process containing the user code (the proxy would be part of the framework and would run in the Docker container). That way the interface on the gameserver side would stay the same for all programming languages.

[1] I've uploaded my test program for POSIX shared memory at https://github.com/schlangenprogrammiernacht/io_speedtest . Unfortunately it seems like I didn't commit the FIFO version, but I think you could easily adapt it to use STDIN/STDOUT to transfer the data.

[pintman]

Thanks for the long and elaborate answer. At least python in a rather recent version (>=3.8) is able to handle shared memory ("System V Style Shared Memory") [1]

However, I don't know how well other languages support shared memory. Especially a Java-scenario in a VM seems to be more complex. But I am not an expert in this domain. Maybe memory-mapped files are an option.

Further it sounds like the format of the information that will be stored in the shared memory must be agreed upon. All data is just bytes. So what could be a suitable format? Is there a standard for serialized objects, something like protocol buffers[2]?

[1] https://docs.python.org/dev/library/multiprocessing.shared_memory.html#module-multiprocessing.shared_memory [2] https://developers.google.com/protocol-buffers/

[cfr34k]

Currently we use memory-mapped files on a tmpfs (in-memory file system), which is equivalent to POSIX shared memory [1]. I've also tried System V-style shared memory, but I found it hard to debug (the data is not simply accessible with normal programs).

The data structure is defined in [2] and uses only fixed-width types, so it should be well defined. The structure alignment (packing) is also specified to make it possible to calculate offsets in the structure without knowing the specific CPU architecture.

For Python, I found the example in [3], combining the mmap and ctypes packages to directly map file contents to Python objects. That should be quite efficient. I don't know about Java, but maybe something similar is possible there.

[1] http://man7.org/linux/man-pages/man7/shm_overview.7.html [2] https://github.com/schlangenprogrammiernacht/gameserver/blob/master/docker4bots/spn_cpp_base/spn_cpp_framework/src/ipc_format.h [3] https://blog.schmichael.com/2011/05/15/sharing-python-data-between-processes-using-mmap/

[pintman]

That approach looks rather promising. I am happy to be alpha tester as soon as the new feature is available. :)

[cfr34k]

It took a long time, but there is finally support for multiple programming languages in SPN!

Additionally to the “classic” C++ framework, we now have one for the Rust programming language. Also, there is now a “process” for adding more languages (see the documentation in the spn-meta repository for details), though it is not tested very well yet.

I will host an “official” instance of SPN at the GPN20 next week (May 19 to 22), so if you want to test feel free to join at https://schlangen.bytewerk.org :)