search

Driftwood2D / Driftwood

Driftwood 2D Tiling Game Engine and Development Suite
http://tileengine.org/
MIT License
24 stars 1 forks source link

🔥Multithreading?🔥 #164

Closed seisatsu closed 7 years ago

seisatsu commented 7 years ago

There are some places we may be able to use some simple Python multiprocessing to get things done much faster, places where race conditions are not a concern and the engine doesn't need to do anything until all threads complete. For example, reading/instantiating tiles into an area, or the draw cycle. Configurable use of threading for certain blocking-type tasks could really speed things up, and not be hard at all to write.

seisatsu commented 7 years ago

I can't actually get it to make a difference. Do Python threads actually use additional CPU cores?

seisatsu commented 7 years ago

Nope, they do not, not with the threading module. The multiprocessing module may do it though.

seisatsu commented 7 years ago

Doesn't look like you can pass class instances between threads. So, at least for the Tile issue, this probably won't be helpful.

pmer commented 7 years ago

Haha, yeah.

“Getting rid of the GIL is an occasional topic on the python-dev mailing list. No one has managed it yet.” -- https://wiki.python.org/moin/GlobalInterpreterLock

If we want multithreading I think we will need to port Driftwood to something like Jython or PyPy. 😕

pmer commented 7 years ago

We can try the multiprocessing library but I expect it to be pretty slow since it probably serializes (maybe with pickle?) all Python objects sent between the processes.

Don't get your hopes up.

pmer commented 7 years ago

Nevermind, I should look before I speak. It looks like multiprocessing supports shared state between processes. This will be very fast but will require us to perform manual synchronization between processes.

Feel like brushing up on your synchronization primitives? 😉

pmer commented 7 years ago

Dang, we may not be as lucky as I was hoping.

The multiprocessing.Queue class will pickle all data we send through it, as I suspected.

https://github.com/python/cpython/blob/master/Lib/multiprocessing/queues.py#L112-L113

And the shared state objects multiprocessing.Value and multiprocessing.Array objects can only hold primitive C values.

https://github.com/python/cpython/blob/master/Lib/multiprocessing/sharedctypes.py#L25-L33

What we need to speed up area loads is the ability to share unpickled native Python objects between processes. Thus, the multiprocessing library can help us with raw computations (maybe useful sometime in the future?) but will not help us if our problem is trying to allocate lots of objects.

EDIT: Looks like you already discovered all this! I guess I'm just confirming it for myself, too.

pmer commented 7 years ago

If we want to port the engine to another Python implementation, let's open a new issue for that. For now I'm going to close this issue.