Open ntoskrnl4 opened 3 years ago
Based on my understanding, there would be more then one main server thread, with seperate ones for chunk loading, generation, and similar. Would this affect networking?
We are likely going to use Queues for the communication between net threads and other threads, where there will be no affect with the multithreading.
Based on my understanding, there would be more then one main server thread, with seperate ones for chunk loading, generation, and similar. Would this affect networking?
The current setup I have in mind is that networking will be its own, single, separate process which runs an asyncio loop to manage connections into the server. Once players successfully log in, the networking thread (well, process) communicates with the rest of the server using internal events and multiprocessing.Queue
s.
I'm not sure if we have the main server thread in place yet, but other server threads shouldn't have any affect on networking since that's in a completely different subprocess.
Management of Player
objects can be done in either the networking thread or server thread, so that is yet to be decided depending on whichever makes it more performant or easier.
For what it's worth, I'd be interested in an asyncio port of quarry itself. The vast majority of quarry's code doesn't use twisted, but I don't understand asyncio well enough to port the bits that do use twisted to asyncio.
Hey, cool to see you interested in this!
Lack of asyncio is definitely what we saw as the main problem, hence this branch. Without asyncio, we're a fair bit more limited in how we can write the server in terms of server thread / network communication, since it's otherwise blocking.
I've been completely blindsided recently due to life events and schoolwork, but I should be free now to work on this more.
If you'd like, I can also attempt a port of quarry to asyncio, although I personally am much less familiar with Twisted.
I'll study your implementation and see if I can glean some understanding of asyncio. I previously gave up on asyncio when I read this. But it's time for another look!
You may still find the quarry.types
and quarry.data
packages useful, as they don't include any twisted stuff.
I think I've got my head around asyncio
, and I have a simple implementation of the Minecraft protocol. Here's a working server:
import asyncio
from mncrft.types import unpack_varint
from mncrft.server import listen
def status(version):
return {'version': {'name': '1.16.4', 'protocol': 754},
'players': {'max': 100, 'online': 0},
'description': {'text': 'blah'}}
async def login(protocol, profile, version):
while True:
buff = await protocol.read_packet()
ident = unpack_varint(buff)
print(ident)
# etc
async def main():
server = await listen('127.0.0.1', 25565, status, login)
async with server:
await server.serve_forever()
asyncio.run(main())
Code coming soon.
Here we go: https://github.com/barneygale/quarry-ng
Using minecraft-data for packet structs now, too.
May change a fair bit as I build on it. Let me know how you get on!
The current backend Quarry is good for quick and simple server implementations and handles nearly all of the networking for the server. However, I think we could benefit from developing our own networking backend, based off of asyncio.
AsyncIO in Python is significantly more performant in a single thread than a similarly written synchronous server implementation and can match the performance of a threaded server, all without the overhead of using separate threads (thread-safety, overhead, limited access to resources, etc). Other advantages of splitting from Quarry is that we are not bound to any of its constructs; for example its server classes, its API format, and its version support.
Per the starting discussion on Discord, here's my thoughts about how asyncio networking could be implemented:
That's my initial thought as to how async networking could be done. It passes a lot of the server functions into it but unless the server has 200 players on it I don't think it'd be too bad. Then again, we could also go with a simpler implementation where only incoming data events and outgoing data events are handled, and everything is done in the main server thread. This should also be something we consider in addition to my thoughts above- it would probably make the overall server implementation simpler and easier at the cost of more work in the main server thread.
Either way we take this, I am down to support the development and oversight of this project on a new branch of the repo (for example named
asyncio-networking
), as I have almost two years of prior asyncio experience developing complex Discord bots.