search

Civcraft / BetterShards

A bukkit mod that supports Civcraft's plugin and sharding.
BSD 3-Clause "New" or "Revised" License
8 stars 8 forks source link

No readme #2

Closed suirad closed 9 years ago

suirad commented 9 years ago

I was looking through different civ plugins to find one to start comitting to. But, as i have been away from Civcraft, reading the sub occasionally, what is better shards and its goal? @ttk2

suirad commented 9 years ago

@rourke750

rourke750 commented 9 years ago

Sure let me help you out. Do you want to be taking over this or just looking to do some stuff here and there?

ttk2 commented 9 years ago

well he asked what it was, better shards goal is to allow Civcraft to shard out and expand, Mercury will handle passing events between servers, but better shards will handle portal creation and management between the worlds.

On Mon, May 25, 2015 at 3:51 PM rourke750 notifications@github.com wrote:

Sure let me help you out. Do you want to be taking over this or just looking to do some stuff here and there?

— Reply to this email directly or view it on GitHub https://github.com/Civcraft/BetterShards/issues/2#issuecomment-105312676 .

suirad commented 9 years ago

@rourke750 i am more or less just looking to regularly commit to civ plugins. Was just looking around for a starting point and i figured asking questions would help others in the same situation. I don't have the time to take on this project completely.

@ttk2 is there a reddit thread that specifies "portals; passed events" or could you go into more technical detail?

ttk2 commented 9 years ago

Sure, some overlap exists with Mercury (a separate plugin designed to handle cross server message passing for plugins that require it once we shard) though, so to start with just the scope of this plugin.

What Better Shards needs to do is allow the creation of portals between different shards and handle detecting when players enter portals and then forwarding the player and the updated contents of their inventory to the appropriate portal on another server.

Sharding is already implemented using BungeeChord, it links servers and allows for players be sent between them, its really a separate server that forwards the data from the client to different MC servers, so from the MC servers perspective a new player logs in, from the players perspective they triggered some event, Bungeechord told their clients it was a world change (think nether to overworld) and hooks them up to the new server. The details of this are not our problem except that it works but inventories and locations are not sent between servers, as far as the server you forward a player to knows its a new login.

This means shards will have to handle sending inventories and ensuring the player comes out in the right position on top of managing the portals themselves as structures and what we need to protect them from in game vandals.

I think that should mostly cover the scope of BetterShards, now on to how it interacts with Mercury, Mercury is a message passing architecture we came up with, each plugin gets its own 'channel' so if on one server a player deletes a Namelayer group namelayer from that server will fire an event, Mercury will catch it, and broadcast that over the Namelayer channel, the copies of mercury running on the other servers hear this message and fire an event to have the other copies of namelayer invalidate their cache, so they then load the definitive copy from the database on the next call.

Bungeechord has its own message passing system (as you would expect) but because of the scope of the bungee server it can only send messages by piggybacking them on connected players connections, so no players, no messages, while this probably would not be an issue for us we decided to make Mercury to make sure of it and simplify things a bit with the channels idea.

I am sure by now you can see that BetterShards will call mercury which will communicate with the other server which will tell better shards on the other end what it needs to know to get the player and their inventory in the right spot .

So yes, non-trivial.

On Mon, May 25, 2015 at 5:57 PM Darius Scott notifications@github.com wrote:

@rourke750 https://github.com/rourke750 i am more or less just looking to regularly commit to civ plugins. Was just looking around for a starting point and i figured asking questions would help others in the same situation. I don't have the time to take on this project completely.

@ttk2 https://github.com/ttk2 is there a reddit thread that specifies "portals; passed events" or could you go into more technical detail?

— Reply to this email directly or view it on GitHub https://github.com/Civcraft/BetterShards/issues/2#issuecomment-105331917 .

suirad commented 9 years ago

Wouldn't the use of a specialty made sort of routing server(non gameserver) facilitate communications between servers? It could free up a lot of unnecessary processing by the game servers along with the game server itself not having to juggle multiple connections(could not scale well on the higher end). Based on how you explained it this would be my approach:

I would love to lend a hand in making that routing server and coordinating with rourke or whoever and making it happen.

ttk2 commented 9 years ago

Well we did do something like that we where just a lot lazier about it.

The point of mercury is that its job is to do message passing by capturing events on a local server. This minimizes the changes to the local plugins and means only the mercury programmers need to be worrying about all the threading issues.

But we never specced that mercury had to do the job itself. In fact why reinvent the wheel at all? There exist several fast functioning frameworks for message passing in large multiprocess applications. Currently we are using reddis pub/sub which has the channel concept all figured out for us.

If we wished we could even put it live. Decide we didnt like reddis and move to a different message passer, or even roll out own if we wanted to waste time, and just upgrading mercury on all the seevers would be the only required change.

On Tue, May 26, 2015, 12:14 AM Darius Scott notifications@github.com wrote:

Wouldn't the use of a specialty made sort of routing server(non gameserver) facilitate communications between servers? It could free up a lot of unnecessary processing by the game servers along with the game server itself not having to juggle multiple connections(could not scale well on the higher end). Based on how you explained it this would be my approach:

  • Game server use bettershards for serialization/generation of a message.(synchronous)
  • Then that message would be made into a final for immutability and thread safety and passed to Mercury.
  • On a separate thread, Mercury would further serialize the message data into a packet friendly buffer and fire the packet(s) to the routing server. (here you could dynamically adjust the size of packets sent based off of server metrics to not saturate actual gameplay bandwidth)
  • The routing server will be made in either a high speed (c like) or highly concurrent / scalable (erlang/elixir[preferred], 99% uptime would be awesome) language. The routing server would receive the packet(s) along with a few bytes of header info like destination, plugin, ect. From there the data is sent appropriately.
  • From there the data is received and deserialized in reverse order by the receiving server(s)

I would love to lend a hand in making that routing server and coordinating with rourke or whoever and making it happen.

— Reply to this email directly or view it on GitHub https://github.com/Civcraft/BetterShards/issues/2#issuecomment-105396405 .

suirad commented 9 years ago

Right so pretty much Mercury will be used as a cross mc server messaging framework plugin. And it seems like bettershards will be the addon plugin that establishes the portals and their use, utilizing mercury to transfer inventories. My previous suggestion of making it use an intermediate routing server may have overlooked the required scope of its use for Civcraft. I suppose the thought of if there are 5+ servers, you could give say, humbug a quick config change via command, and it would send a message saying "BROADCAST:humbug:irongolemspawns=disabled" and from there the gameserver will just use one connection instead of having several individual connections to each game server, but still able to replicate a message to all other servers if needed. I guess i was thinking more towards supporting currently unknown scaling requirements. Not trying to convince you to do it my way, just sharing reasons.

rourke750 commented 9 years ago

Actually BetterShards will handle inventories by serializing custom inventory objects and passing them over mysql to be picked up by the other server. And we could do the broadcast thing via mercury, we would just have to program the mods to do it that way.

On Tue, May 26, 2015 at 11:03 AM, Darius Scott notifications@github.com wrote:

Right so pretty much Mercury will be used as a cross mc server messaging framework plugin. And it seems like bettershards will be the addon plugin that establishes the portals and their use, utilizing mercury to transfer inventories. My previous suggestion of making it use an intermediate routing server may have overlooked the required scope of its use for Civcraft. I suppose the thought of if there are 5+ servers, you could give say, humbug a quick config change via command, and it would send a message saying "BROADCAST:humbug:irongolemspawns=disabled" and from there the gameserver will just use one connection instead of having several individual connections to each game server, but still able to replicate a message to all other servers if needed. I guess i was thinking more towards supporting currently unknown scaling requirements. Not trying to convince you to do it my way, just sharing reasons.

— Reply to this email directly or view it on GitHub https://github.com/Civcraft/BetterShards/issues/2#issuecomment-105555354 .

erocs commented 9 years ago

Mercury abstracts away the details from the plugins. We can start with Reddis pub/sub (which I thought we had decided we weren't going to use due to a recommendation) and we can evolve it later if the need arises. Using Reddis does what you say, as all the pub/sub messages would pass through a central Reddis server. Then it's just a systems question over which server can handle the load.

We don't anticipate the network traffic from any of these plugin interactions to have any impact on the overall throughput. Transferring chunk updates from each shard server through each BungieCord proxy out to the player will chew through most of the bandwidth. It'd be great if the proxy could itself catch bulk-chunk update requests and retrieve the chunks out of the Orebfuscator cache for forwarding to the players.

ttk2 commented 9 years ago

That would be a procollib level optimization, but its not out of the question, it would just need to know when to invalidate its own cached chunks.

Like Erocs said the messaging framework itself is not super relevant unless we run into scaling issues, reddis pub/sub made implementation easy at the cost of potential issues far down the road (we would hopefully need more than 10k players before it really caught up with us) and even then we could modify only mercury and have a drop in fix.

Akka looks appealing though, but you get the general idea at this point, if we can do it better now that's great, if we cant then we made it easy to fix later.

On Tue, May 26, 2015 at 11:22 AM erocs notifications@github.com wrote:

Mercury abstracts away the details from the plugins. We can start with Reddis pub/sub (which I thought we had decided we weren't going to use due to a recommendation) and we can evolve it later if the need arises. Using Reddis does what you say, as all the pub/sub messages would pass through a central Reddis server. Then it's just a systems question over which server can handle the load.

We don't anticipate the network traffic from any of these plugin interactions to have any impact on the overall throughput. Transferring chunk updates from each shard server through each BungieCord proxy out to the player will chew through most of the bandwidth. It'd be great if the proxy could itself catch bulk-chunk update requests and retrieve the chunks out of the Orebfuscator cache for forwarding to the players.

— Reply to this email directly or view it on GitHub https://github.com/Civcraft/BetterShards/issues/2#issuecomment-105588126 .

suirad commented 9 years ago

Im probably gonna fork mercury and make a version of what i have in mind, variety in implementation could prove useful for you guys. Il spawn up the repos later tonight. (am currently deployed, so tonight is EST+7hrs)

ttk2 commented 9 years ago

What is it that you have in mind?

On Wed, May 27, 2015, 12:07 AM Darius Scott notifications@github.com wrote:

Im probably gonna fork it and make a version of what i have in mind.

— Reply to this email directly or view it on GitHub https://github.com/Civcraft/BetterShards/issues/2#issuecomment-105758106 .

suirad commented 9 years ago

On my github, i have forked Mercury to my account. I have also set up another repo suirad/Venus. On that repo i am developing that routing server i mentioned above in elixir. once i have that mostly running, i will go to my fork of mercury and adapt it to communicate with that routing server. Once i have it going it will work like this:

Mercury will be the api used by plugins to send information to the same plugin on different servers, or a different plugin on a different server, or a different plugin on the same server, or the same plugin on all servers. I plan on making it in a way that it can be adapted per use by the plugins using it.

An example use: Prison pearl on server A and B. a player on server A logs in and triggers an alt association. Server A broadcasts the associaton to all servers. Server B responds to a directly saying that players ip is associated with 2 pearled alts. Server A kicks the player for alt ban.

erocs commented 9 years ago

Why write a custom routing server when public frameworks exist? We really don't want to maintain more code.

suirad commented 9 years ago

The routing server is planned to be made as something that can exactly fill the current need while also being generic enough to be useful in future and upgrading previous code. Also as i said previously, it is ment as an additional option, i am going to make it regardless, doesn't mean it has to be used. But i am going to make it and i am going to maintain the routing server as needed.

So mostly, because i can and want to. Making something custom runs the same risks as using something public, if it's no longer maintained, someone else will have to do it.

erocs commented 9 years ago

No worries there if it's for your own enrichment. :) It could be interesting for us anyway because there is some need for a middle tier server to handle some server logic.

As you say, maintenance is a big concern for us, especially with devs coming and going so often. The more code lying around unmaintained the bigger the headache, which is why we'd rather stick to maintained public projects if possible and push our occasional contribution for our own needs.

On Thu, May 28, 2015 at 8:21 AM, Darius Scott notifications@github.com wrote:

The routing server is planned to be made as something that can exactly fill the current need while also being generic enough to be useful in future and upgrading previous code. Also as i said previously, it is ment as an additional option, i am going to make it regardless, doesn't mean it has to be used. But i am going to make it and i am going to maintain the routing server as needed.

So mostly, because i can and want to. Making something custom runs the same risks as using something public, if it's no longer maintained, someone else will have to do it.

— Reply to this email directly or view it on GitHub https://github.com/Civcraft/BetterShards/issues/2#issuecomment-106402310 .

suirad commented 9 years ago

I mean, in all honesty, my mercury and the original could probably be merged.

Oh thought of something else, with the router you could hook up non gameserver applications to it as well. Those applications could interact with plugins directly. Which could be used to provide some interesting statistics, among other uses.

Oh and also, obviously barring real time stuff(maybe, depending on the application), you could very well make pseudo plugins in another language using the router as an interop.

Clearly im pretty excited about it Lol.

suirad commented 9 years ago

I've finished my updates for today on Venus (https://github.com/suirad/Venus/). Its coming along nicely. If you want to keep up with the progress, take a look in the lib folder, each source file(.ex) is commented at the top with a todo list of what i have left and a small description of what it does. The most important part to note for someone looking to work on mercury is; look at the serverman file, and look for the function:

def handle_packet(data) do
    #return {:ok, plugin, action} or {:error, reason}

end

Here you will find message structure for exactly will be accepted by the routing server. I should have it finished by tomorrow after laying it out and testing it.

ttk2 commented 9 years ago

what language will the routing server be in?

On Thu, May 28, 2015 at 4:37 PM Darius Scott notifications@github.com wrote:

I've finished my updates for today on Venus ( https://github.com/suirad/Venus/). Its coming along nicely. If you want to keep up with the progress, take a look in the lib folder, each source file(.ex) is commented at the top with a todo list of what i have left and a small description of what it does. The most important part to note for someone looking to work on mercury is; look at the serverman file, and look for the function:

def handle_packet(data) do #return {:ok, plugin, action} or {:error, reason} end

Here you will find message structure for exactly will be accepted by the routing server. I should have it finished by tomorrow after laying it out and testing it.

— Reply to this email directly or view it on GitHub https://github.com/Civcraft/BetterShards/issues/2#issuecomment-106607303 .

suirad commented 9 years ago

Im making it in Elixir. That fact alone will make it out perform most others. Elixir is a sort of extention language to Erlang. Erlang is a highly concurrent, fault tolerant language that pretty much most telephone systems are made of. The server can have 99% uptime, even through upgrades with its hot code reloading systems. In most cases, the erlang vm outperforms the jvm. Look em up.

There is actually an erlang minecraft server someone made a while back that i think got up to protocol like 1.6 or so. I think it would be cool to eventually update it, to think, a minecraft server capable of scaling to thousands of clients would be crazy.

ttk2 commented 9 years ago

If that would be possible it would be worth doing. But supporting all the 1.8+ stuff and a mod api would be a crazy amount of work.

Our main concern is still having people around who know how to work on these things.

Have you looked at akka? Could you try and make it a drop in replacement? Or at least minimal change.

On Thu, May 28, 2015, 11:47 PM Darius Scott notifications@github.com wrote:

Im making it in Elixir. That fact alone will make it out perform most others. Elixir is a sort of extention language to Erlang. Erlang is a highly concurrent, fault tolerant language that pretty much most telephone systems are made of. The server can have 99% uptime, even through upgrades with its hot code reloading systems. In most cases, the erlang vm outperforms the jvm. Look em up.

There is actually an erlang minecraft server someone made a while back that i think got up to protocol like 1.6 or so. I think it would be cool to eventually update it, to think, a minecraft server capable of scaling to thousands of clients would be crazy.

— Reply to this email directly or view it on GitHub https://github.com/Civcraft/BetterShards/issues/2#issuecomment-106684034 .

suirad commented 9 years ago

Once i have Venus finished, i will try and make a usage identical copy of it in scala/akka. Scala is actually one of the languages i was considering when i was looking at other languages to learn. I ended up picking up Elixir and the erlang vm. Reason being is, while scala is made to run on the jvm(and apparently .net, news to me); the JVM nor CLR are made for concurrent applications. So no matter how well your applications are made, your runtime won't be doing any favors with optimization to your task. Just my 2c on the topic.

It would probably be appropriate to continue this discussion on the forum. http://forum.civcraft.co/t/venus-development/84 Closing ticket