Database and connectivity

[bopke]

I believe that one of the most important things are good choices of database and methods of communication between instances and proxies. For start it could be enough to just assume that there is only one proxy server and number of minecraft server instances that is declared once and never changes, but we can do better than this. It would be perfect to make it possible for server owners to add more sectors if they feel like it, so we shouldn't just trust spigot server with what it believes is player equipment, coordinates etc., we should store it in separate database that will be the single source of truth. We need to pick good database engine for it and it should be great to have only one database engine required for the project to run, so we wouldn't one day wake up with 3 different databases for different purposes, and some other software that we depend on. I think about 3 options, but I'll be happy for other sugestions in this thread:

1. MongoDB - I think that we wouldn't crave for lots of relations in database for such project, and not so strictly structurized documents nature compared to super strict nature of sql databases that would force us to maintain migrations will speed up development process, especially at the start of the project.
2. PostgreSQL - It is probably the best free SQL database we can get. Stricter nature of tables with relations will probably make it easier to maintain project at later stages of developemnt.
3. MySQL - I believe it is most common database engine used my minecraft plugins. It has the same upsides as PostgreSQL, but is a little bit slower and has less functionality than PostgreSQL, but it will be easier for server owner to maintain only one database for most of the plugins.

Also, another thing to consider, a bit related to databases (so its in the same discussion issue), is communication method for plugins to exchange data and notify other nodes about events that they should be aware about. I'm not 100% sure, but it may be achievable by usage of Plugin channels, but this looks very primitive and I believe that use of pub-sub solution may be the better idea. One of such solutions may be redis pub/sub, but if you know something that may suit better - let us know in this issue!

[WcaleNieWolny]

As much as I agree with the "one source of true" part I do not think that pub/sub is fast enough. For the pub/sub thing this already been done by one of the members of our community here From what I heard is would't be ideal for sector but if we want to consider Redis we should ask him for the pros and cons of this solution.

I was more thinking about using proprietary tcp solution that would be fast and extremely scalable when it comes to the dynamic mode

Another problem with using a standard database is speed. If we would to send data to it and receive it on every transport this would create a huge lag. I think we should use it one - on join. Everything later would be happening via spigot data and communication protocol that we would decide on

We also could just use redis - It is fast due to it's in memory way of storing data and it is also persistent. If we decide to use it we can just save a lot of time on pub/sub and also have a great database to relay on

[sadcenter]

As much as I agree with the "one source of true" part I do not think that pub/sub is fast enough. For the pub/sub thing this already been done by one of the members of our community here From what I heard is would't be ideal for sector but if we want to consider Redis we should ask him for the pros and cons of this solution.

I was more thinking about using proprietary tcp solution that would be fast and extremely scalable when it comes to the dynamic mode

Another problem with using a standard database is speed. If we would to send data to it and receive it on every transport this would create a huge lag. I think we should use it one - on join. Everything later would be happening via spigot data and communication protocol that we would decide on

We also could just use redis - It is fast due to it's in memory way of storing data and it is also persistent. If we decide to use it we can just save a lot of time on pub/sub and also have a great database to relay on

bruh redis is a memory cache, not database for storage, databases like postgresql are okay when used in a good way. and yes, message brokers such as redis or nats are okay for sector project.

[WcaleNieWolny]

After a long discussion on bookitty we have decides to do an abstraction that would support easy changing of broker system and for start implement it with redis

[thatbakamono]

For persistent data storage I think we should just use MySQL, we can change it later on if it poses too much trouble. There isn't a point in discussing that subject for days or even weeks when we don't have anything at all so we can't just test our assumptions and pick the best option available.

[CDFN]

I'm not very experienced in distributed programming for Minecraft, but I can express my opinion on options you considered here.

1. Redis should not be used as primary database, however it works well for storing non-important data or data which we can void any time. Keep in mind redis stores all data in memory which can impact server very hard. In my project you mentioned, Redis was used mainly as message broker.
2. Regular databases are faster than you may think. Using them only on join makes no sense however caching frequently used data totally makes sense. Designing plugin to be as asynchronous as possible defeats argument of "slow" database. Of course, considering that you're creating distributed system, it can not be fully asynchronous as e.g. when player gets from one server to another, the data has to be waited for, otherwise data loss could happen. In such cases you want to use message broker so you won't end up with polling database every some period to detect whether or not data is there yet.

Summarizing, in my opinion abstraction layer for message brokers (with default implementation of Redis) + PostgreSQL is the way to go. When performance of Redis will not be satisfying (highly doubt) you can always implement other solutions such as Kafka, RabbitMQ.

[bopke]

redis stores all data in memory which can impact server very hard

Just curious, I believe you're talking about performance, how could it affect it? If used right, redis should be just distributed cache and something wrong must happen to make it perform poorly

Summarizing, in my opinion abstraction layer for message brokers (with default implementation of Redis) + PostgreSQL is the way to go.

I think that abstraction layer for all external services should be created, for any case in which we discover some service is restraining us hard in the future

[thatbakamono]

redis stores all data in memory which can impact server very hard

Just curious, I believe you're talking about performance, how could it affect it? If used right, redis should be just distributed cache and something wrong must happen to make it perform poorly

Summarizing, in my opinion abstraction layer for message brokers (with default implementation of Redis) + PostgreSQL is the way to go.

I think that abstraction layer for all external services should be created, for any case in which we discover some service is restraining us hard in the future

Yeah, you are right, we should do an abstraction layer for all external services.

[CDFN]

Just curious, I believe you're talking about performance, how could it affect it? If used right, redis should be just distributed cache and something wrong must happen to make it perform poorly

I was talking about server in host meaning, not in Minecraft server meaning. Redis taking up a lot of memory can affect other services running on the same host making them run less performant/crash etc. That was my argument against using redis as primary database.

[WcaleNieWolny]

So to recap we are starting with redis and MySql as the first implementation of the abstraction layer?