ghost
commented
8 years ago Speaking of database, what are the thoughts around using NoSql?
Closed greenboxal closed 7 years ago
ghost
commented
8 years ago Speaking of database, what are the thoughts around using NoSql?
hails
commented
8 years ago Friend reporting in!
exectails
commented
8 years ago MySQL / NoSql
I do like the premise of other databases, like Postgre. Personally I like to use MySQL because of its availability. Even with its flaws, its the go to standard for many people, which means that many will already have a MySQL server around, either because they use it themselves or because some application they have does. At the very least most will already be familiar with it. This simplifies the setup of Melia.
I'd be interested in an embedded database, like SQLite, to remove that dependency entirely, though here you have the interoperabiliy problem, because on such a server you want the data to be available to potential Control Panels and the like. Same goes for some NoSQL solutions.
This can cause several inconsistencies among several world servers.
I understand the potential problem, but how much of a risk is that really? You have to keep all the files up-to-date of course, which requires some maintenance if you have several servers/channels on different machines.
On the other side you have servers with very few channels, possibly only one, or maybe two or three running from the same directory on one machine, where this isn't an issue. By moving the data to the db you complicate things for them, because the data has to be kept up-to-date with update files. Changing data in this scenario is also often times easier with files, because you don't have to operate a database tool.
Another point is extensibility. With files we can have a very clean override system via the user folder, which allows us to modify and extend any db, without having to create duplicate SQL tables (item_db2) or modify the core data.
For those reasons, and others that don't apply to Melia yet, we chose to go with JSON db files in Aura, our other server emulator. I guess the question is what's more important in Melia and what most users will need. I'd be inclined to guess that power users with dozens of channels over several machines aren't gonna be the norm.
External Dependencies / NuGet
You're saying we should use it, and we certainly could, but why? The only reason I can think of is so we have to clone less with Git, but you have to download the libraries anyway before compilation. By including them right in the files, we have one less dependency, and in some cases a step less to take to compile.
You don't depend on 3rd-party servers to be online and functional, you don't depend on having the tool or script to actually download them, in fact you don't even need internet if you have a clean copy of Melia lying around, you have everything you need.
Scripts
Why doesn't it work well? I admit, due to this being my first time using Lua like this there might be 1-2 issues initially, but what actual problems do you see?
I researched every way one can use Lua from .NET when I started working on scripts, and every solution lacked something. My Nr. 1 problem with the managed implementations I found was that they didn't support yield, and stateless NPCs are an absolute necessity in my opinion.
Another thing was performance, as LuaJIT vastly outperformed everything else.
Migrations
Yea, we discussed something like this for Aura a while back as well, when it was where Melia is now, but ultimately decided that SQL files are the easiest and most straight forward way to manage updates on an SQL db, instead of manually writing code to do the updating. Though we did add an auto-update system, where the login server runs the update files, so the user doesn't have to do it. The dev makes the changes, copies the queries to the update file, and that's it.
Any suggestions what a migration system might look like?
Unit Tests
Whaaat, we do have unit tests, didn't you see this? xD
You're absolutely right though, we should do that. I must admit that I tend to forget about them until I reach a point where I actually need them.
Packet Handling
I'd say this is a matter of preference. Personally I like to read the values and then use them. Adding another abstraction makes the parsing and the adding of packet handlers more complicated. You don't only have to add the handler, but also the struct for the data and the handler for the struct. What do you gain from it?
Async/Await
In general I agree, but I don't think we have to rewrite the connection classes now that they're done, that would be unnecessary work, just to use a newer technology. The current code works and is nicely simple.
A friend and I want to take care of these things and I want to discuss they here.
Glad to have you^^
greenboxal
commented
8 years ago Just to make sure nobody thinks I'm just a crazy guy throwing a lot of critics, let me just explain my background:
I came from the eAthena world, managed one of its biggest forks, Cronus, the Brazilian version. Saw and helped a lot of big servers to scale up.
Now I have a payment company, and again, what I see everyday, is the need to write code that is future proof and scalable. Even though directly doing it, affects the ease to use of the client, which is one of our main objectives here.
And this could sound harsh but, I think that you're committing the same mistake we all did back in eAthena.
The objective of simplicity to the user shouldn't affect the project core architecture. Simplicity and ease to use should be achieved with the right tools and interfaces.
Database
NoSQL is a lie. I use MongoDB in production today. I want really bad to replace it with Postgres as it supports JSON indexing now.
Installing Postgres is just as straightforward as MySQL.
Multi channel
In the current project stage you're right. When someone put up a server with thousands of players(relax, it will happen), you can't said that anymore.
Again, I come from a project that is exactly like this one, and already reached this stage I'm talking about. The problem is, when you build an architecture thinking on the first and not the second stage, it will fail.
When the second stage kicks in, you'll have no other option other than rewrite a potentially big part of the code base to support it(this also applies to packet handling, which I discuss above).
Dependencies
What do you gain "reducing" the number of dependencies? NuGet is bundled with .NET/Mono, is the native way of doing dependencies.
I think you're afraid of having a lot of steps to setup the project. But again, you can solve this with scripts.
Scripts
I think this is a premature optimisation. Could make sense because of yield, this is really important. But I think that native components should be the last resort.
Migrations
Coming from a web development background, we should just have a tool that keep tracks of existing migrations and applies new ones. This probably should come with an existing ORM.
Packet Handling
This is eAthena's packet handler, looks exactly like melia's one. Except that is already implements dozens of client versions.
Async/Await
You're absolutely right, that's just a nice to have. Although using it permits using the System.Threading.Tasks easily, which is beautiful.
exectails
commented
8 years ago The objective of simplicity to the user shouldn't affect the project core architecture.
I disagree. We are all users of a variety of programs, and we all know some that are just not pleasant to work with, because their creators didn't care about usability. In my opinion the very most important feature of any application is that it's easy and straight forward to use. That's what I expect from the tools I use, and that's what I want to offer the users of my own programs as well. If I come across a project where the first 10 installation steps are "install library x, install program y, setup z, install ...", it's an instant turn off for me. Why make it harder than it needs to be?
Now, a server emulator is a certain kind of beast of course, and you can't viably abstract everything away the common newbie might stumble over, but if there's no explicit need for something, is the replacement really justified if it comes at the cost of usability and interoperability?
In the case of databases MySQL is in many ways inferior, I agree with that, but there are advantages as well, as mentioned, and as long as the devs don't make any stupid mistakes in the relatively simple queries we need, there will likely never be a problem like the ones demonstrated in one of your videos.
I guess we will need more opinions on what's more important for Melia, as we two clearly have different views about the usability subject. Though this is a good thing, no decision in such a project should be unchallenged^^
In the current project stage you're right. When someone put up a server with thousands of players(relax, it will happen), you can't said that anymore.
Oh, I'm not saying that it won't happen, in fact I think I explicitly referenced those power users, what I'm thinking about is the vast majority of users that aren't operating dozens of servers and channels. The question is which audience Melia should primarily cater to, or what the best compromise is.
The issue you were talking about is not a problem that needs reconsideration at a later point, it's just something that makes live harder for one of two groups.
What do you gain "reducing" the number of dependencies? NuGet is bundled with .NET/Mono, is the native way of doing dependencies.
You're not reducing anything, the dependencies are still there, you're just adding more dependencies with NuGet o.o Namely, NuGet and the requirements it comes with, but what do you gain? Native or not, shouldn't you use the best "tool" for the job?
I think you're afraid of having a lot of steps to setup the project.
I'm not afraid of it, but like I said above, usability is very important to me. Why make a user take 3 steps if 1 is enough and those 2 steps can be removed/left out without any effort by the devs? (The numbers are just an example.)
I think this is a premature optimisation. Could make sense because of yield, this is really important. But I think that native components should be the last resort.
I agree, going native should be the last option, but of the given options it seemed like the best one. If someone has a viable alternative we should definitely discuss it.
ORM
In Aura we decided against that, because it was a little hard to model its data against classes without jumping through hoops. Do you have any concrete suggestions in this regard? I actually haven't actively used anything like that before, I like my queries, but this would theoretically make things easier for everybody.
This is eAthena's packet handler, looks exactly like melia's one.
Indeed. Basically. That's what packet handling looks like in many servers. The reason is that it's straight forward, easy to add to, easy to search, easy to maintain. I can't see a technical reason not to do it this way. Some would argue abstracting it further is cleaner, or safer, but is it really? And is the extra work worth it?
It's a little unfortunate that you don't explain why you think something should be done a certain way. What's wrong with the way eAthena, Melia, and other servers do it? I'm not questioning your experience, but without an actual reason for a change, at best it's a pointless task, that you do for the heck of it.
Partially related anecdote: On Aura we had a user who replaced everything he could with Linq extensions, in the entire source. He spent quite some time on that. I didn't really care at first, it wasn't my time he spent, until I came across code that I had to update or modify, and I didn't understand the Linq calls without "analyzing" them first, because they were abstract and complicated. There was no reason for these changes, except for him thinking Linq is superior, but in the end it hindered development.
using it permits using the System.Threading.Tasks easily, which is beautiful.
Absolutely, and nobody should use asynchronous features without awaits outside of the connections.
greenboxal
commented
8 years ago I think you didn't understand what I meant about usability. The project core should have the best architecture possible. It can be as complex as needed(not unneeded complexity though). But the user shouldn't need to care about how complex it is inside the core as the interfaces he uses aren't complex.
Usability comes in the interface, and not at price of core architecture.
One thing I learned during my life is that, the best way to create a project/product, is to think about the common necessities of the hard user, doesn't matter how complex they are, and then build it in a way the simplest user can use it. This is the real challenge of software development.
And of course, you don't need to build everything at day zero, but you need to develop something that can support that in the near future.
About packet handling, indeed, that what packet handling looks in many servers, but this kind of project isn't a simple server. It should care about different versions of packets because the following will happen:
There are some ORMs for .NET, EntityFramework and NHibernate are the most common ones I think. Never used them though.
Again, doesn't matter how many steps are necessary to setup the project, as long you abstract them from the user.
Why make a user take 3 steps if 1 is enough and those 2 steps can be removed/left out without any effort by the devs?
If you develop a software thinking only about the amount of effort required to develop it, you're doomed to failure. It's easy to just throw the dependencies on git, yet you're shipping binaries and not doing the correct way of handling them that the platform dictates. It's easy to just throw some SQL queries around and execute then, yet you will have a lot of problems handling any changes after that. It's easy to just handle events as soon as needed, yet you will have thousands of isolated lines of code that should be together and refactoring will be hard as hell.
(these are just examples, apply where suitable)
Architecture isn't something you do when you need it, it's something you do because someday you will need it.
And hell yes, we need more opinions here.
exectails
commented
8 years ago The project core should have the best architecture possible. It can be as complex as needed(not unneeded complexity though). But the user shouldn't need to care about how complex it is inside the core as the interfaces he uses aren't complex.
I did understand that, and I agree that the user shouldn't care about what's on the inside, as long as it's easy to use, but that's the thing, if something in the core or the design does influence the usability, you have to weigh the advantages and disadvantages.
In the case of our database discussion we have arguably few reasons not to use MySQL, aside from it "being bad", and we have arguably few reason to use Postgre. I don't see using MySQL costing us anything, in our relatively simple use case, but we do gain something. On the other hand a switch to Postgre would get us a more straight forward database, in theory, but the cost is that we switch to a "less popular" database system, that not as many people are familiar with, that not as many people already use, etc. It's one "+" against a "+" and a "-" from my point of view.
think about the common necessities of the hard user
If you have 995 users that only ever operate a single channel, and you have 5 users who have more, you would cater to the 5 users? And don't forget, ease of updating, because all data is in one place, is one thing, but such power users would potentially also be the ones to use heavy customization, which is more comfortable with the user folder system.
Shouldn't such a user find other solutions to load everything from one place? They do have that option, the other user group doesn't. For example, all data could be loaded from a network drive by all channels.
Packet Handling
Melia only supports one packet version because supporting more can get really complicated. We should probably discuss this, as supporting more would be beneficial, but if we don't, yes, the users would simply always need the latest Melia and a clean, updated TOS. Modified executables aren't an viable option in that scenario.
However, I don't see how that's relevant to the packet handling discussion. The packet reading doesn't change, just because you put the values into a struct.
Again, doesn't matter how many steps are necessary to setup the project, as long you abstract them from the user.
If you abstract them, they aren't setup steps anymore, of course they don't matter if they're gone xD
If you develop a software thinking only about the amount of effort required to develop it, you're doomed to failure.
You seem to misunderstand, I don't care about the development effort. What I was trying to say is that if you don't have to invest any time to make the life of your users easier, that's a win-win. "Why 3 steps if 1 is enough, leaving out those 2 steps it didn't cost you anything."
It's easy to just throw the dependencies on git, yet you're shipping binaries and not doing the correct way of handling them that the platform dictates.
I understand that that's your opinion, but I'd really like hear why o.o Why is NuGet superior to shipping the libraries with the repo in your opinion? I did try to tell you exactly why I find the opposite to be case.
It's easy to just throw some SQL queries around and execute then, yet you will have a lot of problems handling any changes after that.
I personally think queries are fine, they do exactly what would happen behind the scenes anyway, though I already did agree that an alternative wouldn't be a bad idea, to be on the safe side.
It's easy to just handle events as soon as needed, yet you will have thousands of isolated lines of code that should be together and refactoring will be hard as hell.
Not sure what you're referring to here.
Architecture isn't something you do when you need it, it's something you do because someday you will need it.
I hope you see that I always try to weigh the pros and cons of all known solutions, picking the one that seems like the best fit for everybody, devs and users alike.
Tachiorz
commented
8 years ago I think that using files to describe items, jobs, etc, is a bad idea. This can cause several inconsistencies among several world servers.
I agree with @exectails, files easy to use and power user can always use sshfs or samba to have them in single place.
In the case of our database discussion we have arguably few reasons not to use MySQL, aside from it "being bad", and we have arguably few reason to use Postgre.
Switch should be painless though, Postgre not so drastically different. Can't say much about it, only heard that it's good with clustering. But I haven't had any issues with mysql NDB, blazing fast selects, inserts are shit though.
A server modifies ToS executable and distributes it
Isn't it illegal?
Server owner now has his update path locked
Server owner would need to port his custom patches for newer melia version anyway.
Regardless of that. I'd like to have packet parsing independent of game version. To support old unprotected cbt executable, for RE purposes ^^
exectails
commented
8 years ago A server modifies ToS executable and distributes it
Isn't it illegal?
It would be illegal to modify and distribute modified executables, yea, though it might very well happen, although not on our forum. Maybe the question is if that should be of any concern to us.
Do we even want to support multiple versions? And if so, how do we handle changes in enums? The way we tried it in Aura ages ago was rather messy.
ovflowd
commented
8 years ago Hi @exectails after reading all those discussion, i will enter it saying some recommendations, in my personal opinion.
1. Multi Channel
You need do it. Like as @greenboxal said, maybe in some day, a ToS private server became full. Running the server in only one instance will be bad. Here is my recommendation: As far i see, Melia already work with separate modules.. The Channel Module, Login Module, and Client Module (as far i know), it's nice to represent the server in many modules, but this is against what Service Modules means. I mean, You don't need literally independent modules running in independent processes, you also forces to the server does external communication. Linux and MS Windows (since Win 7) allows a concept called App Service Component (or in other words, Topshelf), you can have a "top" service managing sub-services. Those services are "modules".
Here is an example:
Resuming, the Melia Core Engine can create multiple instances of any of the Modules, depending of XYZ variables (amount of users, memory amount, cpu, etc), and, the modules are services. You have an main service, that manages the entire Melia's Engine, if any of the other modules have failures, the Server Core need take computational decisions making (Turing Machine) (CDM).
2. ORM ORM in those days is something really important. ORM, creates State Machines, Models and Entity Machines, i mean, it's good for understanding, for process understanding and it's good for god sake. Since Objected Oriented Models make the life a lot easier for the Database Analyst. This author explains why You can also choose by using Entity Framework, or NHibernate + FluentNHibernate.
3. jSON Please don't convert your TXT Databases into JSON Databases. jSON is good, but bad for BigData and Big Volumes of "messages" if you have small computer hardware "power" (sorry my bad english). You can use jSON through REST services for Plugins API for Melia, RCON, or something else, also for anything, except for Database. CRV can really appear being fast, and for those type of data yes. But don't convert the Text File Database into jSON Database, here is a little example of the loading time for a jSON with more than 5.000 Object Entries (http://uims.uiot.org/Apps/Map) this Application (UIMS) is a Management System for IoT (Internet of Things).. IoT works with heavy amount of BigData. We use jSON as mesage body. But we idealize that you never will create a jSON array with more than 1K objects. Is massive. Other comparison, give a look in this.. (i know is stack overflow but..) TXT vs SQL
4. NuGET I highly recommend you to use NuGET, Mono and .NET Core actually are compatible with it. Also specifying static libraries that you putted directly on the solution, really breaks, what Continuous Integration means. That can make your software work strictly with a specific version of a library. What the difference by downloading the amount of bytes of those libraries when cloning/zipping the Repository or by downloading by NuGET? NuGET gives a lot of beneficies like Version Testing, and also it's good for you Software Versioning.
5. .NET Core and C# 6.0 Roslin I recommend you to give a look in C# 6.0 Roslin, that improves a lot of the C# coding semantics and expressions, also internal and security improvements in terms of memory overflow... I don't know if you know about .NET Core, here is: MS .NET Core .NET Core is totally different from .NET Framework, and also i don't recommend to use it now, only do a lecture. .NET Core is totally multiplatform and independent of System.* Dependencies. Also please don't use .NET 4.6, a lot of developers says that have a lot of problems.. really.
6. Don't code all by yourself. The world is continuous, don't try recreate the wheel, thousand of C# libraries exists to do anything that you want. Obviously is good know about the algorithms used behind libraries, but don't try recode everything. It's less time, and good to learn others developers "way to code"
7. Caching If you use NHibernate he do already all Entity Models caching in a good way. But this can be a novel depending in how the rest of the Melia's do caching.
8. Reactively Server I recommend you (if ToS uses Actions as Reactions (Client Action <> Server Action), to use the Reactive Software Pattern. That means, the server does when the client does. That is a novel for packet handling and socket channel listening. Think in the following way, why the server need await for other client incommings when he is already processing a packet for a deterministic heuristic process, example, client is sending USER A is WALKING FROM X TO Y, you don't need to wait other command before you answer to the Client an ACK. (Maybe the ACK can be compost made by ACK + Update User Statusses).. I really don't know how work in ToS but is in this way that works in Habbo. Client and server (NET Module) are synchronous, reactive, and based in NIO processes.
9. Logging? Is Melia already doing (staff actions, server module actions (queries, user log, etc) logs? It's good do the essential log.. You can do Data Mining and statistics for that. In a future Melia's can have a Machine Learning module, as example, based in the collected data, Saturdays at 9:00 PM more than 2K users log in in the server, so the server already prepares itself to handle this amount of users. Because is really better the server create new instances before a lot of packets goes in incoming, than after. Is really "easy" to the server take bad decisions or also crashing in the order of allocating computer resources in the same time that a big amount of data is processed. Data can be lost, the "process" core can be dumped, resulting in crashes. That really happens, trust in me. Ah, i recommend log4net.
10. Packet Handling Read and validate before store. Injections and Exploits and Vulnerabilities, also DoS can be easy done without proper incoming socket + data check before allocating/parsing/routing the Data. Also packet size check is something really important. A specific type of DoS attacks, called RUDY, can easily take down your application, by sending fragment of data. That works easily with HTTP Methods.
Sorry for my bad english. I hope i helped.
ovflowd
commented
8 years ago Observation.: I recommend this for service implementation: http://topshelf-project.com/
exectails
commented
8 years ago @sant0ro
1 - I'm a little confused what you're talking about here. Only one instance? Where did you get that? o__o I also wouldn't call our executables modules or services, because that's not what they are, they're simply separate processes. Personally I'm not even sure if I'd want those servers to run as services, since they aren't background processes in my eyes.
2 - While I don't agree that ORM is important, I do acknowledge that many people like it, and if someone wants to implement it for Melia, I won't stand in the way.
In the past we had various problems with the implementation of ORM in Aura for different reasons, be it complicated database structures or cross-platform compatibility, and since I preferred queries anyway we gave up on it.
To me, saying ORM is important is like saying C#-level OOP is important. It's surely useful in a lot of cases, but it's by no means the only way to do things, and there's other ways because not everyone prefers the one.
3 - You should really take a look at Melia^^ We're already using JSON, and we're using it for good reasons, as I described on the other issue. Your example on SO doesn't really fit our use-case, we only load the data once, we don't modify it, loading indexed data from a dictionary is surely faster than a MySQL query, especially if it goes through ORM, etc.
You seem to be under the impression that JSON is too slow, but we're loading thousands of objects in a second, uncached, and that's fine in my eyes. Not to mention that this is a one-time cost on start-up. It's a small price for the advantages it gives us in my opinion.
4 - I didn't criticize that you have to download stuff from NuGET because of the download size, but because of the fact that you do have to download stuff. And regarding the supposed compatibility on Mono, I just recently had the pleasure of trying to get NuGET to work on Linux, and it was really annoying. Sure, if it works it works, but if it doesn't you're basically screwed.
That can make your software work strictly with a specific version of a library.
I'm not sure I understand what you mean here. Your software will always strictly work with one specific version, no matter where it comes from. And if you don't use a specific version, there will always be the risk of compatibility breaking, and you having to update your program.
5 - We're not yet using C# 6 for compatibility reasons. In the past we've basically targeted the Mono version that comes with the current stable Debian version. People can install newer versions of course, but simplicity, usability, ease-of-use, etc.
I know about .NET Core, but last time I checked compatibility, it would've basically been impossible to port Aura to it, because of everything that's missing from it. Half the server would've needed a rewrite, or at least that's what the list of missing features felt like^^
I understand that you're only giving general suggestions based on your experiences, and I appreciate that, but it's confusing me to get random suggestions^^ I'm looking forward to you taking a closer look at Melia.
ovflowd
commented
8 years ago @exectails thanks, my big explanation was little bad since my english also is bad. But now i see your points.
exectails
commented
8 years ago Don't be discouraged from suggesting changes and features, generally I'm open to anything, and some of your ideas do sound interesting. I'm just giving my two cents and explain what the current implementation is like, and why.
ovflowd
commented
8 years ago No problem! I'm busy this week but next week will help certainly. (busy with university exams)
exectails
commented
7 years ago Closing for now, since this didn't go anywhere.
First of all, I need to congratulate all contributors on the effort. You're all doing an amazing work.
That being said, I have some points about the architecture of the project.
MySQL
I won't describe here why MySQL sucks(be sure to see the humor behind this one).
I suggest that we use Postgres as database.
Props Database
From my background as a developer at eAthena/Hercules/Cronus, the server software for Ragnarok "I", I think that using files to describe items, jobs, etc, is a bad idea. This can cause several inconsistencies among several world servers.
External Dependencies
The project should use NuGet to install its dependencies.
Scripts
Lua is a great scripting language, but the way it is used in the project doesn't work well. I think we should use a managed implementation of Lua.
Migrations
Although the project is in early stage of development, it should be using an automated migration system for dealing with database changes, rather than flat SQL files.
Unit Tests
Although impractical is this kind o project, unit tests should be written to every feature after validation against the official client. Also, the project should be running in continuous integration(Travis?).
Packet Handling
As you should expect packets to always change between versions, specially the its format, actions should be decoupled from the packet parsers, and should accept a neat structure instead of a raw packet.
Async/Await
Instead of using APM, the new async/await keywords should be used when handling asynchronous code.
Help
A friend(@hails) and I want to take care of these things and I want to discuss they here.