Persistent between restarts

[ConstableBrew]

Retain server state so that the world isn't reset when the server is restarted.

[Blecki]

Most of the state will be reset. What we actually need to persist is the players, and for the moment all of them are temporary. I'd like to hit the sweet spot somewhere between LPMUD (Which saves only the player's stats and discards all equipment) and a MUSH (Which saves every little detail of everything) but it's not a problem I've tackled yet. It will come after we add an account database.

Some thoughts that led me to this conclusion: In production, a server restart is a rare and horrifying thing, and can be scheduled to minimize the impacted players. We don't actually want to save every little detail of the world. Things the player dropped and abandoned need to be cleaned up, mobs need to respawn, quests need to reset, etc.

[ConstableBrew]

I got a rough outline written up that currently will save all named objects. I plan on making each MudObject decide if it gets saved or not. I would argue that real world issues shouldn't impact the game, that regular garbage collection should keep the system lean. Constantly writing to a db can slow things down, so I am with you on not every little thing needs to be saved.

I tried to push my branch last night but got a 403 permission denied error. I think I need to be added to the project, but I really don't know much of anything about git security.

[Blecki]

Try pushing it to your own fork.

Bringing over discussion from the guid issue, yes that's pretty much how I see it working. I plan to automate it using attributes on members to be preserved, with a fallback through a dto.

There is a real good reason for holding off on this though. We want to keep it simple to run our own local copies. So the persistence layer would have to be optional. I can bump it up on my schedule if you're eager to contribute.

One rule I want to impose - only instances of named objects can be persisted. This is because we need to be able to find the script for the object to create it while deserializing. There is also a distinction to be made between named objects and instances of them. A room, for example, there is only ever one of. But there may be many instances of a piece of equipment. The room should be persistent. The equipment should only be persisted as part of another named object.

I can mock all this out with attributes if you can provide the actual database reading and writing part.

[ConstableBrew]

Life would be a little easier for me if I could just work out of a branch instead of pushing and pulling between forks. We would then have all the PR comments in one location, easier for others to find. And you can dictate when my branches get merged with master.

I'll add a ticket to make the DB loading/saving configurable, off by default.

I'm not sure that every room should have a 1:1 relationship with .cs files. Couldn't a single room be duplicated to create large regions of similar landscape? Allowing a unique identifiers to be appended to the path would give us the ability to create multiple named instances of the same object - Rooms and Actors in particular.

[Blecki]

They could be duplicated, but do we want to?

Something simple and predictable can be appended - for example, dummy@foobar. The path is dummy, the instance is foobar, the entire thing is the database name. Attempting to load it would load dummy and then deserialize a specific instance.

I don't expect we have to give names to instances of equipment, not if we can embedd that data inside whatever object has the instance. I'm not convinced we want to give them unique names, though it is certainly possible.

[Blecki]

Al right; I've invited you to the organization. Accept that invitation and you'll have the access you need to create a branch and push to it.

I think I know what you're doing but I won't be sure until I see actual code. Please also include what we would need to get the database running locally.

[edit] I mocked up the extended name handling I talked about and got far enough to realize that I don't like it. I feel like these objects should never be listed in the named objects list. However, I am going to commit a revision that should give you the hooks you need to implement what you're doing. Look in Database.cs for a method like 'GetOrCreateInstance'. Also see a DTO fetching method in MudObject. I do not want to pass the DTO to Initialize - in this instance we want to keep authoring objects as simple as possible, even if it means poor design.

[ConstableBrew]

I created a PR: #35 to show what I've got thus far. The serialization isn't working yet since I don't have proper DTOs set up and am just attempting to serialize the full list of MudObjects. I didn't go through your changes yet, but will bring my stuff in line with where you were going when I do. One thing you will notice is that I change the variable name Path to id in most of the database.cs functions since that is the intended use of the value.

[Blecki]

I commented on that in your changes. Its terribly irrelevant.

This is pretty much the same design I had planned all along, though I was probably going to serialize to disc and not to a database. That's where the 'static'/'serialized' path came from. The intent was that, after loading something from the static path, it would check the serialized path and load anything from there, overwriting the static values. I still like the design.

Here's what I'm concerned about: The upgrade to VS2013 and the influx of several libraries. I don't want to break compatibility with mono. The virtual save function. It relies on objects playing nice and calling base.Save(). Giving everything persistent it's own entry in NamedObjects. I don't see where you distinguish between instances and.. 'singleton', for lack of a better word, objects.

Here's what I'd like to see: A way to tag which fields and properties get serialized and which don't. The default should be don't, so the author would just tag persistent members. Using reflection to automatically find these members and serialize them. This would remove the need for any sort of virtual function that a derived type can forget to call. A way to serialize instances without actually giving them a unique instance id. Items such as the player's inventory. This would mean writing them into the player object's serialized data. They would also have to be stored with the full path (id) of the type they are an instance of so they could be recreated when the player is deserialized.

[ConstableBrew]

Fair list of concerns and "todo". I'll get things cleaned up. I've never used Mono, but I am pretty sure that there wont' be a problem. I'll verify.

[Blecki]

I know there will be no problems with your json library, I've used it myself. Is RavenDB using json under the hood?

RevenDB itself can be replaced with something that works on mono, I'm sure.

I'm most worried about the additional microsoft assemblies, actually.

[ghost]

I'm going to throw my question in here:

Would we be able to replace RavenDB with an alternative? It won't support Mono officially until the 3.0 release. There are ways around this with some hackery, but if we can just have everything support Mono from the outset, I think we'd be better off.

[ghost]

Any update on this?

[Blecki]

I'm pretty much leaving this up to @ConstableBrew for now. I have plenty of other things to implement first, but I'll get to it eventually if he doesn't.

[ConstableBrew]

I'll be working on this over the weekend. That will be my general pattern - real work during the week and then RMUD during the weekend.

[Blecki]

@ConstableBrew , did you make any headway on this?

[ConstableBrew]

I just did a bunch of learning about mono and nosql dbs that work with it. I'm going to switch to STSdb. What version of mono are we targeting?

[Blecki]

I'd say we assume the latest. Trevoke has a server running on it now; since DevRW had been missing in action for a few days before trevoke set it up I doubt trevoke checked what version it was on.

[ConstableBrew]

I've noticed the license for STSdb is GPLv2, which would mandate we change our software to the same license (AFAIK). Should I keep on with this? I myself strongly prefer the MIT license.

[Blecki]

I do too. Lets find a different option. Keep in mind that at this scale we can fall back on just using the filesystem if we have to.

[Trevoke]

On Ubuntu, I'm running the Mono JIT compiler 3.2.8, but when I compile in Xamarin studio I ask it to target Mono 4.0 .

[ConstableBrew]

I have dropped the db and am now writing up a file system based solution.

[Blecki]

Some things I think we need to agree on for a file system based solution.. a) The serializer doesn't need to know how to serialize specific kinds of mud objects. Not even the basic ones like 'actor' or 'room'. It should just supply a DTO that the object can use to read or write values. b) The storage should be separate from the 'static' files currently in there. For example, the same directory structure but rooted at database/dynamic/ instead of database/static/. c) It will be much easier to debug persistence issues if the on-disc format is human-readable. JSON would work well.

Above all lets try not to overcomplicate it.. the filesystem is already doing most of the work for us... Lets just stash a Dictionary<String, String> in a json file using the mud object's path and instance name.

[Trevoke]

I agree with everything you just wrote, though I'd recommend YAML: http://yaml.org/ because it's very human readable and writable.

JSON would probably be fine though we'd have to be sure we format the files well when we write them.

On Wed, Oct 15, 2014 at 8:52 PM, Blecki notifications@github.com wrote:

Some things I think we need to agree on for a file system based solution.. a) The serializer doesn't need to know how to serialize specific kinds of mud objects. Not even the basic ones like 'actor' or 'room'. It should just supply a DTO that the object can use to read or write values. b) The storage should be separate from the 'static' files currently in there. For example, the same directory structure but rooted at database/dynamic/ instead of database/static/. c) It will be much easier to debug persistence issues if the on-disc format is human-readable. JSON would work well.

Above all lets try not to overcomplicate it.. the filesystem is already doing most of the work for us... Lets just stash a Dictionary in a json file using the mud object's path and instance name.

— Reply to this email directly or view it on GitHub https://github.com/Reddit-Mud/RMUD/issues/31#issuecomment-59299569.

[Blecki]

I couldn't resist throwing together the simplest possible solution. https://github.com/Reddit-Mud/RMUD/compare/filesystem-database-blecki?expand=1

[ConstableBrew]

My desktop drive died and I spent the past two days trying to fix it, install windows to a new drive, and other things that all failed. Now I have a new laptop! I just cloned your solution and will check it out now.

[Blecki]

I checked out your code and was able to dump the database. The current strategy of dumping every property isn't the way forward, though. We also need to save things that aren't in the named object's list. I'm reluctant to give MudObject a virtual 'Save' method. I'd rather let the mud object configure a DTO during initialization, and point entries in the DTO to specific properties. Then the persistence system can track DTOs and query those properties to save the object. It might be simplest to identify those properties to be persisted with attributes.

It occurs to me that having the persistence system track DTOs solves another problem with GetOrCreateInstance, and allows us to save everything at once while only traversing the objects that actually need to be saved. So I'm going to go implement that.

[Blecki]

Check it out in the current version of the branch. There's an example in database/static/palantine/skull that shows how an object can use it to persist properties.

I'd also like to support another feature - saving objects inside other objects. This will require switching to JSON or XML or something like that. Consider for example a player's equipment. It would be great if the player object could store those items in it's own serialized file instead of littering the database with tiny little files.

[ConstableBrew]

(Havent looked at code yet - on mobile) I was considering using datacontract and datamember attributes. xamarain and VS gave me reference errors. My new pc still needs some configuring.

The all properties is an OK fallback if a class doesn't define a datacontract, but specifying what should be serialized is a much better solution.

[Blecki]

I expect the only sensible default to be to persist nothing, as there's lots of things it won't be able to persist anyway.