ECSLib -> UI Comms

[se5a]

Currently, getting data from the ECSLib to the Viewmodels is messy, makes for increasingly complex datablobs, inefficient in the case of lists of objects that need to be displayed.

In some cases, some data contained within a datablob that is owned by a factions entity should not be known to that factions player. the NameDB is a classic example of this, where the NameDB stores all the names that it's parent entity is known by (different factions can call it different things) I don't believe this behavior should be changed at the datablob level as it would constrict what a datablob can have in it too much, potentially affecting the DOP behavior.

where the UI needs to display lists of items (ie cargo) or worse, lists of ordered items that can be manipulated (ie industry construction queues, ship move queues), the ui is not able to build up a list of control items fast enough to completely refresh the entire control every frame. the ECSLib needs to alert the ui via the viewmodel that the order of a list has changed. There are Observable collections in .net however, these lists need to be publicly viewable but in no way publicly mutable. making a typical list {public get; internal set;} ensures the list itself cannot be replaced, but does not prevent the list or the items inside the list from being publicly manipulated. to make matters even more problematic any event which is fired/invoked by a thread gets run by the thread that invoked it this means that if an event is going to affect the UI, it needs to be marshaled to the (main)UI thread or the UI will throw an exception (iirc this is an issue with most UI libraries including WPF). currently the CargoStorageDB uses a complex custom Public Read, Internal Write Observable Dictionary which implements InotifyCollectionChanged, and Marshales events to the UI thread. I suspect a datatype of this complexity is not ideal for a DOP 'ECS-Component' datablob. It makes hooking up the viewmodels more complex than I'd like.

I'm leaning towards completely separating the UI from the ECSLib, and having a messagepump of some sort on the ECSLib side. I think that creating the UI as if it were a remote client would possibly solve some of the issues by enforcing a single point of read/write (via messages too and from the message pump). this would also make future network clients much easier to implement.

Alternatively we could have a (number of?) facade for the viewmodel(s) to read from / to inside the ECSLib, which would take care of the public/private problem, and could probability handle the marshaling, and remove some of the complexity from the datablobs. however this would likely require a lot of boilerplate. (though completely separating the two may also require boilerplate) and may reduce flexibility of the UI depending on how it was implemented.

[se5a]

related to #177

[se5a]

Current implementation:

[se5a]

I'm rethinking seperating the UI and ECSLib. combining the problem of updating the UI and multiplayer updates is not going to work well.

I'm still not fully sure how I'd like to tackle the way the UI updates. ECS systems normally have a rendering component/datablob. but most games don't have a ui that is quite as complex as what we're trying to do. not every entity gets rendered, or currently in view of the UI. in some cases only part of an entity is currently viewed by the player/UI. One way to do it is to continue the way we are, but giving the viewmodels more work determining if anything has changed. they do do this to a some extent already, the problem happens when the datablob that the viewmodel is looking at has lists or dictionaries of items that need controls of their own. re-creating these lists of UI objects each tick is too slow.

The real problem is keeping two disparate lists in sync. i.e. a datablobs list of guid, and the UI view's list of control objects.

The multiplayer client/server problem would probably be better solved by the server and client both having the ECSLib, with the client getting just a partial copy (everything the faction knows) of the game data. this is how I originally looked at tackling the problem. The client would create a command, which would get sent to the server. the server's ecslib actions the command server side, and sends the command back to the client's ecslib. the client then actions it on it's copy of the game. This would take care of most of the client side stuff, and once the client has it's copy of the faction, should keep data traffic down to a minimum. and random stuff should be made deterministic by using a known seed. Things it wouldn't solve would be telling the client of new things that are not initiated by the client, such as a new enemy ship contact. this would possibly be solved by giving the faction a more solid system of ownership. ie when an entity is given/created to the faction, it'll check for clients and pass the data on.

[DifferentLevelDan]

I think we bit off too much for this project. We don't have enough combined experience to pull off a game of this scale.

I think if we want to continue, we should look into not creating our own game engine.

I've been looking into Xenko, an open-source, ECS, C# 7 crossplatform game engine. It looks like it has some performance issues which we may eventually have to deal with, and we'd be throwing away a large amount of work, but it might just be the right call. We would have to do some serious work for the UI lists we want, however.

If that's not an option, I do think the work I did in my fork/branch with the INotifyPropertyChanged and INotifySubcollectionChanged may be of use. It allows networked partial updates of lists and data found in datablobs. Whenever a property in a datablob is changed, it sends an event to the MessagePump to notify any listeners (Local UI, or remote clients, it's generic). It's similar to the DataSubscription stuff you had, but it's generic and works for every current piece of information stored on a DataBlob (Except nested dictionarys atm), and works for future datablobs that utilize the same pattern (I made it as easy as possible to implement). It's not fully implemented (I got a bit sidetracked working on the network portion), but I got pretty far and could likely finish it if there's interest.

Basically the point then would be for the ECSLib to handle the real data, and the events system automatically syncs any changes to the UI viewmodels' copy of the data (VM uses actual ECSLib classed DataBlobs, same as the ECSLib). Additionally, data can be synced directly back to the ECSLib if we enabled such functionality. The user changes something's name, and we just send the changed DataBlob back to ECSLib with proper authorization (Not the currently-implemented AuthTokens, since we'll be removing the concept of the Player class). This would give SM players the ability to literally change any data, since they have the ultimate authorization code. We just have to give them a UI capable of editing the datablob's data.

Since the data syncs automatically, it would work for orders as well. Setup a datablob that stores current orders using the system, and the UI (Client) simply tells the ECSLib (server) that the data in the OrdersDB has changed, and the ECSLib validates them, saves them, and uses them in further execution.

Additionally, since it's a subscription, we can filter what we send. We don't need to send each client Position updates for Earth, we only need to send that to our Local UI VM "client" Remote clients can figure it out for themselves (using their own ECSLib).

Telling the client of new things, such as a new enemy ship contact is handled through the notification events. Server ECSLib creates an entity, this sparks a Event in the EntityManager (EntityCreated, EntityMoved, (also there's new events on the ProtoEntity: DataBlobSet; DataBlobRemoving; and EntityDestroyed) that is dispatched to all clients (Through the MessagePump) who are able to see the system where the entity exists (or a subscription and/or further filtering). They get a EntityID of the new entity, but no other information.

They then request information about the Entity, which is the server responded to based on authorization. Remote clients stores of copy of any information received in their own ECSLib. Clients (Local VM, or remote) can then request to be notified of any changes in the entity's data. If need be, we can filter it down to even specific pieces of data subscribed to.

Also, since the DataBlobs themselves implement INotifyPropertyChanged and INotifySubcollectionChanged, they can easily be used as DataBinding targets themselves. The VM can literally store ProtoEntity clones of ECSLib's data that's synced through the MessagePump. The ECSLib sends a change to the UI, the UI thread picks it off the messagepump, and uses the automatic sync, the sync triggers a DataBinding update to the UI (And we're already on the UI thread)

This method I was looking at does come with some drawbacks. It doesn't work well hiding partial data on entities. It may be easier for a "Sensor Contact" of a ship to be a separate entity from the ship itself. NameDB will need to be completely redesigned if we do split entity from sensor contacts (Since we can't hide NameDB's partial data), and we'll have increased memory usage (Premature optimization?)

If you look at https://github.com/DifferentLevelDan/Pulsar4x-RSE you can see the work I did towards this goal. I also created a Unit Test (Pulsar4x.Test\Notifier.cs) to prove the ability to automatically sync two datablobs using the events. Additionally, the EntityChangeProcessor in that project is storing every change on every datablob in every frame, in an object that's easily serialized and deserialized. It's extremely close to actually syncing data between the actual UI and ECSLib through the MessagePump.

Commands could work too, but you could implement both Commands, and the data sync together. The DataSync could just be one command (and a pretty useful one imo)

Problem I have with commands is I haven't seen an implementation put forward to Pulsar4x with commands that is generic enough to not be a maintenance pain. Remember, we basically have to sync both from the ECSLib to the UI, and from the UI to the ECSLib, for literally ANY piece of data on an entity, including Data that we haven't even thought about designing yet. We can't do it manually for all cases.

Check out what I have. It is exactly what you wanted when you asked for a streamed data notification protocol that can be used both locally and remotely.