State management and synchronisation library

[Dassderdie]

Requirements

Synchronisation between server/clients

• a client can subscribe to a part of the state (example: all vehicles and personnel in viewport1 (= calculated via coordinates))
• state should be robustly stored (backups) and handled (= database)
• the state change can be executed without conflicts (e.g. a client changing the position of a vehicle during the state-update)
• state changes should be validated by a trusted entity
• a client gets updates to the state in real-time (e.g. "push" like websockets, no polling)

(local) State management client

• in the client one should be able to subscribe to a part of the state (async observer pattern) and get snapshotted values directly in a synchronous way
• nicely encapsulated client-side optimistic updates for more responsive ux

both

• a client can modify the data and send the change to the server/the other clients
• very good typings and reusability across all codebases (client and server)

TODOs

• [x] API design websockets
• [x] senden des States über websocket verbindung
• [x] backend ordentlich machen (file structure, watch mode)
• [x] action validation (@ClFeSc)
• [ ] setup test frameworks
  • [x] backend
  • [ ] shared
  • [ ] frontend unit tests
  • [ ] e2e
• [x] optimistic updates
• [ ] Add actionNr to performAction and getState to fix possible synchronisation issues
• [ ] Resolve all TODOs in the current code base

libraries to investigate

realtime database

• firebase realtime database
• SapphireDb
• supabase
• roomservice
• replicache

client-side state management (stores)

• NGXS
• NGRX

[Dassderdie]

See this branch for a proof of concept.

[Dassderdie]

Updated concept art

[Dassderdie]

We decided to first create the state data structure/typings to better understand the problems we are facing.

[Dassderdie]

Today we finished the state data structure and discussed about reusing the actions from the store to make changes on the server. There is also a second iteration of the algorithm for optimistic updates: Optimistic updates 2.pdf

[Dassderdie]

@ClFeSc I fixed the classes (we didn't initialize any of the properties -> I mostly moved them into a constructor for now). There is now also a solution to import all the stuff in the shared folder in the frontend and backend (incl. adding dependencies (uuid) to the shared package). To see the typings in the frontend you have to run in the shared folder npm i and npm run build as well as npm i in the frontend again. For now always if you change something in shared you have to run npm run build again - maybe project references could be a solution in the future...

[Dassderdie]

Problem:

The frontend relies on immutable objects. Therefore e.g. the native Map type is seldom used with such frameworks and instead the "old" approach of an object dictionary.

new Map(map.entries()).set('key', 'value'); or cloneMap(map).('key', 'value'); The performance for my naive implementation of adding an element to an immutable Map is around half as fast as with the object dictionary.

{ ...object, ['key']: 'value' }; The .stringify() method (superjson etc.) is slower too (no idea by what factor).

We have the following options (the reducer function (= applies action on the state) is shared between frontend and backend):

• use only objectDictionaries (better write and stringify performance, worse read performance)
• use only Maps (worse write and stringify performance, better read performance)
• use a wrapper around maps -> in the frontend immutable.js, in the backend we could (for the best possible performance) skip the immutability contract via a custom wrapper function. But I guess this would only be a big factor for ugly problems later on (especially with unexperienced programmers).

Exactly the same goes for Set...

[Dassderdie]

with immer this doesn't look so bad now:

setState(
    produce(getState(), (state) => {
        // this isn't needed thanks to "immer"s produce
        // state.viewports = clone(state.viewports);
        state.viewports.set(viewport.id, viewport);
    })
);

ImmutableAssign didn't work with Map (the reference wasn't renewed -> the object got mutated).

Therefore I now use immerjs. You have to enableMapSet to patch them. The documentation states that it modifies the methods of Map and Set there -> maybe later on problems when trying to stringify it...

NGXS has the ability to use the Redux devtools. This would in theory enable time travel in the store state for better debugging. Sadly it seems like it doesn't work with Map or Set, but only plain objects.

[Dassderdie]

I'm trying out NGRX out now instead of ngxs, because it seems to make it easier to reuse the actions in the backend without the need of the whole store.

[Dassderdie]

I added socket.io. Actions are now send via websocket to the backend, and then broadcasted to all clients.

https://user-images.githubusercontent.com/18506183/145633041-93c815f4-e746-423c-9ba9-818418981381.mp4

Problems I'm working on:

• [x] the reducer functions from ngrx expect to be compiled within an angular context -> can't use them directly in the backend -> we have to make a custom function that integrates with ngrx

[Dassderdie]

To add any action you have to add the Action and the reducer-function:

exercise.actions.ts

export class RemoveViewport implements Action {
    readonly type = '[Viewport] Remove viewport';

    constructor(public viewportId: UUID) {}
}

exercise.reducer.ts

{
    ...
    '[Viewport] Remove viewport': (state, { viewportId }) => {
        state.viewports.delete(viewportId);
        return state;
    },
    ...
};

Use it like this:

exerciseReducer(state, action)

The code to send an action in the frontend looks like this: app.component.ts

this.apiService.sendAction(new ExerciseActions.AddViewport(viewport));

[Dassderdie]

Im currently thinking about wether the object literal syntax would sometimes be better (especially in cases with a lot of constructor arguments):

this.apiService.sendAction({
      type: '[Viewport] Add viewport',
      viewport,
});

[Dassderdie]

• [x] API design websockets
• [x] senden des States über websocket verbindung
• [x] backend ordentlich machen (file structure, watch mode)
• [x] action validation (@ClFeSc)

[Dassderdie]

I just noticed a problem with our current API design: The current idea is to first get the state and then all apply all the actions that we receive via "performAction". But there is no way of knowing which state we get and which actions were already applied.

I would therefore propose the following:

• getState and performAction have a timestamp (or some other kind of number that gives them an order) in the answer from the backend.

The frontend

1. subscribes to performAction and saves the actions it receives
2. gets the State
3. Forgets all actions whose timestamp is before the timestamp of the state
4. Applies all the other actions in order

We would have to check whether a timestamp (Date.now()) is somehow security relevant...

[ClFeSc]

I think we could just count the actions and return the action number of the last applied action with the state.

[Dassderdie]

Yep, that's better than a timestamp 👍

[Dassderdie]

About performance: I think it would be nice if we could parallelise multiple exercises. So all the code for an exercise should be encapsulated so that you can easily make one thread per active exercise later on.

[Dassderdie]

I'm afraid this issue and pull request is going too big. Therefore I propose the following:

• We only focus on the current TODOs
• If new needs for features arise we do this after this one is finished and the pull request is merged in the dev branch
• The focus should lie on
  • Fix all TODOs in the codebase or create issues for them
  • Refactor the project structure and add documentation, so that the merge on dev is a clean start

[Dassderdie]

• [x] eslint extension has Parsing error: Cannot read file '/media/clemens/Daten/Clemens/Dateien/HPI/WiSe 21-22/Bachelorprojekt/GitHub/digital-fuesim-manv/tsconfig.json'. on linux / @ClFeSc
• [x] validation doesn't work on nested classes
• [ ] watch configuration doesn't seem to work on linux / @ClFeSc

[Dassderdie]

The pull request has been merged and issues for all unresolved tasks have been created. The experiment was a complete success! ;)