v0.14 : KeyNotFoundException: The given key was not present in the dictionary.

[soile1991]

I receive a key not found at https://github.com/colyseus/colyseus-unity3d/blob/master/Assets/Plugins/Colyseus/Serializer/Schema/ReferenceTracker.cs#L51

KeyNotFoundException: The given key was not present in the dictionary.
System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) (at <9577ac7a62ef43179789031239ba8798>:0)
Colyseus.Schema.ReferenceTracker.Remove (System.Int32 refId) (at Assets/Plugins/Colyseus/Serializer/Schema/ReferenceTracker.cs:51)

Fields generated by the last version of c# generator (original version)

Version : colyseus server : 0.14.4 colyseus unity : 0.14.2 schema : 1.0.6

[lee-orr]

I'm getting the same issue, and then next time I try to update the same object I end up with:

Exception: refId not found: 6
Colyseus.Schema.Schema.Decode (System.Byte[] bytes, Colyseus.Schema.Iterator it, Colyseus.Schema.ReferenceTracker refs) (at Assets/Plugins/Colyseus/Serializer/Schema/Schema.cs:226)
Colyseus.SchemaSerializer`1[T].Patch (System.Byte[] data, System.Int32 offset) (at Assets/Plugins/Colyseus/Serializer/SchemaSerializer.cs:31)
Colyseus.Room`1[T].Patch (System.Byte[] delta, System.Int32 offset) (at Assets/Plugins/Colyseus/Room.cs:351)
Colyseus.Room`1+<ParseMessage>d__37[T].MoveNext () (at Assets/Plugins/Colyseus/Room.cs:310)

I have a large object (containing a bunch of MapSchema's) that I replace when this happens. Generally the first replacement will cause this but seem to work, while the second one doesn't. I also tried instead of placing this object in a Map itself, deleting the old one and adding the new one in a new key - but it still causes the exact same issue. For me this is breaking - this object contains the building blocks for user-created maps, and without being able to update them & replace them I am completely stuck.

[endel]

Hi @soile1991 and @lee-orr, thanks for reporting, and sorry for the delay to answer. (I was on vacation the last few weeks)

It is likely that ArraySchema's decoder is the cause of this (C# client-side). Would you mind providing a minimal reproduction scenario where I can reproduce this? I've tried but failed to reproduce.

Ideally, if you can fork this project colyseus-unity3d and adapt the example to output this error in question would help a lot!

Thanks!

[lee-orr]

I just spend a couple hours trying to find a minimal repro, but haven't really been able to do so... as least so far. I'll give that another try when I have the time.

[soile1991]

Hey, @endel I made a fork from colyseus-unity3d and implemented the behavior to trigger the bug.

it's a card game you start with 52 cards and every time you click to play the cards button it plays 1 to 3 cards randomly if you hit it many times (e.g. once per second) it will trigger the bug before the remaining cards hit zero.

Repo: forked repo (card game)

Thanks!

[endel]

Thanks for providing a reproducible example @soile1991 🥳

I've spent some time debugging it, and there were a few small issues decoding ArraySchema.

Could you confirm if copying the latest version of ArraySchema.cs into your project fixes the issue @soile1991 @lee-orr? If it does I can push a new version

Cheers!

[lee-orr]

I'll take a look in the morning! Thanks.

Lee-Orr

[soile1991]

@endel I think that fixed the bug. I tried it many times and I didn't encounter it once. Thank you 🙏

[lee-orr]

When I first tested it it looked like the issue was resolved, but after continuing to work a little I ran into it again...

[lee-orr]

In my case I'm changing a MapSchema rather than an ArraySchema - let me see if I can replicate your changes there...

[lee-orr]

Unfortunately I don't understand the changes well enough to know exactly how to replicate them on a MapSchema... I can try to give you access to my project, but it's an early-phase prototype with many moving parts, so I don't think it'll be easy to understand...

[soile1991]

@lee-orr I added a MapScema on my fork and it works ok (clear set delete) what is the operation you do when you run into the bug?

[endel]

@lee-orr can you try this version and see if works for you? https://gist.github.com/endel/ace4cf759161cb94ffbe07f0552af1f9

The problem on ArraySchema was because GetByIndex() was returning default(T) instead of null for non-existing entries. It's likely to be the same problem for MapSchema.

[lee-orr]

@endel - Unfortunately, it seems like it's not resolved by bringing that file in... I'm still getting the same errors:

System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) (at <9577ac7a62ef43179789031239ba8798>:0)
Colyseus.Schema.ReferenceTracker.Remove (System.Int32 refId) (at Assets/Plugins/Colyseus/Serializer/Schema/ReferenceTracker.cs:51)
Colyseus.Schema.ReferenceTracker.GarbageCollection () (at Assets/Plugins/Colyseus/Serializer/Schema/ReferenceTracker.cs:78)
Colyseus.Schema.Schema.Decode (System.Byte[] bytes, Colyseus.Schema.Iterator it, Colyseus.Schema.ReferenceTracker refs) (at Assets/Plugins/Colyseus/Serializer/Schema/Schema.cs:471)
Colyseus.SchemaSerializer`1[T].Patch (System.Byte[] data, System.Int32 offset) (at Assets/Plugins/Colyseus/Serializer/SchemaSerializer.cs:31)
Colyseus.Room`1[T].Patch (System.Byte[] delta, System.Int32 offset) (at Assets/Plugins/Colyseus/Room.cs:351)
Colyseus.Room`1+<ParseMessage>d__37[T].MoveNext () (at Assets/Plugins/Colyseus/Room.cs:310)```
&
```Exception: refId not found: 6
Colyseus.Schema.Schema.Decode (System.Byte[] bytes, Colyseus.Schema.Iterator it, Colyseus.Schema.ReferenceTracker refs) (at Assets/Plugins/Colyseus/Serializer/Schema/Schema.cs:226)
Colyseus.SchemaSerializer`1[T].Patch (System.Byte[] data, System.Int32 offset) (at Assets/Plugins/Colyseus/Serializer/SchemaSerializer.cs:31)
Colyseus.Room`1[T].Patch (System.Byte[] delta, System.Int32 offset) (at Assets/Plugins/Colyseus/Room.cs:351)
Colyseus.Room`1+<ParseMessage>d__37[T].MoveNext () (at Assets/Plugins/Colyseus/Room.cs:310)
--- End of stack trace from previous location where exception was thrown ---
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <9577ac7a62ef43179789031239ba8798>:```

Is there anywhere I should place breakpoints to track what is happening?

@soile1991 - I am replacing an object that contains a bunch of MapSchema's (and some of the child types also contains MapSchema's) with another object with the same structure. So it's basically a mass clear & many sets.

[soile1991]

@lee-orr can you share a snippet of the code deals with those operations and a snippet of your object with map schemas? If we can it will be easier for @endel to fix the bug on MapSchema

[lee-orr]

It's complex - but here's the best explanation I can give: We have a MapSchema of GameMap objects - each of which has a property called sdfOperators of type OperatorList. This is that type:

export class OperatorList extends Schema {
    @type(['string'])
    roots = new ArraySchema<string>();

    @type({ map: SimpleSDF })
    sdfs = new MapSchema<SimpleSDF>();

    @type({ map: FloorBuilder })
    floorBuilders = new MapSchema<FloorBuilder>();

    @type({ map: WallBuilder })
    wallBuilders = new MapSchema<WallBuilder>();

    @type({ map: PortalBuilder })
    portalBuilders = new MapSchema<PortalBuilder>();

    @type({ map: RoomBuilder })
    roomBuilders = new MapSchema<RoomBuilder>();

    @type({ map: ObjectPlacer })
    objectPlacers = new MapSchema<ObjectPlacer>();

    @type({ map: BlockBuilder })
    blocks = new MapSchema<BlockBuilder>();
}

And here's one example of an object in the maps contained inside:

export enum SDFPrimitive {
    Cube = "cube",
    Sphere = "sphere",
    Cylinder = "cylinder",
    Torus = "torus",
    Curve = "curve"
}

export enum OperationType {
    Add = "add",
    Remove = "remove",
    PaintMaterial = "paintMaterial"
}

export class SimpleSDF extends Schema {
    @type('string')
    id: string;

    @type('string')
    primative: SDFPrimitive;

    @type(Transformable)
    transform = new Transformable();

    @type('string')
    operation: OperationType;

    @type('float32')
    blend: number;

    @type(Color)
    color = new Color();

    @type({ map: 'float32' })
    parameters = new MapSchema<number>();

    @type(['string'])
    children = new ArraySchema<string>();

    static CreateOperation (params) : SimpleSDF {
        let sdf = new SimpleSDF();
        ///setting the contents
        return sdf;
    }
}

When changes are made to the map, a new OperatorList is generated and replaces the sdfOperators field in the GameMap like so:

const list = new OperatorList();
// processing some inputs to generate the operators, using the CreateOperation function for each type -
// and getting an object containing each operation
for (let i in result.sdf) list.sdfs.set(i, result.sdfs[i]); // same thing for all the other types of operations - if I comment this portion out, I have no issues.

// Set the new list as the current list for the map we're in
map.sdfOperators = list;

I have tried a couple of other options - such as creating an empty operator list, and replacing it later (so that it clears the content of the game map), populating the list after a timeout, or setting the "map.sdfOperators" first, and setting the list's properties later. I've also tried re-using the existing object, and just clearing each of the MapSchema/ArraySchema properties on it. None of those seemed to make a difference. I also tried both using consistent id's for the objects in the maps, and using id's that change each time.

[lee-orr]

Do you have any suggestions/ideas of where to look in the library itself to see if I can help debug it?

[lee-orr]

Right now I'm working on some alternative approaches for this project - just because I haven't been able to debug it and need to get this done. But once I have some time again, I'll attempt to reduce this thing down to a minimal reproduction to the best of my ability. Sorry for the delay with that @endel - I don't want to take up more of your time on my specific project - so I'll resolve things differently for now and will get back to you when I have something that doesn't involve 2 separate node servers (colyseus + a rest API), a web based UI layer & and a unity based 3D display...

[LadyAelita]

I'm having an issue with that as well.

KeyNotFoundException: The given key was not present in the dictionary.
System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) (at <437ba245d8404784b9fbab9b439ac908>:0)
Colyseus.Schema.ReferenceTracker.GarbageCollection () (at Assets/Plugins/Colyseus/Serializer/Schema/ReferenceTracker.cs:70)
Colyseus.Schema.Schema.Decode (System.Byte[] bytes, Colyseus.Schema.Iterator it, Colyseus.Schema.ReferenceTracker refs) (at Assets/Plugins/Colyseus/Serializer/Schema/Schema.cs:471)
Colyseus.SchemaSerializer`1[T].Patch (System.Byte[] data, System.Int32 offset) (at Assets/Plugins/Colyseus/Serializer/SchemaSerializer.cs:31)
Colyseus.Room`1[T].Patch (System.Byte[] delta, System.Int32 offset) (at Assets/Plugins/Colyseus/Room.cs:351)
Colyseus.Room`1+<ParseMessage>d__37[T].MoveNext () (at Assets/Plugins/Colyseus/Room.cs:310)
--- End of stack trace from previous location where exception was thrown ---
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <437ba245d8404784b9fbab9b439ac908>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__6_0 (System.Object state) (at <437ba245d8404784b9fbab9b439ac908>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <58a34b0a618d424bb5fc18bb9bcdac20>:0)
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

This happens when calling .splice() on the server on an ArraySchema instance held by the room state, but only if stuff is removed in at least two separate patches in a row - after that, it's thrown on every single state patch regardless of anything else. That does not happen when alternating between adding and removing a single item each time.

The weirdest thing is that so far it seems to have not actually prevented anything from working properly, or at least no cases of broken functionality can be observed. Still, it's pretty alarming.

Was this by any chance fixed in the recent Unity SDK release? If so, I could consider migrating, which otherwise I would prefer not to do, as I have limited a deadline to meet and would rather not waste time on non-essential adjustment.

[mitchLucid]

@Unelith I will try to look into this soon but if it was fixed in the update it was done incidentally, so I wouldn't begin migrating your project on that hope quite yet. I will try to set up a good repro case locally and test with the updated SDK to confirm one way or the other

[LadyAelita]

@mitchLucid Perhaps extra details could be useful then:

1. This ArraySchema holds instances of an AreaState Schema, whose definition is as follows:

   export class AreaSettings extends SettingsGroup {
   @type('int32')
   distanceCalculationMode: number = 0;
   
   @type('int32')
   gridWidth: number = 10;
   
   @type('int32')
   gridHeight: number = 10;
   
   @type('string')
   distanceUnit: string = 'ft';
   
   @type('float64')
   gridSquareSideLength: number = 5.0;
   }
   export class AreaState extends Schema {
   @type('string')
   id: string;
   
   @type('string')
   name: string;
   
   @type( AreaSettings )
   areaSettings: AreaSettings = new AreaSettings();
   }

2. On the client, having retrieved an AreaState schema instance from either OnAdd or OnRemove listener attached to the ArraySchema, I do pass it around quite a lot, for instance:

   room.State.areas.OnAdd += emitter.HandleAddArea;

   public void HandleAddArea(AreaState _area, int _idx) {
   onAddArea.Invoke(_area, _idx);
   
   _area.OnChange += (_changes) => {
       HandleAreaChanges(_area, _idx, _changes);
   };
   }

   emitter.onAddArea.AddListener(delegate (AreaState _area, int _idx) {
   CreateAreaListEntryObject(_area);
   });

   _refs.deleteAreaButton.onClick.AddListener(delegate {
   HandleRemoveAreaButton(_area);
   });

   Which in turn calls this:

   public async void RemoveArea(AreaState _area) {
   await colyseusClientController.CallService("deleteArea", new {
       areaId = _area.id
   });
   }

[mitchLucid]

Thank you, @Unelith this is helpful info!