Data deserialisation error (Object type cannot be converted to target type)

[VictorPhilipp]

While testing the fix proposed in pull request https://github.com/VictorPhilipp/Cities-Skylines-Traffic-Manager-President-Edition/issues/189 I found that mod data was not persisting between game loads. After saving and quitting, then reloading the game, I found that I had to spend an hour remaking all of my lane connections, restrictions, speed limit changes, timed traffic lights, etc.

I'm not sure what the root cause is, but I also don't think it's the proposed pull either. When I was using workshop 1.10.8, this problem occurred (though I did not check for a stacktrace at that time).

Three days ago I was using TMPE 1.10.3, as a test to see if it fixes the PT bug which https://github.com/VictorPhilipp/Cities-Skylines-Traffic-Manager-President-Edition/issues/189 appears to fix. When I switched to that version, there was a bug with data resetting and me having to redo all of the above. This happened again when I switched to 1.10.7, also as a test, and again when I switched back to workshop 1.10.8.

I inspected TMPE.log (%programfiles(x86)%/Steam/steamapps/common/Cities_Skylines/TMPE.log) and found a stacktrace (using the .dll's provided in https://github.com/VictorPhilipp/Cities-Skylines-Traffic-Manager-President-Edition/issues/189).

[Error] @ 1485767776 Error deserializing data: System.ArgumentException: Object type System.Collections.Generic.List`1[TrafficManager.Configuration+ExtCitizenData] cannot be converted to target type: System.Collections.Generic.List`1[TrafficManager.Configuration+ExtCitizenData]
Parameter name: val
  at System.Reflection.MonoField.SetValue (System.Object obj, System.Object val, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0 
  at System.Reflection.FieldInfo.SetValue (System.Object obj, System.Object value) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.ObjectRecord.SetMemberValue (System.Runtime.Serialization.ObjectManager manager, System.Reflection.MemberInfo member, System.Object value) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.FixupRecord.FixupImpl (System.Runtime.Serialization.ObjectManager manager) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.BaseFixupRecord.DoFixup (System.Runtime.Serialization.ObjectManager manager, Boolean strict) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.ObjectRecord.DoFixups (Boolean asContainer, System.Runtime.Serialization.ObjectManager manager, Boolean strict) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.ObjectManager.RegisterObjectInternal (System.Object obj, System.Runtime.Serialization.ObjectRecord record) [0x00000] in <filename unknown>:0 traffic mana
  at System.Runtime.Serialization.ObjectManager.RegisterObject (System.Object obj, Int64 objectID, System.Runtime.Serialization.SerializationInfo info, Int64 idOfContainingObj, System.Reflection.MemberInfo member, System.Int32[] arrayIndex) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.RegisterObject (Int64 objectId, System.Object objectInstance, System.Runtime.Serialization.SerializationInfo info, Int64 parentObjectId, System.Reflection.MemberInfo parentObjectMemeber, System.Int32[] indices) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadNextObject (System.IO.BinaryReader reader) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObjectGraph (BinaryElement elem, System.IO.BinaryReader reader, Boolean readHeaders, System.Object& result, System.Runtime.Remoting.Messaging.Header[]& headers) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.NoCheckDeserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream) [0x00000] in <filename unknown>:0 
  at TrafficManager.State.SerializableDataExtension.DeserializeData (System.Byte[] data) [0x00000] in <filename unknown>:0 
   at CSUtil.Commons.Log.LogToFile(System.String log, LogLevel level)
   at CSUtil.Commons.Log.Error(System.String s)
   at TrafficManager.State.SerializableDataExtension.DeserializeData(System.Byte[] data)
   at TrafficManager.State.SerializableDataExtension.OnLoadData()
   at SerializableDataWrapper.OnLoadData()
   at SimulationManager.LateUpdateData(UpdateMode mode, Single minProgress, Single maxProgress)
   at SimulationManager.Managers_LateUpdateData(UpdateMode mode, Single minProgress, Single maxProgress)
   at LoadingManager+<LoadSimulationData>c__IteratorB.MoveNext()
   at AsyncTask.Execute()
   at SimulationManager.SimulationStep()
   at SimulationManager.SimulationThread()

[Info] @ 1485776767   at System.Reflection.MonoField.SetValue (System.Object obj, System.Object val, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0 
  at System.Reflection.FieldInfo.SetValue (System.Object obj, System.Object value) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.ObjectRecord.SetMemberValue (System.Runtime.Serialization.ObjectManager manager, System.Reflection.MemberInfo member, System.Object value) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.FixupRecord.FixupImpl (System.Runtime.Serialization.ObjectManager manager) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.BaseFixupRecord.DoFixup (System.Runtime.Serialization.ObjectManager manager, Boolean strict) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.ObjectRecord.DoFixups (Boolean asContainer, System.Runtime.Serialization.ObjectManager manager, Boolean strict) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.ObjectManager.RegisterObjectInternal (System.Object obj, System.Runtime.Serialization.ObjectRecord record) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.ObjectManager.RegisterObject (System.Object obj, Int64 objectID, System.Runtime.Serialization.SerializationInfo info, Int64 idOfContainingObj, System.Reflection.MemberInfo member, System.Int32[] arrayIndex) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.RegisterObject (Int64 objectId, System.Object objectInstance, System.Runtime.Serialization.SerializationInfo info, Int64 parentObjectId, System.Reflection.MemberInfo parentObjectMemeber, System.Int32[] indices) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadNextObject (System.IO.BinaryReader reader) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObjectGraph (BinaryElement elem, System.IO.BinaryReader reader, Boolean readHeaders, System.Object& result, System.Runtime.Remoting.Messaging.Header[]& headers) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.NoCheckDeserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream) [0x00000] in <filename unknown>:0 
  at TrafficManager.State.SerializableDataExtension.DeserializeData (System.Byte[] data) [0x00000] in <filename unknown>:0 
[Error] @ 1485780156 OnLoadData: Error while deserializing data: System.ApplicationException: An error occurred while loading
  at TrafficManager.State.SerializableDataExtension.DeserializeData (System.Byte[] data) [0x00000] in <filename unknown>:0 
  at TrafficManager.State.SerializableDataExtension.OnLoadData () [0x00000] in <filename unknown>:0 
   at CSUtil.Commons.Log.LogToFile(System.String log, LogLevel level)
   at CSUtil.Commons.Log.Error(System.String s)
   at TrafficManager.State.SerializableDataExtension.OnLoadData()
   at SerializableDataWrapper.OnLoadData()
   at SimulationManager.LateUpdateData(UpdateMode mode, Single minProgress, Single maxProgress)
   at SimulationManager.Managers_LateUpdateData(UpdateMode mode, Single minProgress, Single maxProgress)
   at LoadingManager+<LoadSimulationData>c__IteratorB.MoveNext()
   at AsyncTask.Execute()
   at SimulationManager.SimulationStep()
   at SimulationManager.SimulationThread()

[Info] @ 1485830500 An error occurred while loading.

To me, this is suggesting that my mod data may be corrupt, but I'm not sure. I will provide my save file if necessary to help find a solution to this.

Since the stacktrace mentions it, I'll link in the SerializableDataExtension file.

https://github.com/VictorPhilipp/Cities-Skylines-Traffic-Manager-President-Edition/blob/8e17bfde3bddb8e6c497d58ffc1bd9e6d069f89e/TLM/TLM/State/SerializableDataExtension.cs

As near as I can tell, the specific lines which are referenced are 40, 82 (and the immediately following catch statement), 121, 125, and 131. I believe it gets to at least line 131, as the stacktrace specifically mentions BinaryFormatter.Deserialize.

My best guess for the cause of this is a corrupt save. That said, I'd still like to report it here and find out whether it is my save that is corrupt.

https://github.com/VictorPhilipp/Cities-Skylines-Traffic-Manager-President-Edition/issues/190

[VictorPhilipp]

I did some testing and found a rather unique result. Please note that I am using the files from PR189 in the entirety of this testing.

Using the file provided in the referenced pull request, I would get this error. I tested the workshop version and it worked fine with no error at all, which I found strange. The serialisation code hasn't been touched, so why is this version erroring while the workshop version isn't?

I then tried altering the file version which you'd see in the Properties window, and this did nothing. My save still did not load correctly. Another idea came in to mind, which goes like this.

1. Disable computer Internet connection so that Steam Workshop cannot sync. This is essential.
2. Copy the files provided by the download in PR189 and overwrite the Steam Workshop-provided files in %programfiles(x86)%\Steam\steamapps\workshop\content\255710\583429740 with the PR189 files.
3. Remove the files from %localappdata%\Colossal Order\Cities_Skylines\Addons\Mods to ensure no conflicts.
4. Launch game and test.

Much to my great surprise, my save then loaded correctly and all of my lane connections, timed lights, crossing restrictions, priority signs, etc., were working as they were.

The only thing which has changed is the file location, right?

Maybe I was mis-remembering when I said this issue occurred with workshop 1.10.8.

[AllegedlyLuca]

Additional information: This bug may have been caused by having two separate instances of the same mod operating. Even though one was disabled, only the original mod (the one first added) would properly read the save data. Would recommend investigating by using the two workshop versions of TMPE (this one and LF's one).

[krzychu124]

I've noticed that today while testing other things, thanks 😃

[VictorPhilipp]

@aubergine10 In my opinion that is not a bug but a problem caused by the method we use to serialize/deserialize custom game data. We could use a different library for (de)serialization that handle differences in data structures more gracefully.

When reworking the load/save system we should not forget #149 and #179.

[originalfoo]

As part of #333 I've just updated mod incompatibility checker so it can detect multiple subscriptions and warn about them (treat as mod conflicts):

It's not perfect, but it will drastically reduce likelihood of end-users finding themselves with multiple TMPE subscriptions and resulting issues with customisations being lost.

EDIT: For example, workshop versions won't detect a local build, and if user alters the mod checker settings (eg. to ignore disabled mods) they won't see the warning.

I'm not sure if it can be considered a 'fix' for this issue, as the true fix is #211. As #211 exists, maybe #333 is sufficient to close this issue (#190) as we have another issue tracking the true fix (#211)?

[originalfoo]

As of PR #400, mod incompatibility checker can now successfully detect any other version of TMPE, whether workshop or local, and warn user about them. :)

[originalfoo]

Considering #211 is unlikely to be achieved, primarily as it wouldn't resolve the issue due to the way the game handles assemblies, it seems we're going to have to rely on mod checker to prevent the save/load issues.

As of v11-alpha3, the mod checker will always warn users if there are duplicate TM:PE subscriptions, even if the user has disabled mod checker. IMO that's probably about as close as we can get to avoiding the main issue that duplicate TM:PE causes issues for users.

[kvakvs]

As discussed in the modders discord, even if we don't eliminate the statics, we can achieve a clean shutdown with threads, event handlers and stuff stopped, nulled and detached to minimize the interference. This also might be a low hanging fruit where we should stop to save energy.