search

Strdate / PythonConsole

Cities Skylines mod
25 stars 4 forks source link

New feature proposal #1

Open HuangFuSL opened 3 years ago

HuangFuSL commented 3 years ago

Strdate:

Great work! This mod is exactly what I need in the C:S game. But I think the project would be even better if this MOD provides the following features:

  1. Python 3 support. Resistence is that the dot-net-based IronPython is Python 2.7 and does not support Python 3. CPython can be used instead of IronPython, with Python-side applications shipped using PyPI. The user should provide path to the Python interpreter or add it to PATH. This approach provides a more convenient way for accessing the in-game data outside the game.
  2. Citizen activity tracker. Apart from controlling buildings and props, I want to collect the citizen activity, public transport and traffic data for further analysis. Is it approachable in this mod? Does the C:S game provide relevent interfaces? Or will these functions affect game performance significantly? Since I am a greenhand for C# programming and MOD development, I'm just proposing my ideas.

If there are any obstacle in developing this MOD, I think I can help with Python.

Best regards, HuangFuSL

Strdate commented 3 years ago

Hi, I am happy that you like this mod :)

  1. When I started this project I grabbed the first python implementation in dotnet that I could find. I'm not a regular user of Python so the version didn't matter to me much at that time. I found this implementation of python 3. Switching should not be that difficult. I will try to ship that with next bigger update.
  2. The current set of APIs that this mod provides is only a scratch of what can be done. I can definitely expose more functionality regarding citizens etc., but implementing some-kind of constant monitoring is quite challenging. The game itself doesn't collect much statistical data, so you need to do this yourself and that is quite difficult (you need to patch a lot of gme methods etc). Due to performance reasons this monitoring should be done directly in dotnet anyway. Then you could poll the data with python only when you need.

Of course, there are some mods that already do some kind of monitoring (such as IPT2), but then there is the overhead of making the mods work together.

mbeleuca-cpa commented 3 years ago

Hi Strdate, did you manage to replace the IronPython package with the Python.NET?

Strdate commented 3 years ago

I didn't. Since the mod's first release I just added a few APIs that were requested by the mod's only regular user I know. Switching to a different library would require me now to rewrite a big chunk of the code and unfortunately I'm not very motivated to do that unless the mod would gain more traction or somebody would convince me that they really need it for their project and I would clearly see the benefit.

Anyway, the ideal case is that you can change the engine as easily as selecting it in the game settings and the code being generic without any direct references to any of the engines. I hope one day we'll get to that.

HuangFuSL commented 3 years ago

Here, I have a possible solution. The Python engine of this mod is an external application and communitates with the game via network. Maybe we can build a client in Python, which can be executed on any Python engines? As far as I know, module SkylinesRemotePythonDotnet does not contain reference to game core APIs such as ICities or UnityEngine so I think it's feasible.

Strdate commented 3 years ago

Sure enough, SkylinesRemotePythonDotnet can be replaced by any application that would mimic its APIs. But you probably don't want to do it, as it means rewriting everything from scratch. Instead it would be better to replace engine-specific code with a wrapper class that could be then implemented in any engine.

You can see some engine specific code here https://github.com/Strdate/PythonConsole/blob/cb372b652106440a5a69ffb8b16274f47c305503/SkylinesRemotePythonDotnet/PythonEngine.cs#L23

jqdq commented 3 years ago

Are you planning on implementing anything related to citizen activity/traffic flow in near future?

HuangFuSL commented 3 years ago

@Strdate I don't think rewriting the application is something difficult, but I do think re-implementing .NET specific protocols such as BinaryFormatter in Python is troublesome. 😥

Strdate commented 3 years ago

@PogromcaPapai I would like to, but it would be quite a performance killer to have these things directly linked to python. It needs a lot of intermediate logic which is quite time consuming to implement and maintain

Strdate commented 2 years ago

@HuangFuSL You can reimplement the protocol with JSONs instead. I can merge it to master if it would be fast enough. It requires a few changes here https://github.com/Strdate/PythonConsole/blob/master/SkylinesPythonShared/Protocol/TcpConversation.cs

Just now I merged quite a lot of changes to the mod so watch out for API changes.

HuangFuSL commented 2 years ago

@Strdate I've completed serialization and deserialization methods just now (see commit ea50a5a) using DataContractJsonSerializer. The work is INCOMPLETE up to now due to the lack of data contracts.

However when I started the game to see its behavior. The external application throws the following exception:


This is remote python engine window for Cities:Skylines

You can change the default tcp port in Cities_Skylines\PythonConsoleMod.xml
Startup...
Listening on 6672...
Accepted new connection
System.Runtime.Serialization.SerializationException: There was an error deserializing the object of type SkylinesPythonShared.MessageHeader. Encountered unexpected character ''. ---> System.Xml.XmlException: Encountered unexpected character ''.
   at System.Xml.XmlExceptionHelper.ThrowXmlException(XmlDictionaryReader reader, XmlException exception)
   at System.Runtime.Serialization.Json.XmlJsonReader.ReadAttributes()
   at System.Runtime.Serialization.Json.XmlJsonReader.ReadNonExistentElementName(StringHandleConstStringType elementName)
   at System.Runtime.Serialization.Json.XmlJsonReader.Read()
   at System.Xml.XmlBaseReader.IsStartElement()
   at System.Xml.XmlBaseReader.IsStartElement(XmlDictionaryString localName, XmlDictionaryString namespaceUri)
   at System.Runtime.Serialization.XmlReaderDelegator.IsStartElement(XmlDictionaryString localname, XmlDictionaryString ns)
   at System.Runtime.Serialization.XmlObjectSerializer.IsRootElement(XmlReaderDelegator reader, DataContract contract, XmlDictionaryString name, XmlDictionaryString ns)
   at System.Runtime.Serialization.Json.DataContractJsonSerializer.InternalIsStartObject(XmlReaderDelegator reader)
   at System.Runtime.Serialization.Json.DataContractJsonSerializer.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName)
   at System.Runtime.Serialization.XmlObjectSerializer.InternalReadObject(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
   --- End of inner exception stack trace ---
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.Json.DataContractJsonSerializer.ReadObject(XmlDictionaryReader reader)
   at System.Runtime.Serialization.Json.DataContractJsonSerializer.ReadObject(Stream stream)
   at SkylinesPythonShared.TcpConversation.JsonDeserialize(Byte[] src, Type T) in C:\Users\huang\Documents\GitHub\PythonConsole_working\SkylinesPythonShared\Protocol\TcpConversation.cs:line 125
   at SkylinesPythonShared.TcpConversation.JsonDeserialize[T](Byte[] src) in C:\Users\huang\Documents\GitHub\PythonConsole_working\SkylinesPythonShared\Protocol\TcpConversation.cs:line 142
   at SkylinesPythonShared.TcpConversation.Deserialize(Byte[] data) in C:\Users\huang\Documents\GitHub\PythonConsole_working\SkylinesPythonShared\Protocol\TcpConversation.cs:line 79
   at SkylinesPythonShared.TcpConversation.AwaitMessage() in C:\Users\huang\Documents\GitHub\PythonConsole_working\SkylinesPythonShared\Protocol\TcpConversation.cs:line 58
   at SkylinesRemotePython.ClientHandler.GetMessage(Int64& requestId, Object& result, String& error) in C:\Users\huang\Documents\GitHub\PythonConsole_working\SkylinesRemotePythonDotnet\ClientHandler.cs:line 59
   at SkylinesRemotePython.ClientHandler.HandleClient() in C:\Users\huang\Documents\GitHub\PythonConsole_working\SkylinesRemotePythonDotnet\ClientHandler.cs:line 40
Press any key to exit...

It seems that the internal mod and the external application use different versions of SkylinesPythonShared library. But I've already update the files in the following directories.

  1. C:\Program Files (x86)\Steam\steamapps\workshop\content\255710\2415465635
  2. C:\Users\huang\AppData\Local\Colossal Order\Cities_Skylines\Addons\Mods\PythonConsole

Any idea?

Strdate commented 2 years ago

You are using different serializers to convert string to byte[] and vice versa. Try using Encoding.UTF8 everywhere

HuangFuSL commented 2 years ago

I've turned to XML serialization (see commit 9390551). Though I've not finished RPC part, it can now execute scripts irrelevant with game API. The kernel is based on Jupyter, which supports Python 3.5+. With it, you can use your local Python environment and all your PyPI packages are available. RPC part is a bit troublesome ant will take some time as the python interface and python kernel are separate processes.

HuangFuSL commented 2 years ago

The demo version is almost done (commit 08aadf3). Summary of API changes are listed below:

Not yet finished:

  1. Test for whether they run as expected
  2. Object caching
  3. Object iterator
  4. Nature resource support
  5. Docstrings
  6. Some unexpected behavior when executing scripts
  7. Server-side exception handling
  8. Jupyter-notebook support

Features deprecated:

  1. PythonHelp, I've turned to Python built-in help() method, which is easier to implement.
  2. engine.async_mode, which is moved to game class

Features added:

  1. PyPI package support
  2. asyncio support: you can use await expression directly in a cell
  3. logging support: five logging levels are present (debug, info, warning, error and critical)
  4. auto-print execution result: result of the last expression executed in a cell will be automatically displayed

I've tried my best to maintain compatibility with C# version of the client, but I cannot ensure that two clients are 100% compatible.

You can execute the following command in the root directory of this repo to start the client

python -m client
Strdate commented 2 years ago

Good job. I will try to merge the xml communication protocol soon after i test it out.

Later I can add the python client as an option in the menu after you finish most of the key features.

On Sun 31. 10. 2021 at 10:35 HuangFuSL @.***> wrote:

The demo version is almost done (commit 08aadf3 https://github.com/HuangFuSL/PythonConsole/commit/08aadf374b7e266c2e1e9b5da3151a1cd0a725df). Summary of API changes are listed below:

Not yet finished:

  1. Test for whether they run as expected
  2. Object caching
  3. Object iterator
  4. Nature resource support
  5. Docstrings
  6. Some unexpected behavior when executing scripts
  7. Server-side exception handling
  8. Jupyter-notebook support

Features deprecated:

  1. PythonHelp, I've turned to Python built-in help() method, which is easier to implement.

Features added:

  1. PyPI package support
  2. asyncio support: you can use await expression directly in a cell
  3. logging support: five logging levels are present (debug, info, warning, error and critical)
  4. auto-print execution result: result of the last expression executed in a cell will be automatically displayed

I've tried my best to maintain compatibility with C# version of the client, but I cannot ensure that two clients are 100% compatible.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Strdate/PythonConsole/issues/1#issuecomment-955667006, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRWLORJSJINPWPKOPY7NXLUJUEWRANCNFSM4YW427XA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

VirtualCartographer commented 2 years ago

I would like too see integration with procedural objects, so I can place a PO cubes with custom texture and location from the console.

Strdate commented 2 years ago

https://github.com/Strdate/PythonConsole/pull/11 is almost done. I just need to resolve a few bugs.

VirtualCartographer commented 2 years ago

What is #11 I don’t see it mentioned earlier

Strdate commented 2 years ago

That's just migration to JSON serialization - just an internal thing.

Unfortunately no new features for now :(