search

RenCloud / scs-sdk-plugin

ETS2 (Euro Truck Simulator 2) & ATS (American Truck Simulator) SDK plug-in. Telemetry data is shared via SharedMemory/Memory Mapped Files.
MIT License
185 stars 35 forks source link

Cross-platform plugin on TCP #91

Open ndelta0 opened 3 years ago

ndelta0 commented 3 years ago

This version uses no external libraries. It's implemented with TCP sockets for each of the OSes (Windows and Unix). The server listens on 0.0.0.0:45454. The transmission format is as follows:

  1. Send 4 byte (32 bit) integer (Int32) with the size of the data that will follow.
  2. Send the telemetry data. The format is unchanged and is compatible with previous releases and clients.

On my computer (i7-4790k) there were no perfomance drops in the game with the telemetry being sent every simulation frame. From what I saw in IO Ninja, the bandwidth that it requires is applicable only for local network data transfer and fast networks (I think it was about 1.5-2 MB/s).

Known Issues:

Compatibility: The project is set to build on Linux x64, Windows x64 and Windows x86. I only tested Windows x64 and Windows x86. It's not confirmed nor said to work on MacOS, but it's possible to work with minor or no tweaks.

RenCloud commented 2 years ago

I will probably make some tests today/tomorrow to check the performance of the implementation.

On first view I had again some problems with TCP, but I think when the performance is fine it allows some feature to build much faster than with shared memory.

In addition, I think it makes the development of other clients simpler.

That means a lot of plus points for TCP if the performance is right.

ndelta0 commented 2 years ago

I've also considered using UDP instead of TCP, but it would be too unreliable and you could lose important events

RenCloud commented 2 years ago

Performance seems to be ok. Also, I have some ideas for better performance, so that should not be such a problem.

2 other things are more of a problem:

ndelta0 commented 2 years ago

I'll look into that

ndelta0 commented 2 years ago

Fixed the issue with crash on sdk unload. I couldn't reproduce the Alt + Tab issue, it might be related to how the game is made since it doesn't really like to be alt tabbed from. If anyone reproduces it send the game log here.

alphavector commented 2 years ago

I was able to build a plugin for mac os with minor edits in the code. The game itself runs, but I don't really understand where I should look at the telemetry readings. When I access 0.0.0.0:45454 I get ERR_INVALID_HTTP_RESPONSE, in logs <WARNING> Error sending: 32. @ndelta0, I will send pr to your branch if you are still interested in developing it.

ndelta0 commented 2 years ago

I was able to build a plugin for mac os with minor edits in the code. The game itself runs, but I don't really understand where I should look at the telemetry readings. When I access 0.0.0.0:45454 I get ERR_INVALID_HTTP_RESPONSE, in logs <WARNING> Error sending: 32. @ndelta0, I will send pr to your branch if you are still interested in developing it.

Well, the listening on 0.0.0.0:45454 means that it listens on every interface, i.e. you can access it on localhost:45454 and by using your local device IP from other device.

Regarding ERR_INVALID_HTTP_RESPONSE, it was not meant to be read by a browser (i.e. not JSON or any other plain text data). It's a raw stream of bytes exactly as you would read it from the memory mapped file. You need to open a raw/stream TCP socket to be able to read the data. Tell me what language you're using and I'll send you a sample code.

alphavector commented 2 years ago

If you don't mind, send a sample, in Python, for example

ndelta0 commented 2 years ago

If you don't mind, send a sample, in Python, for example

The code for receiving data in Python would look something like this:

import socket # import the socket library

def main():
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: # create the socket object with the Internet address family (IPv4) and stream socket type
        s.connect(("127.0.0.1", 45454)) # connect to the plugin listener
        while True:
            data = s.recv(1024) # receive the bytes
            print(data) # print the bytes

if __name__ == '__main__':
    main()

I was able to build a plugin for mac os with minor edits in the code. [...] I will send pr to your branch if you are still interested in developing it.

I'd love to, I don't have any Apple device myself so it would be really useful for me

ndelta0 commented 2 years ago

NOTE: Marked the PR as draft since it's not 100% finished and there's still room for improvement

alphavector commented 2 years ago

Yes, we almost synchronously managed to write the code to get the data :D I get a sequence of bytes, but how to deserialize it? I understand correctly that this sequence of bytes is a structure scsTelemetryMap_t?

ndelta0 commented 2 years ago

I get a sequence of bytes, but how to deserialize it? I understand correctly that this sequence of bytes is a structure scsTelemetryMap_t?

Exactly, except that every "message" is prefixed with a 4 byte (Int32) value specifying the length of the following data. It can be ignored if you know the data size and have a buffer of the appropriate size (which is required to be pre-allocated before receiving the data in languages like C/C++/C#/etc, I don't this is required in Python). The actual data is exactly as in the telemetry map.

ndelta0 commented 2 years ago

Also, since this is a draft pull request, anyone should be able to commit although I don't really know how that works.

ndelta0 commented 2 years ago

TODO:

alphavector commented 2 years ago

I was able to serialize the data into a structure using ctypes.Structure. It was quite time consuming to fully describe the whole scsTelemetryMap_t structure in Python, luckily I was able to partially automate it with pycparser. @ndelta0, have you thought about looking into gRPC? It should still be pretty fast for data transfer (HTTP 2.0, data compression), supports streaming. In that case we would need to describe only 1 proto file, and with a code generator we would get a client for the desired programming language

In any case I will try to sketch a minimal working client for Python and send pr.

ndelta0 commented 2 years ago

[...] have you thought about looking into gRPC? It should still be pretty fast for data transfer (HTTP 2.0, data compression), supports streaming. In that case we would need to describe only 1 proto file, and with a code generator we would get a client for the desired programming language

gRPC is also an option. Initially I wanted to use ZeroMQ and use different topics (events) for things like telemetry, job start, end, crash, etc. But @RenCloud wanted this plugin to stay dependency-free, so I went with pure sockets implementation.

EDIT: It's always possible to create a separate branch/fork with a ZeroMQ/gRPC implementation

RenCloud commented 2 years ago

Yeah i would like to work without a library, but i would like it, that if someone want it, it is simple to do. Kinda like just change the imported header file or write a small class that exdends an interface. But this is maybe the next step.

I will take a look at the new changes and test it tomorrow. When it works i will test win 10 and linux ( ubuntu 21.10 i guess) . After that i would see what to do then to finish this here ( i think client was missing, but that should be than in a new repository or so)

To see that it works en all 3 major platforms would be nice, so i will try to get this pr done. However, i can not test mac os.

@ndelta0 I think you mentioned somewhere that you planned to make the events in their own calls or something like that? I would like that. My plan was to split the data in 2 sets and than the events their own calls. But i need to look how you make that with tcp with good practice.

2 sets are similar to the ones in the SCS files:

This would reduce the packets that are send often, because then the big string fields are not send constantly and i think than the performance impact on the game is really small. Also for slower PC's.

Did you think that change( using 2 separated dada sets) make sense?

ndelta0 commented 2 years ago

I think you mentioned somewhere that you planned to make the events in their own calls or something like that? I would like that. My plan was to split the data in 2 sets and than the events their own calls. But i need to look how you make that with tcp with good practice.

2 sets are similar to the ones in the SCS files:

  • more or less static data (refreshed barely just on specific game events like truck model)
  • rapidly refreshed date( location, speed, ...)

This would reduce the packets that are send often, because then the big string fields are not send constantly and i think than the performance impact on the game is really small. Also for slower PC's.

Did you think that change( using 2 separated dada sets) make sense?

Yes, something similar. I thought, that it could be possible (with the TCP server) for the client to specify what events/topics it would like to subscribe to and have events e.g. "telemetry" (for information like truck speed, location, rotation, etc), job related (i.e. "jobStart", "jobEnd"), config (i.e. truck/trailer info, "substances" or more) and other like "speeding", "crash" and other fines. The client could either send the subscribed topics at the connection time or anytime during the connection if need be.

ndelta0 commented 2 years ago

Also one thing that could be done is instead of sending data as received from the game send it in a more optimised format (at least change the arrays from fixed length to length prefixed, which would cut the current data size at least by 3-5 times - the default is 10 trailers, of which 8 are always empty - and the strings to UTF-8 length prefixed strings.

RenCloud commented 2 years ago

That would be a way. I will have to read a bit about TCP to see what and how a good way is to do this.

Indeed the 10 possible trailers are only available through mods i guess, so for most of the users there are useless data. With prefix you mean something like 0trailer... That should do the job.

And for strings 4Test? I am not sure why a prefix should be used here instead of something like Test\0?

ndelta0 commented 2 years ago

That would be a way. I will have to read a bit about TCP to see what and how a good way is to do this.

Indeed the 10 possible trailers are only available through mods i guess, so for most of the users there are useless data. With prefix you mean something like 0trailer... That should do the job.

And for strings 4Test? I am not sure why a prefix should be used here instead of something like Test\0?

I thought about it in a more binary sense, i.e. prefixing the count of items in array with a Int32 or less if it would be enough, that way only the required amount of items can be transmitted, omitting the unused entries (in vanilla ETS2/ATS that would save 8 trailer entries each time).

Regarding the strings they also can be null terminated, but usually it's easier to know the length beforehand, you can then allocate a big enough buffer before getting the actual UTF-8 bytes. It makes the receiving easier, since arrays cost a lot to expand and adding to a dynamic list is costly as well.

N95JPL commented 2 years ago

Is there any kind of "estimated date" for when this will drop? I am working on a new application which will tie in to the SDK which will make it easier for people to use in their programs/Arduino Connections/WebServers etc. Currently writing it in C# / ASP.NET with WinForms however, if this is going to be in the near future I will double-back and start working on a leaner, meaner React project for it instead...

Thanks, JPL!

ndelta0 commented 2 years ago

Is there any kind of "estimated date" for when this will drop?

Technically it's ready, you can already use it when you download the artifact on my fork or build it yourself (keep in mind that artifacts are currently x64 only).

N95JPL commented 2 years ago

Is there any kind of "estimated date" for when this will drop?

Technically it's ready, you can already use it when you download the artifact on my fork or build it yourself (keep in mind that artifacts are currently x64 only).

I had actually downloaded the src and built with CMake (Wins x64) and then built the Solution and it appears to be working, I didn't notice the artifact!

I connected via a console and received data (albeit a bunch of scrambled chars/symbols)... i will hook up a JS script later and get some bytes!

ndelta0 commented 2 years ago

I connected via a console and received data (albeit a bunch of scrambled chars/symbols)... i will hook up a JS script later and get some bytes!

Actually it's supposed to be what you received, these bytes are actually exactly what was used before, with addition of an Int32 (4 bytes) at the beggining, which indicate the size of the following payload. Current implementations that receive and parse the data can be adapted by basically receiving the data from a TCP socket instead of a Memory Mapped File and skipping the first 4 bytes when receiving the data.

EDIT: Unlike ETCARS which sends the data in a JSON format, this plugin basically sends the memory that's assigned to the telemetry map type

RenCloud commented 2 years ago

I finally managed to look at the changes in more detail. I'm sorry for delaying this pr so long, but it's a big change, so I need a bit of time which I did not manage to put into this project lately. Beside some smaller support, this big change was sadly too much.

Thanks for that much afford. Also fixing old mistakes and formatting mistakes.

Most parts look good. Here and there I will change code formatting, but that's more or less personal preferences. I will check for a linting GitHub action in the future.

I currently prefer the current event handling implemented with rev 11, but that was later introduced than the start of this pr. ... Maybe neither the new nor the old way will stay, when subscriber pattern is used? At least, I think it would be possible to broadcast the data direct at the gameplay event. Then it would be an event by itself and no extra flag/switch/state/... needed. But that's just an idea at the moment.

Some minor things I need to test, because I may have some questions on them. There are also some small changes I may do. Possibly I can get rid of some commentaries and old to-dos before creating the new dev state out of it.

Beside the other points, I think

Investigate freezing the game when receiving socket is blocking

is one of the important to fix, before merging.

For memory leaks, it will probably code check up and observing application memory while doing a longer test. Perhaps I find some tool that can help here.

Additional to fixing mentioned things, I'm currently thinking if it is useful to add the subscriber pattern in this pr or later with an additional pr same with string and trailer optimizations.

N95JPL commented 2 years ago

ETS2-TCP-Telemetry-Byte-Map.xlsx

This is a Excel document I made to quickly reference where each Value can be found. Saves having to count and use my calculator all the time!

It's here for anybody to use if they wish too!

You'll notice "End Byte" has 2 columns, this is because some languages want the byte after the range you want (non-inclusive), whereas others are happy with inclusive ranges!

Enjoy, Thanks JPL!

RenCloud commented 2 years ago

I felt free to directly pushed into your branch, I hope that's fine for you.

Most/all are format changes or changes on comments. And remove the logic from events bits, like in the current main branch.

I tested it a bit with the above python script and displayed in game speed. Seems to still work after my changes. Also, logging seems to work wonderful. The Excel sheet was already helpful for that. Thanks JPL. May I should include a Markdown table with that information.

Performance seems to be ok/good already (windows 10). However, I'm exited to work on the optimizations for strings, trailer and the pattern.

-- Currently, I think getting this stage into dev-branch and after that working on the next steps (creating client lib, subscriber pattern, optimizations on code and transmitted object) is the way to go here to bring this big change to a "production ready" state.

I will check this on Linux and go another time through the code if I overlooked something. If I have then no questions/changes, and you have no changes, I would like to merge it and work than one the next steps

alphavector commented 2 years ago

I would like to suggest a few additional changes:

Further ideas:

In spite of all the above, I've tested it working in OSX (and even on an Apple Mac M1) -- it works, works great.

ndelta0 commented 2 years ago
  • I propose to allow you to specify an arbitrary port via environment variables and not be tied strictly to port 45454.

That's a great idea, I've thought about that but never got around to implement it

  • any connection to the server (i.e. the game) blocks the game [...] Perhaps the problem is due to the fact that in osx you can't specify SOCK_NONBLOCK in the accept function

I'm aware of this issue, it's just hard to do any asynchronous socket communications on any system, especially in C/C++. I haven't encountered this issue on Windows and Linux (with accepting), but I've encountered a problem with the game hanging when the client does not receive data and fills the buffer.

EDIT: I don't have any way of running ETS2/ATS on a MacOS system (unless you have some way to run MacOS in a virtual machine, DM me then), so any help with MacOS sockets is very appreciated.

N95JPL commented 2 years ago

@alphavector I have sort of worked around the game stutters because of the connection system by using socket.pause() // socket.resume() whilst I'm parsing the last buffer stream. Admittedly I'm using Node.JS wrapped in Electron with Async functions, even though it's only paused for a slight fraction of a second it stops the client program getting bogged down trying to constantly parse the data when it's being fired at continuously. I'm hoping to speed things up again and not require the pause but for now it doesn't seem to affect anything. Also, make sure you skip any data <1000 bytes, I've found this helps. I've noticed a load of 0 packets getting sent? This might throw it off too! Hope that helps! JPL

Edit: Disclaimer, I've not focused on events yet, but everything else works as normal! Truck/Game/Job etc If I notice issues with events with my current code structure I'll update the thread!

RenCloud commented 2 years ago

The ideas are great. Will come on the list. Some of them are already on that list. The functions will be added when this pr is in dev. At least that is my current plan.

That there is still some blocking is not so great. I also can not test OSX. However, I found this code

int srv = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
int flags = fcntl(srv, F_GETFL);
fcntl(srv, F_SETFL, flags | O_NONBLOCK); // setting the socket nonblocking

This sets the non-blocking with another syscall. Maybe this is possible? Could you try this out?

Tested Code for Windows and non-blocking... added a sleep in python and non-blocking, but totally behind the live values for sleeps bigger than maybe 100ms or so. For 1 sec, the data doesn't seem to update at all. I'm not sure if I did here every thing correct. Need to check that.

set the socket to non-blocking

    u_long iMode = 1;
    int iResult = ioctlsocket(srv, FIONBIO, &iMode);
    if (iResult != NO_ERROR) {
        printf("ioctlsocket failed with error: %ld\n", iResult);
    }

and the error WSAEWOULDBLOCK needs to be handled after accept like for Linux and while sending it needs to be ignored as error.

Just for curiosity, how big was the zeroQ implementation at the end? I need to know how much salt this non-blocking and subscribing will create, because I said to not use an external library.

alphavector commented 2 years ago

@RenCloud, thanks for the idea, I'll try it on the mac. Sorry for being away for a while. :D I had the idea to try to implement a GRPC server, so far I managed to convert the telemetry structure into a proto file and write an asynchronous grpc server. In the near future I plan to finish it. And we will be able to evaluate all advantages and disadvantages of tcp/zeromq/grpc and choose the best way, if nobody minds.

Regarding the implementation of the possibility to specify the user port: there are problems here, if we specify the environment variable in the terminal, it is available only during the session (while the terminal is open). If you specify in ~/.bashrc ~/.zshrc, then again, this variable is only available for the terminal, the game does not see it. I think it would work if you run the game through the terminal, something like SCS_TELEMETRY_PORT=123 zsh ~/path/to/game/executable, but do people often run the game this way? :D I see 3 ways here: 1) the game itself has a configuration file, if we have access to it from the plugin, we can ask the user to specify the port there 2) Have your own configuration file 3) In steam we can specify additional launch options, is it possible to catch this.

N95JPL commented 2 years ago

@alphavector I use the Terminal for a lot of things but running games isn't one of them 🤣 I can be 99% certain 99% of people will just click "Play Game" in Steam or use the desktop shortcut! So, unless you provided a shortcut for each OS which in turn launches the Game (IIRC it's a URL type-based system for Steam Shortcuts?) and sets a parameter that a user can change if they wish? Most people will default to the "standard" port, more advanced users might want different ones but then they will know how to simply edit the shortcut?

--EDIT-- Further, I've built a JS Application which parses all the stream data into JSON and publishes it either via a WebSocket or via a API-Based system (can enable/disable to your liking), I'll upload a demo soon!

JPL

alphavector commented 2 years ago

That's what I meant, it was a rhetorical question :) 99% don't run games through the terminal

RenCloud commented 2 years ago

Sorry for being away for a while. :D I had the idea to try to implement a GRPC server, so far I managed to convert the telemetry structure into a proto file and write an asynchronous grpc server. In the near future I plan to finish it. And we will be able to evaluate all advantages and disadvantages of tcp/zeromq/grpc and choose the best way, if nobody minds.

We can do this. I'm just a little dissatisfied that ndelta0 first created a version with zeromq, then at my request another one without library and now maybe it will be one with library 😅.

But when the end result is better for the user and makes the development so much simpler, I guess I have no real argue against using one header file... 😄

blitzcaster commented 2 years ago

Hi, I'm also interested in Protocol Buffers format. Even without GRPC and using the current memory mapped file it would help a lot. Since protobuf parsers are already written in many language and are production ready codes. @alphavector any chance to open source your proto files? It would help implementing at least for the memory mapped structure.

alphavector commented 2 years ago

Hi, I'm also interested in Protocol Buffers format. Even without GRPC and using the current memory mapped file it would help a lot. Since protobuf parsers are already written in many language and are production ready codes. @alphavector any chance to open source your proto files? It would help implementing at least for the memory mapped structure.

I promise to finish it soon For now, I'm sharing the proto file https://gist.github.com/alphavector/ea6b427812db418f12d983a2d9e01dc0

blitzcaster commented 2 years ago

I promise to finish it soon For now, I'm sharing the proto file https://gist.github.com/alphavector/ea6b427812db418f12d983a2d9e01dc0

Thank you for answering and sharing the proto files, it would help me quite a lot. Also there's no need for promises, it's an open sourced project and you should finish it at your own pace.

AltriusRS commented 1 year ago

Hi, just curious if there has been any progress on this PR since the last comment.

I am writing an application based on Electron however versions of electron higher than 20.3.8 have issues with memory mapped files being loaded due to some security changes, so I am stuck using an older version of Electron.

If this PR is almost ready to use id be happy to trial it for a few weeks across my application as a development rollout and report back any difficulties I have in getting it set up?

Massive appreciation for you all who have worked on this PR so far, it is a life saver for people trying to use this with JS/TS projects, and anyone trying to integrate this project into a CI/CD pipeline.

ndelta0 commented 1 year ago

@AltriusRS as You said it, the PR is basically ready, with exception of few quirks and minro bugs. One big downside is that it's for SDK version 1.17 (ETS2: 1.44, ATS: 1.44), but it will work (just won't have the new available data). Since it's nor merged into the main repo, you need to compile it yourself and redistribute it. If you need, I can provide you with help on how to do that. In that case contact me through any means listed here: https://k4rasudev.me/contact.

AltriusRS commented 1 year ago

sounds good, Ill be busy for a couple more weeks with my current work, but I shall get back to you once I have gotten it working to report my findings and difficulties.

TheBreekoMan commented 8 months ago

Hi there, I'm currently working on a StreamDeck plugin which get telemetry data from your plugin and change display of icons keys according to parsed data. The main limitation is about reading SharedMeMoryFile which is not possible as Streamdeck plugin is running within a closed process environment AFAIK. BTW, I succeed to wrap data from SHM and expose it with a custom python REST server. So the addition of network streaming stack could be really great. Some remarks :

Anyway you all made a great job and would like to thank you for that and your effforts.