search

GeekyDeaks / sim-to-motec

Simple scripts to convert sim telemetry to MoTeC i2 log files
37 stars 5 forks source link

GT7 Lap packet timing discrepancy #8

Open GeekyDeaks opened 1 year ago

GeekyDeaks commented 1 year ago

GT7 appears to send a packet when a new lap starts which is out of sequence of the usual 60Hz telemetry packets. Since i2 requires a steady sample rate, this can cause slight timing discrepancies between what is reported in i2 and in the game.

cybercic commented 1 year ago

Hello there!

Unfortunately, the difference is not that small ...

And I don't believe we have a single difference either. In this worksheet I calculated the differences on a small track, with the lap time around 40 seconds..

Here's what I found:

https://drive.google.com/drive/folders/1Mnz8RPtzfRZLFG64qgU-jeU6IL4Cmdxo?usp=sharing

Can you help me understand how MoTeC handles frequency? One of the hypotheses to correct times would simply be to create a file per lap, indicating the corrected frequency considering the lap time shown by the game and the number of packets received in that lap.

I also wanted to understand the beacon a little better. You will see in the documents that I can separate the distance traveled in the last package by lap. But how does MoteC handle this?

Sei que estou te incomodando com isto, mas representar corretamente os tempos de volta está sendo um obstáculo considerável.

No próximo sábado, em Amsterdam, teremos a World Series Showdown. Você joga Gran Turismo? Ou só AMS2?

GeekyDeaks commented 1 year ago

Ok, you are showing differences of nearly 50ms there which is more than 3 packets. I'm seeing ~10ms in my testing, so I think there is more than just the 'new lap' packet going on here, but either way the 'correction' logic is only supposed to kick in once the error is more than 10 samples (i.e. around 160ms). Would you be able to provide a raw dump of the telemetry for a few laps by enabling the 'rawfile' option and uploading the sqlite db that gets dumped to the logs/raw ?

You could create a file per lap, but it seems like a bit of a sledgehammer approach for something that should only be on average 1/100 of a second error. MoTeC also seems to like having the out and in laps for it's own consistency logic, so it might need some way to keep the samples of the previous and next lap to bleed into the log

On the MoTeC frequency, from what I can tell it simply expects sampling to be done at a specific rate by the sensors in the car. For simplicity, we set the sample rate to 60Hz to match that coming from the game. When the 'new lap' packet comes in, it's like a bit of 'jitter' for that packet, which MoTeC seems to compensate for, but when the error is a lot more I started to see drift. I found MoTeC seemed to be able to cope with up to about 10 missing samples before drift started to be noticeable on a fairly short track like Brands Indy. On a longer track it could handle more.

GeekyDeaks commented 1 year ago

Você joga Gran Turismo? Ou só AMS2?

I mostly play AMS2 as I run events for it, but I play a bit of GT7 from time to time for more casual racing (although the AMS2 racing is pretty casual too!)

cybercic commented 1 year ago

Você joga Gran Turismo? Ou só AMS2?

I mostly play AMS2 as I run events for it, but I play a bit of GT7 from time to time for more casual racing (although the AMS2 racing is pretty casual too!)

But do you know who Igor Fraga is? :-)

cybercic commented 1 year ago

Sorry for the portuguese! My bad on CTRL+C/CTRL+V!

GeekyDeaks commented 1 year ago

But do you know who Igor Fraga is? :-)

Oh that is a blast from the past! Is he still competing in GT? I thought he went back to real life racing?

cybercic commented 1 year ago

But do you know who Igor Fraga is? :-)

Oh that is a blast from the past! Is he still competing in GT? I thought he went back to real life racing?

Himself! He is supported by Polyphony and Gran Turismo in this adventure in Japanese Super Formula and GT300. Because of this, I feel like I'm walking on eggshells lol... He raced yesterday at Fuji for the Lexus. He is very enthusiastic about the project and has been giving suggestions on how to build the workspace in MoTeC. I told him that I started studying your work, Nenkai's and Snipem's! ;-). He asked me to say thank you!

cybercic commented 1 year ago

Ok, you are showing differences of nearly 50ms there which is more than 3 packets. I'm seeing ~10ms in my testing, so I think there is more than just the 'new lap' packet going on here, but either way the 'correction' logic is only supposed to kick in once the error is more than 10 samples (i.e. around 160ms).

Much more than 50ms! Take a look:

Lap 8065. GT7 LaptTime: 43.2530s. Ticks LapTime: 43.0000s. Diff: 0.2530s. Lap 8526. GT7 LaptTime: 89.4520s. Ticks LapTime: 89.1500s. Diff: 0.3020s. Lap 8662. GT7 LaptTime: 104.6300s. Ticks LapTime: 103.9333s. Diff: 0.6967s. Lap 8730. GT7 LaptTime: 48.6990s. Ticks LapTime: 48.4333s. Diff: 0.2657s. Lap 8781. GT7 LaptTime: 104.0520s. Ticks LapTime: 103.2833s. Diff: 0.7687s. Lap 8067. GT7 LaptTime: 43.3870s. Ticks LapTime: 42.1000s. Diff: 1.2870s. Lap 8527. GT7 LaptTime: 89.6820s. Ticks LapTime: 88.8167s. Diff: 0.8653s. Lap 8663. GT7 LaptTime: 104.5310s. Ticks LapTime: 103.4500s. Diff: 1.0810s. Lap 8731. GT7 LaptTime: 48.4000s. Ticks LapTime: 47.8500s. Diff: 0.5500s. Lap 8782. GT7 LaptTime: 103.9040s. Ticks LapTime: 103.1833s. Diff: 0.7207s.

I've just uploaded 2 new files on the Google Drive for you!

Would you be able to provide a raw dump of the telemetry for a few laps by enabling the 'rawfile' option and uploading the sqlite db that gets dumped to the logs/raw?

For sure! I'll collect the data in RAW and let you know as soon as I upload it to Google Drive!

cybercic commented 1 year ago

If I understood correctly, MoTeC deals with packets in sequence, considering 1/60 (in the case of GT7) and we need to inform the beacons, which is the moment when the lap ended, correct?

What if we used the correct lap time that we have in the first packet to adjust the beacon?

Because we already know that this "first" packet also contains the route to finish the previous lap!

cybercic commented 1 year ago

If I understood correctly, MoTeC deals with packets in sequence, considering 1/60 (in the case of GT7) and we need to inform the beacons, which is the moment when the lap ended, correct?

What if we used the correct lap time that we have in the first packet to adjust the beacon?

Because we already know that this "first" packet also contains the route to finish the previous lap!

Tomorrow the Brazilian drivers will be going for the World Series Showdown that will take place on Friday and Saturday in Amsterdam.

At the level they drive, 0.150 is a very big difference, lol...

cybercic commented 1 year ago

Take a quick look:

https://www.gtplanet.net/forum/threads/gt7-is-compatible-with-motion-rig.410728/post-13810797

GeekyDeaks commented 1 year ago

Take a quick look:

https://www.gtplanet.net/forum/threads/gt7-is-compatible-with-motion-rig.410728/post-13810797

Yeah, the current logic that handles the jitter was written based on that info. Thing is, that jitter is only still about 0.016 seconds maximum, which doesn't explain the really high stuff you are seeing, so I suspect something else is amiss

cybercic commented 1 year ago

Take a quick look: https://www.gtplanet.net/forum/threads/gt7-is-compatible-with-motion-rig.410728/post-13810797

Yeah, the current logic that handles the jitter was written based on that info. Thing is, that jitter is only still about 0.016 seconds maximum, which doesn't explain the really high stuff you are seeing, so I suspect something else is amiss

And, as You can see, I'm tracking missing packets ...

GeekyDeaks commented 1 year ago

And, as You can see, I'm tracking missing packets ...

Can you explain where you are doing that? I only really looked at the deltas in the spreadsheet (where I got the 50ms from), but I see you have a bunch of text files however I'm not exactly sure what info is shown in them.

GeekyDeaks commented 1 year ago

What if we used the correct lap time that we have in the first packet to adjust the beacon?

Sorry, I missed this earlier. We are actually doing this. If you look in the .ldx file it has the actual laptime, not one aligned to the sample frequency. We should only be doing a hard correction on the sample time if there is a big difference of more than 160ms or about 10 samples (i.e. GT7 has sent 10 fewer samples than it should have done for the reported laptime). This is normal on the outlap, but seeing it regularly on laps after indicates there is a problem somewhere.

The correction logic is here: https://github.com/GeekyDeaks/sim-to-motec/blob/119cb4aa999d5333bb09eaa279742ecebb4ef010/stm/logger.py#L147-L150

GeekyDeaks commented 1 year ago

For sure! I'll collect the data in RAW and let you know as soon as I upload it to Google Drive!

something just occurred to me looking at the text log files. I was assuming you had found a problem with my scripts, but I'm wondering now if actually you are making your own scripts and hitting the same problems I had at the start. Are those logs from your own scripts?

cybercic commented 1 year ago

The drivers are heading to Amsterdam today. Yesterday we did the last training and I forgot about RAW, can you believe it? :-( But I saved the interface log and recorded a stint video for you to see!

https://www.youtube.com/watch?v=h6bXvBEMgvA

I ulpoad the Id and Idx files in Google Drive.

The log:

cybercic commented 1 year ago

And, as You can see, I'm tracking missing packets ...

Can you explain where you are doing that? I only really looked at the deltas in the spreadsheet (where I got the 50ms from), but I see you have a bunch of text files however I'm not exactly sure what info is shown in them.

The captured data is stored in a BLOB (in Microsoft Azure) and I multiply the PacketId by 10, in order to make the necessary corrections if needed. And the first thing I do in the algorithm is check if I'm missing any packets. I already learned how to calculate the GPS position to recreate these packages!

cybercic commented 1 year ago

For sure! I'll collect the data in RAW and let you know as soon as I upload it to Google Drive!

something just occurred to me looking at the text log files. I was assuming you had found a problem with my scripts, but I'm wondering now if actually you are making your own scripts and hitting the same problems I had at the start. Are those logs from your own scripts?

There are 2 scenarios. In World Series training, as there are only 3 drivers, I use a Python script adapted from yours. See in the picture here (https://drive.google.com/file/d/1gLUb4q2uNL9uawJZy8-9ksOtQj9bhTPX/view?usp=drive_link).

I removed the AMS2 part, lol... And I've carefully studied the details of your work.

I want to have everything running in the cloud, in C#, via Azure Functions. But without paying attention to details like lap time, it's not worth it, is it?

When I decided to ask for help, it was with the feeling that, together, we will be able to improve the result. I want to find some way to collaborate with the community too!

Sorry for some Portuguese, but here is my notes:

ToDo:

Done:

C:\Hobby\MoTeC\SimToMoTeC, 7 arquivos [OK]

BuildGT7.bat - Comando para compilação do executável gt7.spec - Configurações para compilação do executável gt7.py - mVc: Interface com o usuário gt7-cli.py - mvC: Controle do fluxo do programa LaudaGT.cfg - Arquivo INI do programa ReadMe.md - Informações sobre o projeto Requirements.txt - Requisitos básicos do ambiente Python

C:\Hobby\MoTeC\SimToMoTeC\stm, 6 arquivos [OK]

channels.py - Arquivo com as configurações dos canais para o MoTeC [CHANNELS] event.py - Arquivo com as configurações dos eventos para o MoTeC [STMEvent] gps.py - Função para converter as coordenadas do GT7 para Lat/Lon [convert(x,z)] logger.py - Base para a captura e processamento dos pacotes do GT7 [BaseLogger] sampler.py - Captura e organização dos pacotes recebidos [BaseSampler] version.py - Arquivo com a versão atual do projeto [version]

C:\Hobby\MoTeC\SimToMoTeC\stm\gt7, 4 arquivos [OK]

logger.py - Captura e processamento dos pacotes do GT7 [GT7Logger] packet.py - Leitura dos pacotes recebidos [GT7DataPacket] pure_salsa20.py - Funções para descriptografar os pacotes [GT7DataPacket] sampler.py - Captura, organização e redirecionamento dos pacotes recebidos [GT7Sampler]

C:\Hobby\MoTeC\SimToMoTeC\stm\gt7\db, 8 arquivos [OK]

cars.py - Função para localizar os nomes dos carros a partir do ID [CARS] tracks.py - Função para identificar as pistas a partir das coordenadas [TRACKS] e [GT7TrackDetector] bounds.csv - CSV com as coordenadas das pistas cars.csv - CSV com os nomes dos carros courses.csv - CSV com os nomes das pistas logo.ico - Ícone do arquivo EXE logo.png - Logo LaudaGT para a interface partners.png - Outros logos para a interface

C:\Hobby\MoTeC\SimToMoTeC\stm\maths, 2 arquivos [OK]

quaternion.py - Função matemática para calcular a rotação do carro vector.py - Classe para retornar os vetores de posição e velocidade

C:\Hobby\MoTeC\SimToMoTeC\stm\motec, 2 arquivos [NEXT]

ld.py - Estrutura do arquivo ld ldx.py - Estrutura do arquivo ldx