Define a clear JSON api

[zelogik]

First draft:

{
    "config": {
        "lap": 10,
        "player": 4,
        "gate": 3
    },
    "state": "stop",
    "live": {
        "rank": 1,
        "best": 0,
        "current": 0,
        "last": 0
    }
}

[Peck07]

sounds good, two comments:

• I wouldn't set the player number (if not necessary), should be dynamic, coming from the transponder IDs.
• the "live" part could be exactly the same as in the working webui code at least for starters.

[zelogik]

• thinking about sending only the "rank"/"order" of the transponder ID of course (as the controller/server already have a fast and working algorithm) and it could help at frontend side (or maybe not...). But we need to find a clean way to "dynamically" set a transponder ID with a name/number player from the frontend and sync with all others frontend/browser connected to it....(the server don't have to know/save the name/number player, just receive and sync to all clients.)

maybe add a client JSON request like { state: waiting }, asking for server to send the first transponder ID received and stop.

• yes the "live" part seem a good start as I have already passed so much time, to get it working :-D

IE: Sorry to making a big spring cleaning, and reorder/rename folder, but even me can't remember what I have done ;-D

[zelogik]

Edit:

Example ot the message sent to the "serial live console": Serial.print(F(" |ID: ")); Serial.print(F(" |laps: ")); Serial.print(F(" |Time: ")); Serial.print(F(" |last: ")); Serial.print(F(" |best: ")); Serial.print(F(" |mean: ")); Serial.print(F(" |1: ")); Serial.print(millisToMSMs(idData[j].lastCheckPoint[1])); Serial.print(F(" |2: ")); Serial.print(millisToMSMs(idData[j].lastCheckPoint[2])); Serial.print(F(" |3: ")); Serial.print(millisToMSMs(idData[j].lastCheckPoint[0]));

That mean that server DOESN'T know anything about player number/name only the rank/position/order (that information only missing if we want to store the 8 best lap on the EEPROM with name :-)... or maybe to display name in the serial console/TFT too)

[Peck07]

"as the controller/server already have a fast and working algorithm) and it could help at frontend side (or maybe not...)." yes, I would leave it to the server, and keep the frontend lightweight.

player name/number: my idea: don't overcomplicate it, if there's nothing set, the transponder ID will be the name, or some random string :) Of course it should be changeable from client (simple JSON with transponder id, player name)

[zelogik]

I don't want to overcomplicate it, just find a solution to sync client name to all other client connected to the server. In a nice and clean way. But we can go without that feature for the moment ;) Same problem I have faced before... what the better way to "refresh" current time of each player lap (check point portion) and current total race time:

• On the client side (with possible drift between displayed client time and the real(and official) server time)
• On the server side, but with the problem to send JSON message every 100-200ms minimum to refresh all times on all clients, and so, don't have a "non-sluggish" UI.

[Peck07]

The way I proposed, the sync will be automatic between the clients, as the server "stores" the names, if you change a name from one client, after a while, it will be synced between clients (as the information comes from the server eventually). Of course, this is a nice to have feature, but not a top priority. And we can proceed in steps. Like first step: introduce a name, have a fixed/random name for everyone :). Seconds step: make it changeable.

race time: I have seen the Stopwatch class :) I think it's a great idea to run some timer on the client side, but sync/update sometimes from server data. (I think this was the original idea, but I don't think it's working correctly right now)

[zelogik]

The player name is not a top priority at all of course, but a nice to have feature we need (and need implementation in many functions). At first we can/should only display transponder ID. Because nothing is adapted on the server side to set a name/color etc... but should not be too difficult to add in the ID_Data_sorted struct ( RicinoNext-master-sortAlgo.ino)

My top priority is to merge my last version of: RicinoNext-receiver_i2c_robi.ino + RicinoNext-master-sortAlgo.ino + RicinoNext.ino (I just need to find all last working and optimized version...)

• RicinoNext-receiver_i2c_robi.ino -> have good communication with receiver and i2c, must be fully fonctionnal.
• RicinoNext-master-sortAlgo.ino -> very good sort algo (from my POV), good struct/class for player data. (post-processing data), non-blocking implementation, sort of buffer to process only ONE line at a time (to get a pseudo multi-thread loop) it's the most important code after all for a lap counter ;-)
• RicinoNext.ino -> have an almost working WEBUI / JSON communication. (and need to be optimized in many way).

But haven't touched that code since more than 6months, and I have forgot many things, I have made so much version to debug each part one by one... So I will merge RicinoNext-master-sortAlgo.ino with RicinoNext.ino, to get a debug version and hardware-free version. Easier to debug/test without the hardware.

edit: Yes the stopwatch class was a beta/test function ;-) The idea was here but non acceptable and not really working as you have already seen.

[zelogik]

First "official" JSON api draft:

Server -> Client

{
    "config": {
        "lap_total": 10,
        "players_number": 4,
        "gate_number": 1,
        "light_state": 0,
        "light_brightness": 0,
        "players_conf": [{
            "id": "xxxxxxx",
            "name": "player01",
            "color": "blue"
        }, {
            "id": "xxxxxxx",
            "name": "player02",
            "color": "red"
        }, {
            "id": "xxxxxxx",
            "name": "player03",
            "color": "green"
        }, {
            "id": "xxxxxxx",
            "name": "player04",
            "color": "yellow"
        }]
    },
    "race": {
        "state": "STOP",
        "lap": "0",
        "time": "0",
        "message": "xxxxxxxxxxxxxxxx",
        "detected_ID": "XXXX"
    },
    "live": {
        "id": "XXXX",
        "rank": 1,
        "lap": 0,
        "0": [0, 0, 0]
    }
}

Client -> Server

{
     "set_laps": 10, // 1 - ? unlimited ?
     "set_number_player": 4, // 1 - ? 32 max ?
     "set_number_gate": 3, // 1…?8 max?
     "toggle_light": 0, // 0-1
     "set_brightness_light": 0, // 0-255
     "state": "STOP", // START, ?CONNECT?
    "set_player_name": ["ID", "name", "color"], // set player name one by one? Or all at a time…
    "read_ID": 1 // wait for echoing a detected ID from gate
}

Ie: I need to check the Json form validity ;)

[zelogik]

@Peck07 : You can test the first beta for a working hardware-less version. it's in the devel branch -> software/controller/RicinoNext-controller.ino (yes I know... need to do more name cleaning :-D)

Some details that don't work yet:

• "race"->"message", should be ok this evening/next release
• rework on the JSON sent (ie: "conf"), to have an unique JSON function for a broadcast message or unique client a connection. ( just need to merge from my side)
• "live" and "race" JSON message works, but the .js frontend need update (have you already done some work on the frontend?)

The only way tested to start/stop a race is with the serial command (old function with JSON should work but not tested, and API will be changed on the client->server message with the new API) for the moment I have only tested JSON server->client with a simple python script.

[Peck07]

Thanks, I will check it! Regarding the JSON: I would keep the same JSON format (where possible), so no difference in the server->client, client->server JSON, no "set_".

[zelogik]

Have updated the RicinoNext-controller.ino.

The confToJSON work well! But got some troubles getting JSONtoConf working (need to read the "Mastering arduinoJSON" more :-D). The old code works but was not optimized and the technique was not recommended because not efficient from the arduinoJson author himself.

Edit: Yes your right with the same JSON format, it's was just to me for debug (easier in my head :-D), and shorter name give smaller JSON message, so better :-)

you can test every JSON now server->client (with serial interface for start a race and send the conf message)

[Peck07]

Having trouble with Button2,and ESPAsync_WiFiManager :)

[zelogik]

Will remove button2, as it's specific for TT-Go display.

What the trouble with ESPAsync_WiFiManager?

edit: last version have don't have Button2 call. and an almost full working JSON working in two way :-D

edit2: for ESPAsync_WiFiManager, tested with 1.11.0 ( maybe need uncomment ESPAsync_wifiManager.resetSettings(); -> upload -> readd comment -> reupload.)

[Peck07]

ESPAsync_WiFiManager: that's a new library/dependency for me, but managed to add (I am using platformio) Button2: I put that behind compiler switch :)

"have you already done some work on the frontend?" Yes! (on the ricino/webui, but I will try to merge into this project my changes)

[zelogik]

Yes, now that the JSON algo server<->client is almost working ( but not bug-free) i will merge devel repo to master. The ricino/webui is a good start but every JSON message have changed, but good New is that the JSON API is not far from being finalized.

[Peck07]

Ok, the frontend wasn't even working as before, so I've put back some functions, and added the base of the line generation code: https://github.com/Peck07/RicinoNext/commit/5f62dd5070626b09d14e55368230831c029d3992

Also tried to make the ino ESP32 compatible: https://github.com/Peck07/RicinoNext/commit/516c155c8cd6d67f6270f8c9cfd5e45329a5e021

[zelogik]

My bad for the JSON API breaking change, and so on the frontend side too :-/ but it's was necessary. As I have already said, shouldn't be changed as much now :-D, the last API worked but has so many problems/tricks/bug. (and performance was very bad when simulated 4 or 8 players, because message was very big), now only the triggered player/racer/ID is sent on JSON live (and not all at same time). The JSON race, is minimal and sent with variable frequency. And the JSON conf only when the UI change. The last change we could make to optimizing at the extrem is reduce the key with only one letter for example. And I let the "serializeJsonPretty(...)" to have an human readable JSON with the downside to have a bigger JSON message.

And yes I have seen that the onEvent interrupt was totally broken on the backend side... it's in my "urgent" todo list before removing many already seen bug. the last ino was ESP32 compatible but not TT-GO Display compatible (esp32 too), you have added the esp8266 compatibility , nice, need to check available memory later on that architecture :-D. I will manually push the "https://github.com/Peck07/RicinoNext/commit/516c155c8cd6d67f6270f8c9cfd5e45329a5e021" commit with the new cleaned ino, shouldn't take long.(edit: done)

I have quickly read the frontend job you made! It's wonderful :-D, your javascript level is better than me for sure :-D don't hesitate to issue a pull request on the main branch. (folder architecture have changed too... sorry but imo easier to understand now)

Edit: you can look for a example of json message sent from the server Json Server->Client

[Peck07]

1 |ID: 4660 |laps: 5 |Time: 00:29.539 |last: 00:06.975 |best: 00:06.975 |mean: 00:05.907 |1: 00:02.325 |2: 00:01.744 |3: 00:02.906 2 |ID: 4919 |laps: 5 |Time: 00:30.127 |last: 00:08.725 |best: 00:06.289 |mean: 00:06.025 |1: 00:02.325 |2: 00:02.906 |3: 00:03.494 3 |ID: 1638 |laps: 4 |Time: 00:23.727 |last: 00:07.556 |best: 00:07.556 |mean: 00:05.931 |1: 00:03.487 |2: 00:02.325 |3: 00:02.907 4 |ID: 9320 |laps: 4 |Time: 00:24.308 |last: 00:08.137 |best: 00:08.033 |mean: 00:06.077 |1: 00:03.487 |2: 00:03.494 |3: 00:03.488

[BlaineEhrhart]

Any chance for more raw and live data for integration purposes? Basically push events to serial comms as they happen. They are not amazing for clients connecting mid stream, but a combination of what you are doing and events could help improve the "live" aspect of the frontend.

1. Fetch current state for frontend display
2. Listen to new events on the frontend
3. Compute frontend display using events. Doesn't even have to be all the data.

Example of some live events:

Transponder Passes Gate

{
  "type": "transponder_passed_gate",
  "gate": 1,
  "transponder": 4660,
  "time": 223335
}

Race Starts

{
  "type": "race_started",
  "time": 0
}

Race Ends

{
  "type": "race_completed",
  "time": 12000
}

[zelogik]

@Peck07 : amazing :-D, waiting your push request, as I haven't do much change on the frontend yet. edit: you changing the millis time to 00:00.000 format, on the frontend side? I don't know if the output should already be changed on the server side.. (each conversion take less than 80uS on the server side)

@bladesling : don't know if I have understand all your need...

1. Fetch current state for frontend display

   • Every 1sec in waiting state (or 500ms when race running if I remember), the race status is broadcast sent (look at the wiki). : {race:{}}
   • what different with the current {race:{}}, yes, there is a 1 or 2sec of "unknown" state for the client.

2. Listen to new events on the frontend

   • At each new websocket connection, the server send "frontend" status (player number/gate number/name/etc...) : {conf:{}}
   • each client could send a {conf:{}} (except wait race running). Only a stop command could be send at race (maybe need to forbidden if browser fingerprint is not who triggered the start command)

3. Compute frontend display using events. Doesn't even have to be all the data.

   • live status is sent at each triggered gates. {live:{}}, and only one transponder is sent, not what you look for?
   • at each modification from one UI, the new {conf:{}} is broadcasted.

Maybe you asking for "serial" interface ? for the moment void WriteSerialLive(..) output and ReadSerial() is "only" to try/debug. I need to implement a SAMD21 <-> esp01 connection with serial (or i2c), and data will be the same as the JSON sent/receive from websocket. So remove the esp01, and add a computer instead, should work. But it's not in my priority list for the moment, sorry ;-) Add a Zround protocol (so Serial) but remove the availability if gates > 1;, same not more in my priority but should "easily" implementable too.

ie: look on the draft wiki :-D

[Peck07]

The original code had the conversion, just had to refactor it a little bit, but I didn't check the performance :) shouldn't be a problem, also the code is quite flexible, removing the conversion is pretty easy, so I'd say let's keep it that way right now, and change it when we feel it's necessary :)

[zelogik]

Yes it's just optimization... timeToChar(..) use many / and % the worst thing for uC so could be less than 80uSec. (bitwise to our help, but losing some precision) And as i start to thinking about implementing dynamic allocation of the JSON message size. length of each message will be important (millis(ie: 16.5min = 999999us) vs 00:00.000'\0'), so 6 vs 10 bytes... but yes it's not the priority ;-)

[BlaineEhrhart]

Apologies- I didn't understand the frequency of the live updates. Carry on lol.

[zelogik]

@bladesling : live update is sent every time a gate is triggered :-D, only one player at a time

[Peck07]

Actually I've put back the "Current" column, so a timer is running on the frontend side as well, pretty accurate, and it's more "live"

[zelogik]

Actually I've put back the "Current" column, so a timer is running on the frontend side as well, pretty accurate, and it's more "live" :+1:

[zelogik]

7 , wiki draft, ...