Allow JSON for API - different formats for dashboard, MQTT and API

[tp1de]

The data content for holiday-modes and/or switch programs can be quite complex and can contain quite an number of datapoints and parameters. E.g. switch program for EMS+ (RC310) might contain up to 6 switch points per day - a maximum of 42 switch points per week. Each switch point is associated with a temp parameter (eco / comfort / temp).

The RC310 has 2 switch programs for each hc and one for ww and one for circulation pump. For my quite standard heating system with 2 hc's and ww in total 6 switch programs with up to 42 switch points. A bit similar but less complex are the holiday modes hm1 to hm5.

I believe it makes sense to add JSON for the API-interface to cover this complex data structures within one JSON-structure similar to the km200 web-API interface. In this case it does not make sense to include this into the dashboard and as far as I understood HA MQTT can't process this either.

Let me recommend to differentiate between dashboard / HA-MQTT and API in this case. Either different data-content / data-structure or just API-content not shown in dashboard.

[proddy]

I'd prefer to create a special feature from the web where you can upload a JSON file for programs, rather than changing the API to support this. And the API already takes JSON as input, just not in the old km200 format

[tp1de]

I'd prefer to create a special feature from the web where you can upload a JSON file for programs, rather than changing the API to support this

I do not understands what this means? My proposal was to register the telegram and then not creating individual variables but rather one JSON-structure for the whole telegram content. E.g. switchProgram:

[ { "dayOfWeek": "Mo", "setpoint": "comfort2", "time": 300}, { "dayOfWeek": "Mo", "setpoint": "eco", "time": 1320}, { "dayOfWeek": "Tu", "setpoint": "comfort2", "time": 300 }, ...... ]

I do have the coding to extract this out the respective telegrams identical to km200 JSON-structure or any other JSON you like.

Within API-call read/write it should show as: ems-esp/api/thermostat:

_{ ..... "hc1": { "seltemp": 22, "currtemp": 22.8, "mode": 1, "switchProgram_A": [ { "dayOfWeek": "Mo", "setpoint": "comfort2", "time": 300}, { "dayOfWeek": "Mo", "setpoint": "eco", "time": 1320}, { "dayOfWeek": "Tu", "setpoint": "comfort2", "time": 300 }, ...... ], "switchProgramB": [ { "dayOfWeek": "Mo", "setpoint": "comfort2", "time": 300}, { "dayOfWeek": "Mo", "setpoint": "eco", "time": 1320}, { "dayOfWeek": "Tu", "setpoint": "comfort2", "time": 300 }, ...... ]

... and switchProgram_A and switchProgram_B should be writable.

Within the actual ems-esp coding logic I need to create Individual variables as far as I understood. Within the existing Change Value dialog box this does not work either.

[MichaelDvP]

This full json switch program is more than 5kB per circuit, > 30kB for all circuits. We have to hold this in memory and it has to fit in the json buffers for api, mqtt and web. No way in esp32 ram.

We have already discussed that in #461 and RC35/RC30 is implemented as single switchpoint read/write.

The api uses json commands, but only the "value" is passed to the command functions. We can extend to json commands with: Command::process():

    } else if (data.is<JsonObject>()) {
        char jsondata[measureJson(data) + 1];
        serializeJson(data, jsondata, sizeof(jsondata));
        return_code = Command::call(device_type, command_p, jsondata, is_admin, id_n, output);

and use commands like {"value":{"val1":1,"val2":2}} and the command function can deserialize again.

To save memory and read the whole program we can only store a copy of the telegram (84byte per program) uint8_t switchtimeA[84]; and memcpy(&hc->switchtimeA[telegram->offset], telegram->message_data, telegram->message_length); The buffers for web/api-info/mqtt are to small, but the single value info is ok. So make a json only on the output of api value info ems-esp.local/api/thermostat/hc2/switchtimeA For RC35 this is for testing: (the json creation should go to thermostat class to use DeviceValueType::JSON in all classes.

            case DeviceValueType::JSON:
                json[type] = F("json");
                JsonObject ja = json.createNestedObject(value);
                for (uint8_t i = 0; i < 42; i++) {
                    uint8_t * v_p = (uint8_t *)dv.value_p + 2 * i;
                    char n[3];
                    snprintf(n, 3, "%02d", i);
                    if (((*v_p) >> 5) != 7) {
                        JsonObject d = ja.createNestedObject(n);
                        d["day"]     = F_(enum_dayOfWeek[(*v_p) >> 5]);
                        char time[6];
                        snprintf(time, 6, "%02d:%02d",(*(v_p + 1) / 6), 10 * (*(v_p + 1) % 6));
                        d["time"]    = time;
                        d["mode"]    = (*(v_p) & 1) ? "on" : "off";
                    }
                }

Output is like this, one try with jsonArray, one with nestedObject. output_switchtimeA.txt

But i'm not really happy with this solution, needs more thinking.

[tp1de]

Yes a jsonArray is similar to km200 API response. Just the numbering is not needed. To create a shorter JSON-structures "day" could be "d", "time" "t" and "mode" "m" and mode content 0/1 (on /off) or e/c (eco/comfort) in my case. In RC310 there is a possibility to go for fixed temps per switch point. Then content would be 2 digit int.

Nevertheless I could expand the raw telegram in my ioBroker Adapter - coding is already in by reading syslog hex telegrams. You need to decide if a short jsonArray will be acceptable or not . In most cases it should be between 500 bytes to 1kB for each switch program. E.g.: [ {"d": "mo", "t": "06:20", "m": "1"}, {"d": "mo", "t":"21:30", "m": "0"}, {"d": "tu", "t": "06:20", "m": "1"}, ....... ],

Just tell if I should implement raw telegram version or a short jsonArray ?

What about my multiline version of holiday modes? For me the implemented version works fine. Or do you prefer a jsonArray for each holiday mode?

[MichaelDvP]

Just the numbering is not needed.

I have RC35 and need the numbering. RC300 is a bit different with 6 points per day, but i think numbering is only obsolet if all 6 points are written at once.

The shortend json is also much to big for normal publishing. Also the command buffer (256 byte) can't hold a complete new setting. I suggest to use the same way as implemented for RC30/35 in #461 and set/read the setpoints one by one.

What about my multiline version of holiday modes?

As written it seems not logical to me, but i don't have RC300 and can't check and it's proddys project, he decides.

[proddy]

yes, it's too complicated and will blow up the ESP32 this way. As I mentioed in an earlier comment https://github.com/emsesp/EMS-ESP32/issues/564#issuecomment-1167847703 for specific use cases like this the best forward is to create a new feature with a file import via the web UI.