Generated schema JSON file cannot be loaded

[debackerl]

Hello,

I'm running the latest version of the script: "v3.11-0.22.40".

I issued a "mosquitto_pub -t 'evohome/_zone_independent/command' -m '{"sys_config":"SAVE_SCHEMA"}' -q 1" to generate the schema.json and devices.json files, because upon closing the script, nothing was saved.

Now, if I try to restart it, I get:

Nov 13 18:57:38 assistant systemd[1]: Started "Service for evogateway".
Nov 13 18:57:38 assistant python3[3962965]: Traceback (most recent call last):
Nov 13 18:57:38 assistant python3[3962965]:   File "/home/evogateway/lib/evogateway.py", line 1248, in <module>
Nov 13 18:57:38 assistant python3[3962965]:     asyncio.run(main())
Nov 13 18:57:38 assistant python3[3962965]:   File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
Nov 13 18:57:38 assistant python3[3962965]:     return loop.run_until_complete(main)
Nov 13 18:57:38 assistant python3[3962965]:   File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
Nov 13 18:57:38 assistant python3[3962965]:     return future.result()
Nov 13 18:57:38 assistant python3[3962965]:   File "/home/evogateway/lib/evogateway.py", line 1224, in main
Nov 13 18:57:38 assistant python3[3962965]:     GWY = Gateway(serial_port, **lib_kwargs)
Nov 13 18:57:38 assistant python3[3962965]:   File "/opt/venv/evogateway/lib/python3.10/site-packages/ramses_rf/gateway.py", line 355, in __init__
Nov 13 18:57:38 assistant python3[3962965]:     **SCH_GLOBAL_CONFIG({k: v for k, v in kwargs.items() if k[:1] != "_"}),
Nov 13 18:57:38 assistant python3[3962965]:   File "/opt/venv/evogateway/lib/python3.10/site-packages/voluptuous/validators.py", line 232, in __call__
Nov 13 18:57:38 assistant python3[3962965]:     return self._exec((Schema(val) for val in self.validators), v)
Nov 13 18:57:38 assistant python3[3962965]:   File "/opt/venv/evogateway/lib/python3.10/site-packages/voluptuous/validators.py", line 355, in _exec
Nov 13 18:57:38 assistant python3[3962965]:     raise e if self.msg is None else AllInvalid(self.msg, path=path)
Nov 13 18:57:38 assistant python3[3962965]:   File "/opt/venv/evogateway/lib/python3.10/site-packages/voluptuous/validators.py", line 351, in _exec
Nov 13 18:57:38 assistant python3[3962965]:     v = func(v)
Nov 13 18:57:38 assistant python3[3962965]:   File "/opt/venv/evogateway/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 272, in __call__
Nov 13 18:57:38 assistant python3[3962965]:     return self._compiled([], data)
Nov 13 18:57:38 assistant python3[3962965]:   File "/opt/venv/evogateway/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 818, in validate_callable
Nov 13 18:57:38 assistant python3[3962965]:     return schema(data)
Nov 13 18:57:38 assistant python3[3962965]:   File "/opt/venv/evogateway/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 272, in __call__
Nov 13 18:57:38 assistant python3[3962965]:     return self._compiled([], data)
Nov 13 18:57:38 assistant python3[3962965]:   File "/opt/venv/evogateway/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 595, in validate_dict
Nov 13 18:57:38 assistant python3[3962965]:     return base_validate(path, iteritems(data), out)
Nov 13 18:57:38 assistant python3[3962965]:   File "/opt/venv/evogateway/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 433, in validate_mapping
Nov 13 18:57:38 assistant python3[3962965]:     raise er.MultipleInvalid(errors)
Nov 13 18:57:38 assistant python3[3962965]: voluptuous.error.MultipleInvalid: extra keys not allowed @ data['schema']

My config:

[Serial Port]
COM_PORT         = /dev/hgi80
COM_BAUD         = 115200

[Files]
EVENTS_FILE                 = /home/evogateway/data/events.log
PACKET_LOG_FILE             = /home/evogateway/data/packets.log
LOG_FILE_ROTATE_COUNT       = 4
LOG_FILE_ROTATE_BYTES       = 1048576

DEVICES_FILE                = /home/evogateway/data/devices.json
ZONES_FILE                  = /home/evogateway/data/zones.json
LOAD_ZONES_FROM_FILE        = True
SCHEMA_FILE                 = /home/evogateway/data/schema.json

[MQTT]
MQTT_SERVER                 = 127.0.0.1
#MQTT_USER                   =
#MQTT_PW                     =

MQTT_PUB_TOPIC              = evohome
MQTT_SUB_TOPIC              = evohome/_zone_independent/command

#Publish as a json string. If False, the key/values of the json will be published individually
MQTT_PUB_AS_JSON            = False

# Either group messages by zone name (default), otherwise by device name
MQTT_GROUP_BY_ZONE          = False

[MISC]
THIS_GATEWAY_NAME           = evoGateway

# Display full json string from evohome_rf, or just key data item for each row
DISPLAY_FULL_JSON            = False

# If no device can be loaded from the schema file, then eavesdropping will be auto-enabled
SCHEMA_EAVESDROP            = False

# Assumes that there is only a single HGI device on the network (in case of spurious HGI device addresses)
FORCE_SINGLE_HGI            = True

DISABLE_SENDING             = False

The generated schema:

{
    "schema": {
        "system": {
            "appliance_control": "13:093345"
        },
        "orphans": [],
        "stored_hotwater": {},
        "underfloor_heating": {},
        "zones": {
            "00": {
                "_name": "Living room",
                "class": "radiator_valve",
                "sensor": "01:165018",
                "actuators": [
                    "04:060219",
                    "04:060255",
                    "04:060259",
                    "04:063251"
                ]
            },
            "01": {
                "_name": "Bath room",
                "class": "radiator_valve",
                "sensor": "04:072215",
                "actuators": [
                    "04:072215"
                ]
            },
            "02": {
                "_name": "Kitchen",
                "class": "radiator_valve",
                "sensor": "04:063237",
                "actuators": [
                    "04:063237"
                ]
            },
            "03": {
                "_name": "Bedroom M",
                "class": "radiator_valve",
                "sensor": "04:063243",
                "actuators": [
                    "04:063243"
                ]
            },
            "04": {
                "_name": "Toilet 1",
                "class": "radiator_valve",
                "sensor": "04:132321",
                "actuators": [
                    "04:132321"
                ]
            },
            "05": {
                "_name": "Basement",
                "class": "radiator_valve",
                "sensor": "04:060231",
                "actuators": [
                    "04:060231",
                    "04:060267"
                ]
            },
            "06": {
                "_name": "Laundry",
                "class": "radiator_valve",
                "sensor": "04:060261",
                "actuators": [
                    "04:060261"
                ]
            },
            "07": {
                "_name": "Bedroom G",
                "class": "radiator_valve",
                "sensor": "04:060229",
                "actuators": [
                    "04:060229"
                ]
            },
            "08": {
                "_name": "Bedroom E",
                "class": "radiator_valve",
                "sensor": "04:132319",
                "actuators": [
                    "04:132319"
                ]
            },
            "09": {
                "_name": "Upstairs",
                "class": "radiator_valve",
                "sensor": "04:060257",
                "actuators": [
                    "04:060257"
                ]
            },
            "0A": {
                "_name": "Hall Floor",
                "class": "underfloor_heating",
                "sensor": "34:038865",
                "actuators": []
            },
            "0B": {
                "_name": "Kitchen Floor",
                "class": "underfloor_heating",
                "sensor": "34:038877",
                "actuators": []
            }
        }
    }
}

When I looked at MQTT message, I saw that some schema was published as {"orphans": [], "stored_hotwater": {}, ... but trying to load that I got extra keys not allowed @ data['orphans']

Any idea?

[smar000]

Nov 13 18:57:38 assistant python3[3962965]: voluptuous.error.MultipleInvalid: extra keys not allowed @ data['schema']

Unfortunately I do not have much time at the moment to look at this fully, but from the error message, it seems as if it does not like the "orphans": [], . Try deleting this and see if it helps?

[debackerl]

Thanks @smar000 , I will try to understand your code when I have more time myself :-) Removing orphans didn't work either.

What I found interesting is that the schema saved by evoGateway is completely different than the one from the other bug ticket: https://github.com/smar000/evoGateway/issues/60#issuecomment-1718076896

In his case, he has:

    "main_tcs": "01:187689",
    "01:187689": {

while I have

    "schema": {

Error messages from voluptuous are terrible :-)

[smar000]

What I found interesting is that the schema saved by evoGateway is completely different than the one from the other bug ticket: #60 (comment)

The schema you posted in your earlier post is not the full schema but just a list of the devices. Can you try modifying it, copying the format of the other one you linked to, and see if that works. If not, I'll try to spend some time over the weekend and have a proper look.

[chris-hunter]

I had this problem, this fixed it for me:

[chris-hunter]

Putting disable_discovery = False in my config file didn't work. However, changing line 157 in evogateway.py RAMSESRF_DISABLE_DISCOVERY so fallback=True did work, so it is now running using my devices.json and ramses_rf_schema.json files. If I delete these files it returns to eavesdrop mode and writes new devices and ramses_rf_schema files as it should.

I did come across a couple of other issues in the process: When evogateway.py runs in eavesdrop mode and writes its own ramses_rf_schema.json file, it includes a "packet.log" section. This creates an error which is resolved if you delete this section from the ramses_rf_schema.json file

Also, if the ramses_rf_schema.json file contains "appliance_control": null, this creates an error - it seems to need a value. I have an S-plan system so my boiler is turned on by my DHW relay or heating relay, so my "appliance_control" is null. This doesn't create an error if it is null in ramses_rf's equivalent config.json file.

Finally, in evogateway.py line 129 SCHEMA_FILE fallback=ramsesrf_schema.json - I think it should be rames_rf_schema.json

Thanks for all your help - great program.

[chris-hunter]

It's in a closed issue but I'm not sure how to link to it

[smar000]

Thanks @chris-hunter . Very helpful and I will look to incorporate the above next time I review the code.