code generator issue with external schema?

[pljakobs]

I figured I'd put all my re-used definitions in one devs.cfgdb schema module (even though, for reasons, this one, hsvct, ended up not really being re-used)

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "store":"false",
  "properties":{
  },
  "$defs":{
    "raw": {
      "type": "object",
      "properties": {
        "ww": {
            "type": "integer",
            "minimum": 0,
            "maximum": 1023,
            "alias":"warmwhite"
        },
        "r": {
            "type": "integer",
            "minimum": 0,
            "maximum": 1023,
            "alias":"red"
        },
        "b": {
            "type": "integer",
            "minimum": 0,
            "maximum": 1023,
            "alias":"blue"
        },
        "cw": {
            "type": "integer",
            "minimum": 0,
            "maximum": 1023,
            "alias":"coldwhite"
        },
        "g": {
            "type": "integer",
            "minimum": 0,
            "maximum": 1023,
            "alias":"green"
        }
      }
    },
    "hsv": {
      "type": "object",
      "properties": {
        "s": {
            "type": "number",
            "minimum": 0,
            "maximum": 100,
            "alias":"sat"
        },
        "v": {
            "type": "number",
            "minimum": 0,
            "maximum": 100,
            "alias":"val"
        },
        "h": {
            "type": "number",
            "minimum": 0,
            "maximum": 359,
            "alias":"hue"
        }
      }
    },
    "hsvct": {
      "type": "object",
      "properties": {
        "s": {
            "type": "number",
            "minimum": 0,
            "maximum": 100,
            "alias":"sat"
        },
        "v": {
            "type": "number",
            "minimum": 0,
            "maximum": 100,
            "alias":"val"
        },
        "h": {
            "type": "number",
            "minimum": 0,
            "maximum": 359,
            "alias":"hue"
        },
        "ct":{
          "type":"number",
          "maximum":10000,
          "minimum":0,
          "alias":"colortemp"
        }
      }
    },
    "color":{
      "oneOf":
        [
          {"$ref":"#/$defs/hsv"},
          {"$ref":"#/$defs/hsvct"},
          {"$ref":"#/$defs/raw"}
      ]
    },
    "channel": {
      "type":"object",
      "properties":{
        "pin":{
            "type":"integer",
            "minimum":0,
            "maximum":255
        },
        "name":{
            "type":"string"
        }
      }
    }   
  }
}

I then use those in my actual database schemas (I'm still thinking to have two databases, one for the app configuration, the other for working data)

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "store":"json",
  "properties": {
    "last-color":{
      "$ref":"defs/$defs/hsvct"
    },
    "presets": {
      "type": "array",
      "store":"json",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string"
          },
          "color": {
            "$ref":"defs/$defs/color"
          },
          "ts": {
            "type": "integer"
          },
          "id":{
            "type":"integer"
          },
          "favorite":{
            "type":"boolean",
            "default":false
          }
        }
      }
    },
    "controllers": {
      "type": "array",
      "store":"json",
      "items": {
        "type": "object",
        "properties": {
          "id":{
            "type":"integer"
          },
          "ip": {
            "type": "string"
          },
          "name": {
            "type": "string"
          },
          "last-seen": {
            "type": "integer"
          }
        }
      }
    },
    "scenes":{
      "type":"array",
      "store":"json",
      "items":{
        "type":"object",
        "properties":{
          "name":{
            "type":"string"
          },
          "settings":{
            "type":"array",
            "items":{
              "type":"object",
              "properties":{
                "controller_id":{
                  "type":"integer"
                },
                "color": {
                  "$ref":"defs/$defs/color"
                }
              }
            }
          }
        }
      }
    }
  }
}

so here, I create a root property last-color that uses the hsvct definition. It took me a while (and your answer yesterday) to understand that, even if only used to store defs, the schema file requires a properties section, so I've added an empty one - this file only holds the definitions that I use on both databases.

compiling this gives me quite a few errors in ConfigDB even though the code generator does not report an error:

In file included from /home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:23,
                 from /home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.cpp:7:
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/defs.h:216:33: error: invalid use of incomplete type 'class Defs::ContainedHsvct'
  216 |                 ContainedHsvct::Struct hsvct;
      |                                 ^~~~~~
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/defs.h:17:11: note: forward declaration of 'class Defs::ContainedHsvct'
   17 |     class ContainedHsvct;
      |           ^~~~~~~~~~~~~~
In file included from /home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.cpp:7:
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:28:37: error: invalid use of incomplete type 'class AppData'
   28 |     using ContainedColor = AppData::ContainedColor;
      |                                     ^~~~~~~~~~~~~~
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:25:7: note: definition of 'class AppData' is not complete until the closing brace
   25 | class AppData: public ConfigDB::DatabaseTemplate<AppData>
      |       ^~~~~~~
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:29:35: error: invalid use of incomplete type 'class AppData'
   29 |     using ColorUpdater = AppData::ColorUpdater;
      |                                   ^~~~~~~~~~~~
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:25:7: note: definition of 'class AppData' is not complete until the closing brace
   25 | class AppData: public ConfigDB::DatabaseTemplate<AppData>
      |       ^~~~~~~
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:48:17: error: 'ContainedHsvct' does not name a type; did you mean 'ContainedRoot'?
   48 |                 ContainedHsvct::Struct lastColor{};
      |                 ^~~~~~~~~~~~~~
      |                 ContainedRoot
In file included from /home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.cpp:7:
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:60:15: error: 'ContainedHsvct' does not name a type; did you mean 'ContainedRoot'?
   60 |         const ContainedHsvct lastColor;
      |               ^~~~~~~~~~~~~~
      |               ContainedRoot
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:73:9: error: 'HsvctUpdater' does not name a type; did you mean 'RootUpdater'?
   73 |         HsvctUpdater lastColor;
      |         ^~~~~~~~~~~~
      |         RootUpdater
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:87:17: error: 'ContainedColor' does not name a type; did you mean 'ContainedRoot'?
   87 |                 ContainedColor::Struct color{};
      |                 ^~~~~~~~~~~~~~
      |                 ContainedRoot
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:123:15: error: 'ContainedColor' does not name a type; did you mean 'ContainedRoot'?
  123 |         const ContainedColor color;
      |               ^~~~~~~~~~~~~~
      |               ContainedRoot
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:177:9: error: 'ColorUpdater' does not name a type; did you mean 'RootUpdater'?
  177 |         ColorUpdater color;
      |         ^~~~~~~~~~~~
      |         RootUpdater
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:316:21: error: 'ContainedColor' does not name a type; did you mean 'ContainedRoot'?
  316 |                     ContainedColor::Struct color{};
      |                     ^~~~~~~~~~~~~~
      |                     ContainedRoot
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:334:19: error: 'ContainedColor' does not name a type; did you mean 'ContainedRoot'?
  334 |             const ContainedColor color;
      |                   ^~~~~~~~~~~~~~
      |                   ContainedRoot
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:358:13: error: 'ColorUpdater' does not name a type; did you mean 'RootUpdater'?
  358 |             ColorUpdater color;
      |             ^~~~~~~~~~~~
      |             RootUpdater
/home/pjakobs/devel/esp_rgbww_firmware/out/ConfigDB/app-data.h:443:57: error: 'ContainedHsvct' was not declared in this scope; did you mean 'ContainedRoot'?
  443 |         using LastColor = ConfigDB::OuterObjectTemplate<ContainedHsvct, HsvctUpdater, AppData, 0, ContainedRoot, 0, 0>;
      |                                                         ^~~~~~~~~~~~~~
      |                                                         ContainedRoot

is this a fault on my side or with the generator?

[mikee47]

It's the code generator. I've got a work-in-progress to address some of these issues, these are the main ones:

• Make properties optional for definition-only schema
• Sort out paths/namespaces for re-used definitions. Stuff under $defs gets generated as a direct parent of its containing database, but this doesn't always get tracked correctly.
• Try to sort out declaration order by dependency. Errors like incomplete type with ContainedHsvct::Struct arise because the object definition order isn't correct.

I'll use your provided schema for testing, hopefully get this fixed.

[pljakobs]

would an explicit "include" for cfgdb files make this easier because it would create a defined order of generation?

[mikee47]

would an explicit "include" for cfgdb files make this easier because it would create a defined order of generation?

Short answer

No, the order of parsing doesn't matter. It's the order in which the generated C++ classes are emitted which is the issue.

Long answer

The code generator loads all the schemas as a first step, so as long as they're all in the same project then there's no need to explicitly provide a path. If you wanted to start using schemas from other locations, perhaps even shared between projects, then that can be accommodated by invoking the generator manually.

After loading schemas, the generator parses the properties value from each database in turn. Stuff from other locations in the schema (typically $defs) only gets parsed when referenced. Note: This means unused definitions won't appear in .h or .cpp files.

As each object is parsed they are added to an internal list which is then iterated in reverse order during code generation. That generally works OK because dependent objects appear after the class which first uses them. However, with your example above, last-color refers directly to the hsvct so that appears first in the list. That means it gets emitted after everything else. However, it's also a dependent of color, which gets emitted earlier. So the generator now actually has to do some work to establish the correct order of definition rather than just winging it.

[pljakobs]

those changes mostly work, but since I import both app-data and app-config, defs gets iported twice, adding #pragma once to the defs.h works.