yaml validation

[udeved]

https://calamares.io/bugs/browse/CAL-390

Continuing here...

pykwalify -d settings.yaml -s settings-schema.yaml
INFO - validation.valid

Schema exists:

• [x] settings.conf
• [x] bootloader.conf
• [x] grubcfg.conf
• [x] luksopenswaphookcfg.conf
• [x] packages.conf
• [x] servicescfg.conf
• [x] welcome.conf
• [x] chrootcfg.conf
• [x] initcpio.conf
• [x] machineid.conf
• [x] partition.conf
• [x] services.conf
• [x] displaymanager.conf
• [x] keyboard.conf
• [x] mhwdcfg.conf
• [x] plymouthcfg.conf
• [x] umount.conf
• [x] finished.conf
• [x] license.conf
• [x] mount.conf
• [x] postcfg.conf
• [x] unpackfs.conf
• [x] fstab.conf
• [x] locale.conf
• [x] netinstall.conf
• [x] removeuser.conf
• [x] users.conf

https://github.com/manjaro/manjaro-tools/blob/master/data/schemas

@teo

If you have use for the schemas let me know.

ToDo:

• fix pykwalify package, something is wrong with python-yaml and python-dateutil version dep
• conf files despite yaml are not accepted by pykwalify, they must have yaml extension

[udeved]

I used the validation in check-yaml

$ check-yaml -cv
 --> Loading Packages: [Packages-Root] ...
  -> Writing Packages-Root-x86_64-openrc.yaml ...
  -> Checking validity [Packages-Root-x86_64-openrc.yaml] ...
 INFO - validation.valid
 --> Loading Packages: [Packages-Lxqt] ...
  -> Writing Packages-Lxqt-x86_64-openrc.yaml ...
  -> Checking validity [Packages-Lxqt-x86_64-openrc.yaml] ...
 INFO - validation.valid
 --> Configuring [Calamares]
  -> Writing settings.conf ...
  -> Writing locale.conf ...
  -> Writing welcome.conf ...
  -> Writing netinstall.conf ...
  -> Writing packages.conf ...
  -> Writing bootloader.conf ...
  -> Writing mhwdcfg.conf ...
  -> Writing unpackfs.conf ...
  -> Writing displaymanager.conf ...
  -> Writing initcpio.conf ...
  -> Writing machineid.conf ...
  -> Writing finished.conf ...
  -> Writing plymouthcfg.conf ...
  -> Writing postcfg.conf ...
  -> Writing servicescfg.conf ...
  -> Writing users.conf ...
 --> Done configuring [Calamares]
  -> Checking validity [bootloader.conf] ...
 INFO - validation.valid
  -> Checking validity [displaymanager.conf] ...
 INFO - validation.valid
  -> Checking validity [finished.conf] ...
 INFO - validation.valid
  -> Checking validity [initcpio.conf] ...
 INFO - validation.valid
  -> Checking validity [locale.conf] ...
 INFO - validation.valid
  -> Checking validity [machineid.conf] ...
 INFO - validation.valid
  -> Checking validity [mhwdcfg.conf] ...
 INFO - validation.valid
  -> Checking validity [netinstall.conf] ...
 INFO - validation.valid
  -> Checking validity [packages.conf] ...
 INFO - validation.valid
  -> Checking validity [plymouthcfg.conf] ...
 INFO - validation.valid
  -> Checking validity [postcfg.conf] ...
 INFO - validation.valid
  -> Checking validity [servicescfg.conf] ...
 INFO - validation.valid
  -> Checking validity [unpackfs.conf] ...
 INFO - validation.valid
  -> Checking validity [users.conf] ...
 INFO - validation.valid
  -> Checking validity [welcome.conf] ...
 INFO - validation.valid
  -> Checking validity [settings.conf] ...
 INFO - validation.valid

[philmmanjaro]

Looks fine. Only have to ckeck it also myself. Great work @udeved

[udeved]

I switched yesterday to ruby-kwalify, it seem pykwalify has some bugs with schema validation and version depends, plus it won't accept *.conf files.

$ sh test.sh
Validating data bootloader.conf ...
/usr/share/calamares/modules/bootloader.conf#0: valid.
Validating schema bootloader.schema.yaml ...
/usr/share/manjaro-tools/schemas/bootloader.schema.yaml#0: valid.
Validating data chrootcfg.conf ...
/usr/share/calamares/modules/chrootcfg.conf#0: valid.
Validating schema chrootcfg.schema.yaml ...
/usr/share/manjaro-tools/schemas/chrootcfg.schema.yaml#0: valid.
Validating data displaymanager.conf ...
/usr/share/calamares/modules/displaymanager.conf#0: valid.
Validating schema displaymanager.schema.yaml ...
/usr/share/manjaro-tools/schemas/displaymanager.schema.yaml#0: valid.
Validating data finished.conf ...
/usr/share/calamares/modules/finished.conf#0: valid.
Validating schema finished.schema.yaml ...
/usr/share/manjaro-tools/schemas/finished.schema.yaml#0: valid.
Validating data fstab.conf ...
/usr/share/calamares/modules/fstab.conf#0: valid.
Validating schema fstab.schema.yaml ...
/usr/share/manjaro-tools/schemas/fstab.schema.yaml#0: valid.
Validating data grubcfg.conf ...
/usr/share/calamares/modules/grubcfg.conf#0: valid.
Validating schema grubcfg.schema.yaml ...
/usr/share/manjaro-tools/schemas/grubcfg.schema.yaml#0: valid.
Validating data initcpio.conf ...
/usr/share/calamares/modules/initcpio.conf#0: valid.
Validating schema initcpio.schema.yaml ...
/usr/share/manjaro-tools/schemas/initcpio.schema.yaml#0: valid.
Validating data keyboard.conf ...
/usr/share/calamares/modules/keyboard.conf#0: valid.
Validating schema keyboard.schema.yaml ...
/usr/share/manjaro-tools/schemas/keyboard.schema.yaml#0: valid.
Validating data license.conf ...
/usr/share/calamares/modules/license.conf#0: valid.
Validating schema license.schema.yaml ...
/usr/share/manjaro-tools/schemas/license.schema.yaml#0: valid.
Validating data locale.conf ...
/usr/share/calamares/modules/locale.conf#0: valid.
Validating schema locale.schema.yaml ...
/usr/share/manjaro-tools/schemas/locale.schema.yaml#0: valid.
Validating data luksopenswaphookcfg.conf ...
/usr/share/calamares/modules/luksopenswaphookcfg.conf#0: valid.
Validating schema luksopenswaphookcfg.schema.yaml ...
/usr/share/manjaro-tools/schemas/luksopenswaphookcfg.schema.yaml#0: valid.
Validating data machineid.conf ...
/usr/share/calamares/modules/machineid.conf#0: valid.
Validating schema machineid.schema.yaml ...
/usr/share/manjaro-tools/schemas/machineid.schema.yaml#0: valid.
Validating data mhwdcfg.conf ...
/usr/share/calamares/modules/mhwdcfg.conf#0: valid.
Validating schema mhwdcfg.schema.yaml ...
/usr/share/manjaro-tools/schemas/mhwdcfg.schema.yaml#0: valid.
Validating data mount.conf ...
/usr/share/calamares/modules/mount.conf#0: valid.
Validating schema mount.schema.yaml ...
/usr/share/manjaro-tools/schemas/mount.schema.yaml#0: valid.
Validating data netinstall.conf ...
/usr/share/calamares/modules/netinstall.conf#0: valid.
Validating schema netinstall.schema.yaml ...
/usr/share/manjaro-tools/schemas/netinstall.schema.yaml#0: valid.
Validating data packages.conf ...
/usr/share/calamares/modules/packages.conf#0: valid.
Validating schema packages.schema.yaml ...
/usr/share/manjaro-tools/schemas/packages.schema.yaml#0: valid.
Validating data partition.conf ...
/usr/share/calamares/modules/partition.conf#0: valid.
Validating schema partition.schema.yaml ...
/usr/share/manjaro-tools/schemas/partition.schema.yaml#0: valid.
Validating data plymouthcfg.conf ...
/usr/share/calamares/modules/plymouthcfg.conf#0: empty.
Validating schema plymouthcfg.schema.yaml ...
/usr/share/manjaro-tools/schemas/plymouthcfg.schema.yaml#0: valid.
Validating data postcfg.conf ...
/usr/share/calamares/modules/postcfg.conf#0: valid.
Validating schema postcfg.schema.yaml ...
/usr/share/manjaro-tools/schemas/postcfg.schema.yaml#0: valid.
Validating data removeuser.conf ...
/usr/share/calamares/modules/removeuser.conf#0: valid.
Validating schema removeuser.schema.yaml ...
/usr/share/manjaro-tools/schemas/removeuser.schema.yaml#0: valid.
Validating data servicescfg.conf ...
/usr/share/calamares/modules/servicescfg.conf#0: valid.
Validating schema servicescfg.schema.yaml ...
/usr/share/manjaro-tools/schemas/servicescfg.schema.yaml#0: valid.
Validating data services.conf ...
/usr/share/calamares/modules/services.conf#0: valid.
Validating schema services.schema.yaml ...
/usr/share/manjaro-tools/schemas/services.schema.yaml#0: valid.
Validating data settings.conf ...
/usr/share/calamares/settings.conf#0: valid.
Validating schema settings.schema.yaml ...
/usr/share/manjaro-tools/schemas/settings.schema.yaml#0: valid.
Validating data umount.conf ...
/usr/share/calamares/modules/umount.conf#0: empty.
Validating schema umount.schema.yaml ...
/usr/share/manjaro-tools/schemas/umount.schema.yaml#0: valid.
Validating data unpackfs.conf ...
/usr/share/calamares/modules/unpackfs.conf#0: valid.
Validating schema unpackfs.schema.yaml ...
/usr/share/manjaro-tools/schemas/unpackfs.schema.yaml#0: valid.
Validating data users.conf ...
/usr/share/calamares/modules/users.conf#0: valid.
Validating schema users.schema.yaml ...
/usr/share/manjaro-tools/schemas/users.schema.yaml#0: valid.
Validating data welcome.conf ...
/usr/share/calamares/modules/welcome.conf#0: valid.
Validating schema welcome.schema.yaml ...
/usr/share/manjaro-tools/schemas/welcome.schema.yaml#0: valid.

#!/bin/sh

for f in /usr/share/manjaro-tools/schemas/*.yaml;do
    sp=/usr/share/calamares/modules
    n=${f##*/}
    n=${n%*.schema.yaml}
    case $n in
        netgroups) continue ;;
        settings) sp=/usr/share/calamares ;;
    esac
    echo "Validating data $n.conf ..."
    kwalify -lf $f $sp/$n.conf
    echo "Validating schema ${f##*/} ..."
    kwalify -lm $f
done

[udeved]

@teo @philmmanjaro

I think it would be useful to commit the schemas in the calamares source tree? However, it would be nice if the schemas and conf files were packaged separately, so eg manjaro-tools can use them for validation too without pulling full calamares package.

[philmmanjaro]

Packaging will be done always by the distribution. We are already thinking about split packages for the brandings. So why not also split the configurations. It only has to make sense after all. Let me think about it.

[udeved]

I have looked a bit through the settings code, my rusty c++ asks the question, if it is possibly in c++ to serialize generated objects from yaml? @teo I was thinking about, how a schema validation could fit in, I don't know, but I suppose if this was to be implemented, calamares needs a different loading of settings.conf, perhaps schema with a python class behind, like pykwalify supports?

Another question, perhaps related to serialization, I noticed that if some value in yaml conf is set, the ui is not updated. For example a selected group in netinstall is not shown as selected, however, it is selected initially by "selected" bool.

[teo]

Awesome work, @udeved ! I don't think we'd gain much by adding full validation to Calamares startup. We have to keep in mind that most Calamares startups happen on live systems for end users, after Calamares was already packaged and its configuration (hopefully) validated by system integrators.

We should however put those schema files some place where they will be found and used.

[teo]

For instance, the hacking directory in the main calamares repo.

Based on experience I fear that if the validation stuff is in a separate repo, no one will give a fuck except Manjaro developers who know of its existence.

[udeved]

I don't mind putting them in hacking dir, but perhaps data is better? Anyhow, I can make our tools depend on a schema package, just tell final destination, we don't need them in our tools source.

[udeved]

@teo

I have moved the schemas temporarily in a separate repo. https://github.com/udeved/calamares-schemas I was wondering, perhaps have a git submodule pull them from eg calamares/calamares-schemas? This way you wouldn't get commits of schemas in source tree? They are not perfect yet, some data types need changes to text, but good enough to do a general validation.

My thought earlier was, with xsd and xml, you can have a data object, a dataset, which is generated from schema and initialized with data xml. You can then query the dataset with lambda whatever and do all kind of cool stuff with it. Its like some kind of object oriented tables.