Model Validation Workflow

[chgio]

We've set up a very nice docgen pipeline and, as it turns out, it also includes a model validation step.

However, as long as there's no bug in the workflow code, the validation step would pass regardless of model quality, as its success comes from the generation of the validation log and not from the absence of errors in the model. So, it would be nice to turn this into a workflow that propagates any error in the validation log to a pipeline failure: this way, we don't have to check the validation log just to see if the model is alright.

Eventually the pipeline should work like this:

stateDiagram-v2
direction LR

val     : Validate Model
dg      : Build Docs
dpl     : Deploy to Pages

[*] --> val
val --> dg : pass
val --> [*] : fail
note left of val : validation report
dg  --> dpl: main
note left of dg: HTML export
dg  --> [*] : model/*
dpl --> [*]

It should be fairly straightforward to parse the HTML validation report with something like BeautifulSoup (docs here) in :b:ython -- I'm thinking about this instead of a quick grep as we'll probably want some logic:

• [x] Parse the HTML
• [x] Disregard the following errors, but don't hard-code them:
  • TJ_SA_04: System Analysis (SystemAnalysis) does not realize Operational Analysis (Operational Analysis does not exist).
  • TC_DF_15: Root "Root System Function"(System Function) does not realize the Operational Analysis (Not Found), Please create Operational Analysis.
• [x] If there's any other Error in the Level column, fail the run
• [x] Otherwise, pass the run

Here are some sample validation runs and their expected outcome for testing:

• [x] build run on 8bc6a98 shall fail due to:
  • DWF_DS_14
• [x] build run on 9b41aa6 shall fail due to (any of):
  • DWF_DC_15
  • DWF_DC_09
  • I_19
• [x] build run on b71b610 shall fail due to:
  • I_19
• [x] build run on d8c4e4a shall pass

The test dataset isn't by any means comprehensive but the requirements are fairly simple and I'm basically giving you the test logic, so you can TDD it if you want :wink:

---

The validation step was originally included as a hack to get the Capella CLI to work and might not be needed in the future but, worst-case scenario, we can always make our own by spawning the following issue:

• [ ] Add Validation-Only Step to Capella Docker Action

[chgio]

For the moment, the logic can be placed into the current Build step, which still includes validation; in the future, if they get split either upstream or by ourselves, it should be moved to the Validation-only step.

Also, you may want to work on this kind of stuff on a merger of ci-test into cd-test (called cid-test or something like that)

[chgio]

you know me -- a further step could be turning it into one of my beloved pre-commit git hook (for the pre-commit framework, not a pure git hook):

• [ ] Model Validation Pre-Commit Hook

:wink: