(feat): schema validation, duplicity check

[lukasmrtvy]

Would be nice to have some schema validation. Maybe agains json schema ( https://github.com/xeipuuv/gojsonschema )?

vars:
    projects:
    - name: one
      path: /tmp/foo 
      desc: "bla"
      color: red
    - name: two
      value: /tmp/bar
      color: red
spec: 
    - id: name
      type: string
      required: yes
      unique: yes
    - id: path
      type: regex
      regex: .*
      required: yes
      unique: yes
    - id: desc
      type: string
      required: no
      unique: no
    - id: color
      type: string
      required: yes
      unique: no
      oneof:
      - red
      - blue
      - yellow

[stephencheng]

• Thought about the schema validation, it's good to have, however it (!sometimes) really restricts the flexibility and loose couple the data(the structure of up.yml) and the implementation(the execution)

I have put validation in node/element level to have done what necessary(using test cases) but not a framework to:

1. increase the complexity
2. reduce the flexibility

For example, I use dox for a mark to deactivate the step execution, also for simple cmd,eg

- func: cmd
  do: 
    - name: exit

If I/user would like to disable that part for an execution, simply use namex to mark that, it is kind of implicitly silly but convenient to do it so, for super turn around test -> dev -> deploy agile SDLC

Do you have some concrete cases that the schema validation is a must for certain scenarios? I'd hear that

I will keep this as a maybe feature to consider. Sorry, this just reminds lots of pain of XML schema validations .......

• As to the duplicity check

Yes, I am aware of the issue and it's in the plan for the enhancement, this is a must

Would you please raise this as enhancement? I will put this into my list

Thanks

[lukasmrtvy]

I want to create a CLI for git monorepo handling. Prototype of data loading:

• import list of projects from a file ( .json, .yaml ) ( via vars ref ).

  [{"name":"project1","value":"/foo/path/bar/"},{"name":"project2","value":"/hoo/path/fart/"}]

• validate agains schema
• validate for duplicates
• autodected mod via git diff -> detect changes in loaded projects
• run command on changed projects
• autodetect mod should be overridable with some env var like PROJECT=project1, for manual selection ..

Of course I can run some tool for checking schema and duplicates inside specific task, but I guess that it should be validated at import time via upcmd framework

[stephencheng]

I want to create a CLI for git monorepo handling. Prototype of data loading:

• import list of projects from a file ( .json, .yaml ) ( via vars ref ).

[{"name":"project1","value":"/foo/path/bar/"},{"name":"project2","value":"/hoo/path/fart/"}]

you can load it via varfile (I actually like to refactory the name to be something else like ref), but it requires a vars prefix header, or it is better to use:

• load dvar from data file: https://upcmd.netlify.app/dvars/c0088/

note that just use flags: toObj, it will automatically convert the data all to object, in your case, it is a array, then you can loop through it

or

• use dvar, but load it using template func: fileContent, see example from project up.yml

          - name: releaseNote
            value: '{{ printf "%s%s%s" "./release/tags/" .ver ".md" |fileContent}}'
            flags:
              - v
              - toObj
              - taskScope

so basically below will load the file for you and then toObj convert it to cached Object '{{ "/this/is/file" |fileContent}}'

• validate agains schema

UPcmd will automatically do the validation of yml for you, it is not schema level validation, but yaml syntax sanity check

convert it to yml if you are using json, '{{ yml_content |toJson}}'

• validate for duplicates

not sure how you will achieve this, you may use external tool to do it?? or loop it through, flatten the object to row by row and then take kind of md5 value of the full row to see if there is any duplicates?

• autodected mod via git diff -> detect changes in loaded projects
• run command on changed projects
• autodetect mod should be overridable with some env var like PROJECT=project1, for manual selection ..

These 3 can all be done, just use shell func and better wrap them into a task, then use call func with a local var to pass in the controlled parameter to dictate it is the project instead of the default

see: https://upcmd.netlify.app/call-func/c0111/

Of course I can run some tool for checking schema and duplicates inside specific task, but I guess that it should be validated at import time via upcmd framework

Yes, the yaml will be validated automatically

[stephencheng]

To conclude, below is in todo:

• Add task duplicity check

[stephencheng]

@lukasmrtvy the task duplicity check is added now, please use latest/latest rolling release

[lukasmrtvy]

@stephencheng This looks great as validation plug-in https://cuelang.org/

[stephencheng]

@lukasmrtvy It's already doable to validate the yml/json as builtin, but I just have not exposed the service yet as a cmd. If you intend to use the validation, I am happy to add the feature, please let me know if you really want it