Add a new `maestro validate` ("dry run") command

[prasanta-biswas]

Is your feature request related to a problem? Please describe. Facing issues in maintaining the sanity of flow files in the repo where multiple people are adding commits and there is no way to verify the sanity of the flow files since maestro does not have a build system that can verify for complication error.

Describe the solution you'd like In a CI system where multiple teams are adding tests in the same repo, it is essential to check the flow files' sanity and syntax in each merge request. A feature like checking the syntax, and file paths in the flow files of the entire workspace is necessary so that no bad flow files are added. A dry-run like feature can help validate the workspace and yaml flow files instead of running any.

Describe alternatives you've considered Currently, there is no way of validating the entire workspace.

[bartekpacia]

Hey @prasanta-biswas, thanks for filing this thoughtful feature request!

To concretize your idea a bit, you want to be able to run:

• Run on a single file:

  $ maestro validate ./flows/flow.yaml

• Run on a directory:

  $ maestro validate ./flows

• Run on a list of files/directories:

  $ maestro validate ./flows/subdir ./flows/flow_1.yaml ./flows/flow_2.yaml

• Run recursively:

  $ maestro validate --recursive ./flows/

• If validation on any of input files fails, maestro exits with non-zero exit code

• Paths to invalid files are printed in stdout line by line, e.g.:

  $ maestro validate --recursive ./flows
  [stderr] The following flows are invalid:
  [stdout] ./flows/invalid_flow_1.yaml
  [stdout] ./flows/subdir/invalid_flow_1.yaml

I'm not sure how much validation we can do. I think we can check if e.g. paths to assets used with addMedia, or runScript, are correct. But there are things that must happen at runtime, e.g. checking if appId is correct.

Overall, what do you think? Would this work for you? And, would you be willing to contribute this?

[bartekpacia]

There's also a thirdy-party Maestro YAML Validator tool (I found it linked here), might be useful as a stopgap solution

[prasanta-biswas]

@bartekpacia That is exactly what I meant.

[prasanta-biswas]

@bartekpacia Validating file paths runScripts, runFlow etc in yaml files as well as the commands and syntax used in the yaml files would do just fine. Runtime things can be ignored in this validation. I will see if I can contribute on this. I already have a PR raised to fix the setLocation command to accept coordinates as parameters.

[bartekpacia]

Awesome! And thanks for letting me know about that other PR of yours, I am taking a look at it right now:)