search

OpenFn / lightning

OpenFn/Lightning ⚡️ is the newest version of the OpenFn DPG and provides a web UI to visually manage complex workflow automation projects.
https://openfn.github.io/lightning/
GNU Lesser General Public License v3.0
125 stars 33 forks source link

Don't generate invalid yaml #2446

Open midigofrank opened 2 weeks ago

midigofrank commented 2 weeks ago

Projects having workflows with special YAML characters like : fails to pull

Yaml does not do well with special characters and that means cli pull and deploy operations will always run into an error when a project has workflows with those characters in names.

There are two ways to do this, enforce some data validation on lightning or do data manipulation. I'd default validation because data manipulation might be expensive with multiple edge cases to deal with/test.

christad92 commented 1 week ago

A fix for this from @stuartc (assumed) is that we should quote special strings that contain yaml-controlled characters. This might be a fix on the CLI

christad92 commented 1 week ago

This needs to be applied to condition expressions. for example where the user has !state.path as a condition expression, and we are trying to deploy, we get an error that says "field can't be blank" (See below). 

{
  "errors": {
    "workflows": {
      "WF2.1-Get Patients from CommCare": {
        "edges": {
          "GET by IHS->GET by NIK": {
            "condition_expression": [
              "This field can't be blank."
            ]
          },
          "GET by NIK->GET by GDN": {
            "condition_expression": [
              "This field can't be blank."
            ]
          },
          "GET by NIK->Process next step": {
            "condition_expression": [
              "This field can't be blank."
            ]
          }
        }
      }
    }
  }
}
stuartc commented 1 week ago

In the ExportUtils module, when building the workflow YAML keys, if the name contains any characters other than [A-Za-z-_] then wrap the key in double quotes.

@christad92 your example is super interesting, but I think its a different code path; we need to test this out (it looks like an error on push).

stuartc commented 1 week ago

@christad92 could you give a list of values or keys that we believe are culprits here, things that when parsed by the CLI produce values or keys that are either not valid YAML or incorrect. Your condition_expression example there is 👌 .

christad92 commented 1 week ago

@stuartc these are two issues we found.