Kestra Flow Validate marking 1 flow as invalid when it's valid

[japerry911]

Describe the issue

Execute:

./run kestra:cli 1 flow validate kestra_flows/automations/bigquery --server=https://us.kestra.cloud --api-token=<API-TOKEN> --tenant=<TENANT-DEV-ID>

Result:

✘ - x.automations.bigquery.bigquery_dedupper
        Unable to parse flows due to the following error:
        - tasks[0].taskRunner.region: must not be null

Workflow YAML:

...
tasks:
  - id: run_deduplication_python_command
    type: io.kestra.plugin.scripts.python.Commands
    beforeCommands:
      - export PYTHONPATH=src
      - export GCP_PROJECT_ID={{ secret('GCP_PROJECT_ID') }}
    containerImage: <valid-image-uri>
    commands:
      - >
        python src/automations/bigquery/dedupper/main.py 
        ...
    warningOnStdErr: false
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      waitUntilCompletion: 18000
      machineType: e2-micro
      computeResource:
        cpu: "2000"
        memory: "1024"
...

Plugin Defaults in root namespace:

- type: io.kestra.plugin.scripts.python.Commands
  forced: false
  values:
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      bucket: "{{ secret('GCS_BUCKET') }}"
      projectId: "{{ secret('GCP_PROJECT_ID') }}"
      region: <gcp valid region string here>
      serviceAccount: "{{ secret('GCP_SA_JSON') }}"

I think it does not validate well when plugin_defaults is present for ee.batch region.

Let me know your all's thoughts, or if you need me to do anything debug-wise.

Thank you team 🙏 🚀

Environment

• Kestra Version: 0.18.6
• Operating System (OS/Docker/Kubernetes): Kestra Cloud (validate command ran locally in docker container with CLI command 0.18.6)
• Java Version (if you don't run kestra in Docker): n/a

[anna-geller]

@smunteankestra hey Stefan, could you have a look and try to reproduce first? Jack also shared the info: "It's invalidating everything that uses plugin defaults for the ee.batch region. Flows work fine though, just validation issues"

[Ben8t]

indeed something going wrong with pluginDefaults here. I'm able to reproduce. When putting the region prop in the taskRunner directly the validate command is successful. cc @smunteankestra

(command to run the validate command without full infra ./kestra flow validate flows/test.yml --local)

[japerry911]

sorry just rereading, @Ben8t , do you comment adding --local at the end of my command for GitHub Actions Kestra workflows validations?

[Ben8t]

@japerry911 adding --local flag is only useful to validate Kestra Flow on the client side; so you don't have to provider server url and/or auth. Convenient for quick validation 👍

[loicmathieu]

@japerry911 do you use a Kestra EE image for the CLI or a Kestra image? Namespace plugin defaults is an EE feature so if you use an image which is not EE it cannot use them.

[Ben8t]

@loicmathieu FYI I reproduced on EE image on my side 👍

[loicmathieu]

With --local it cannot work as namespace config would not be available

[Ben8t]

Ok I see, I reproduced with pluginDefault at the flow level... so still something to elucidate

[japerry911]

I utilize image kestra/kestra:v0.18.6

[japerry911]

Any update on this? Also I realized that it's using the GitHub action validate - https://github.com/kestra-io/validate-action

so using whatever image is in there

[anna-geller]

Jack for transparency this is on 0.20 scope so we'll investigate after the release on Tuesday and if you need it soon, we can try to ship it in a bugfix release

[japerry911]

@anna-geller , 0.20 is totally ok, no worries. It's technically still working, it just fails on those, but it still checks everything else, which is all I need. Thank you for heads up 🙏