pydantic_core._pydantic_core.ValidationError: 47 validation errors for PipelineSpec

[amaddio]

I stumbled over the pipeline-runner project today. The tool looks exactly what I was looking for. It seams that this tool fills the gap as Atlassian does not provide a feature to properly lint nor test the pipeline prior to pushing it.

Unfortunately my complex bitbucket-pipelines.yml is considered invalid even though the pipeline works well when run on bitbucket cloud.

The command I use was pipeline-runner list. My pipeline configuration is quite complex but basically it is a repetition of steps in the parallel section. It looks like this with the repetitions removed:

image: atlassian/default-image:4
pipelines:
  branches:
    main:
      - step:
          name: Run Unit Tests if this is a Merge Commit
          deployment: test
          script:
            - set +e
            - git rev-parse HEAD^2 >/dev/null 2>/dev/null && echo "is a merge commit" || exit 0
            - set -e
            - pipe: atlassian/ssh-run:0.4.1
              variables:
                SSH_USER: 'root'
                SERVER: ${unit_test_server}
                COMMAND: 'cd /var/www/my_app && git pull'
                EXTRA_ARGS: '-T'
                ENV_VARS: ''
                DEBUG: 'true' # Optional
      - parallel:
        - step:
            name: Deploy to webserver
            deployment: production
            script:
              - set +e
              - if $(git show --shortstat | grep -q "2 files changed"); then printf "Deployment skipped, more than pyproject.toml has been changed\n"; exit 0 ; fi
              - set -e
              - echo "Deploying to webserver"
              - pipe: atlassian/ssh-run:0.4.1
                variables:
                  SSH_USER: 'root'
                  SERVER: ${webserver_01}
                  COMMAND: 'cd /var/www/my_app && git pull'
                  EXTRA_ARGS: '-T'
                  DEBUG: 'true' # Optional
            condition:
              changesets:
                includePaths:
                - "pyproject.toml"

I receive a ton of errors:

pydantic_core._pydantic_core.ValidationError: 47 validation errors for PipelineSpec
pipelines.branches.main.0.StepWrapper.step.runs-on
  Extra inputs are not permitted [type=extra_forbidden, input_value=['self.hosted', 'linux.shell'], input_type=list]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.0.ParallelStep.parallel
  Field required [type=missing, input_value={'step': {'name': 'Run Un...'', 'DEBUG': 'true'}}]}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.0.ParallelStep.step
  Extra inputs are not permitted [type=extra_forbidden, input_value={'name': 'Run Unit Tests ... '', 'DEBUG': 'true'}}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.0.Variables.variables
  Field required [type=missing, input_value={'step': {'name': 'Run Un...'', 'DEBUG': 'true'}}]}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.0.Variables.step
  Extra inputs are not permitted [type=extra_forbidden, input_value={'name': 'Run Unit Tests ... '', 'DEBUG': 'true'}}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.StepWrapper.step
  Field required [type=missing, input_value={'parallel': [{'step': {'...['pyproject.toml']}}}}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.1.StepWrapper.parallel
  Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step': {'name': 'Deplo... ['pyproject.toml']}}}}], input_type=list]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.ParallelStep.parallel.0.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.ParallelStep.parallel.1.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.ParallelStep.parallel.2.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.ParallelStep.parallel.3.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.ParallelStep.parallel.4.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.ParallelStep.parallel.5.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.ParallelStep.parallel.6.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.ParallelStep.parallel.7.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.ParallelStep.parallel.8.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.1.Variables.variables
  Field required [type=missing, input_value={'parallel': [{'step': {'...['pyproject.toml']}}}}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.1.Variables.parallel
  Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step': {'name': 'Deplo... ['pyproject.toml']}}}}], input_type=list]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.2.StepWrapper.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.2.ParallelStep.parallel
  Field required [type=missing, input_value={'step': {'name': 'check ...: ['pyproject.toml']}}}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.2.ParallelStep.step
  Extra inputs are not permitted [type=extra_forbidden, input_value={'name': 'check if web01,...': ['pyproject.toml']}}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.2.Variables.variables
  Field required [type=missing, input_value={'step': {'name': 'check ...: ['pyproject.toml']}}}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.2.Variables.step
  Extra inputs are not permitted [type=extra_forbidden, input_value={'name': 'check if web01,...': ['pyproject.toml']}}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.3.StepWrapper.step
  Field required [type=missing, input_value={'parallel': [{'step': {'...my_script.py']}}}}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.3.StepWrapper.parallel
  Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step': {'name': 'Resta..._my_other_script.py']}}}}], input_type=list]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.3.ParallelStep.parallel.0.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP..._another_script.py']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.3.ParallelStep.parallel.1.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...second_other_script.py']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.3.ParallelStep.parallel.2.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...: ['third_other_script.py']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.3.ParallelStep.parallel.3.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...fourth_other_script.py']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.3.ParallelStep.parallel.4.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...fith_other_script.py']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.3.ParallelStep.parallel.5.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...sixth_other_script.py']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.3.Variables.variables
  Field required [type=missing, input_value={'parallel': [{'step': {'...my_script.py']}}}}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.3.Variables.parallel
  Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step': {'name': 'other_script.py']}}}}], input_type=list]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.4.StepWrapper.step
  Field required [type=missing, input_value={'parallel': [{'step': {'...['pyproject.toml']}}}}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.4.StepWrapper.parallel
  Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step': {'name': 'Deplo... ['pyproject.toml']}}}}], input_type=list]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.4.ParallelStep.parallel.0.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.4.ParallelStep.parallel.1.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.4.ParallelStep.parallel.2.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.4.ParallelStep.parallel.3.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.4.ParallelStep.parallel.4.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.4.Variables.variables
  Field required [type=missing, input_value={'parallel': [{'step': {'...['pyproject.toml']}}}}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.4.Variables.parallel
  Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step': {'name': 'Deplo... ['pyproject.toml']}}}}], input_type=list]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.5.StepWrapper.step.condition
  Extra inputs are not permitted [type=extra_forbidden, input_value={'changesets': {'includeP...s': ['pyproject.toml']}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
pipelines.branches.main.5.ParallelStep.parallel
  Field required [type=missing, input_value={'step': {'name': 'check ...: ['pyproject.toml']}}}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.5.Variables.variables
  Field required [type=missing, input_value={'step': {'name': 'check ...: ['pyproject.toml']}}}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing
pipelines.branches.main.5.Variables.step

I know that debugging this issue is impossible if I don't provide the full yml file. Unfortunately I can't. I would be already happy to be advised how to fix the yml problems or ignore the errors and run the pipeline-runner run <pipelinename> command without failing on syntax errors.

[mathieu-lemay]

Thanks for the bug report @amaddio.

I can reproduce the issue with your sample so no worries there. The problem is the condition: bit, which I don't support yet. I will push a fix for that today. In the meantime, if you remove the conditions, it should work.

[amaddio]

You are a gun @mathieu-lemay. Just tried it. I can confirm that the pipeline-runner list cmd works, when the condition part is removed entirely

2024-08-20 16:47:04.798 pipeline_runner.cli: Available pipelines:
        branches.main 

[mathieu-lemay]

Happy to hear that!

I've added partial support for the condition block, meaning they will be parsed, but ignored with a warning when running a step. That way, it will work and the user will be made aware that the condition are explicitly ignored my the tool.

[mathieu-lemay]

Version 0.4.4 has been released to fix this issue. Please re-open the issue if you still encounter a problem!