`--preview` or a new option to have a merged YAML compatible with the `ci/lint` API route

[e-picas]

The actual result of the --preview option is not compatible with the GitLab internal ci/lint API route which validates if a CI config is valid or not.

If it's not the purpose of the --preview option, maybe creating a new --merged or --preview-merged option could do the job ?

Final purpose

The final goal is to be able to validate a complete local development of CI, with all the concerned includes, sending the result as the ci/lint API route content parameter:

jq --null-input --arg yaml "$(gitlab-ci-local --preview)" '.content=$yaml' \
| curl "https://scm.clubmed.com/api/v4/projects/${GITLAB_PROJECT_ID}/ci/lint?include_merged_yaml=true" \
  --header "Authorization: Bearer ${GITLAB_TOKEN}" \
  --header 'Content-Type: application/json' \
  --data @- \
| jq '.valid'
# should always be 'true'

Minimal .gitlab-ci.yml illustrating the issue

I haven't tested all the combinations of all keywords but I've identify at least the following case which is failing:

# .gitlab-ci.yml
---

.anchored_vars: &anchored_vars
  MY_VAR: 'my var'

my_job:
  rules:
    - if: $CI_COMMIT_REF_NAME
      variables:
        <<: *anchored_vars
  script:
    - echo "$MY_VAR"
  environment:
    name: $MY_VAR

Below is the result of the --preview option:

$ gitlab-ci-local --preview

---
stages:
  - .pre
  - build
  - test
  - deploy
  - .post
my_job:
  rules:
    - if: $CI_COMMIT_REF_NAME
      variables:
        MY_VAR: my var
  script:
    - echo "$MY_VAR"
  environment:
    name: ''

Passing this to the API will fail:

$ jq --null-input --arg yaml "$(gitlab-ci-local --preview)" '.content=$yaml' \
  | curl "https://scm.clubmed.com/api/v4/projects/${GITLAB_PROJECT_ID}/ci/lint?include_merged_yaml=true" \
    --header "Authorization: Bearer ${GITLAB_TOKEN}" \
    --header 'Content-Type: application/json' \
    --data @- \
  | jq

{
  "valid": false,
  "errors": [
    "jobs:my_job:environment name can't be blank",
    "jobs:my_job:environment name can contain only letters, digits, '-', '_', '/', '$', '{', '}', '.', and spaces, but it cannot start or end with '/'"
  ],
  "warnings": [],
  "merged_yaml": "---\nstages:\n- \".pre\"\n- build\n- test\n- deploy\n- \".post\"\nmy_job:\n  rules:\n  - if: \"$CI_COMMIT_REF_NAME\"\n    variables:\n      MY_VAR: my var\n  script:\n  - echo \"$MY_VAR\"\n  environment:\n    name: ''\n",
  "includes": []
}

Expected behavior

In that particular case, the enviroment name and url should not be replaced by an empty variable. This is probably a more global rule: if we can't find a variable value (it is undefined or empty), keep the original '$VALUE' in place...

In deed, if we change the payload like the following:

# .gitlab-ci-validated.yml

---
stages:
  - .pre
  - build
  - test
  - deploy
  - .post
my_job:
  rules:
    - if: $CI_COMMIT_REF_NAME
      variables:
        MY_VAR: my var
  script:
    - echo "$MY_VAR"
  environment:
    name: '$MY_VAR'

We have a validated result:

$ jq --null-input --arg gitlabci "$(<.gitlab-ci-validated.yml)" '.content=$gitlabci' \
  | curl "https://scm.clubmed.com/api/v4/projects/${GITLAB_PROJECT_ID}/ci/lint?include_merged_yaml=true" \
    --header "Authorization: Bearer ${GITLAB_TOKEN}" \
    --header 'Content-Type: application/json' \
    --data @- \
  | jq

{
  "valid": true,
  "errors": [],
  "warnings": [],
  "merged_yaml": "---\nstages:\n- \".pre\"\n- build\n- test\n- deploy\n- \".post\"\nmy_job:\n  rules:\n  - if: \"$CI_COMMIT_REF_NAME\"\n    variables:\n      MY_VAR: my var\n  script:\n  - echo \"$MY_VAR\"\n  environment:\n    name: \"$MY_VAR\"\n",
  "includes": []
}

Host information

• gitlab-ci-local 4.46.1

[e-picas]

Let me know if I can help in any way about this ticket!

[e-picas]

Sorry this one was not really clear. I split it in three other tickets and made (or will) corresponding merge-requests:

• 1137 (PR #1138 )

• 1139 (PR #1140 )

• 1141 (PR to come)