feat(#1141): 'preview' option with a valid YML against `ci/lint` Gitlab API route #1142

Closed e-picas closed 6 months ago

new 'merge' option and treatment
new 'expandVariables' for the Job constructor to flag if we should expand or not
new 'merge' test-case for validation

Closes #1141

Explanation

This introduces a new option named --merge (not sure about that, let me know if you have a better idea) which do basically the same job as the --preview one but without variables expansion. The result should be (quite) the same as the "Full configuration" tool of GitLab's internal CI editor: https://docs.gitlab.com/ee/ci/pipeline_editor/#view-full-configuration.

Dev implementation

I just added a new option called merge which follows the same logic as the preview one, adding a expandVariables bollean flag to switch between one or the other.

I also added correponding test-case to validate it on an environment block in a CI using a variable as 'name' that can not be guessed outside a git tag.

NOTE - I'm having a failure running the tests but I don't think it was due to my dev. Please see the final notes below.

Illustration

With the same YML as in the #1141 ticket:

# .gitlab-ci.yml
---

.anchored_vars: &anchored_vars
  MY_VAR: 'my var'

my_job:
  rules:
    - if: $CI_COMMIT_REF_NAME
      variables:
        <<: *anchored_vars
  script:
    - echo "$MY_VAR"
  environment:
    name: $MY_VAR

We now have a not-expanded result using the --merge options:

$ gitlab-ci-local --merge

---
stages:
  - .pre
  - build
  - test
  - deploy
  - .post
my_job:
  rules:
    - if: $CI_COMMIT_TAG
      variables:
        MY_VAR: my global var from .anchored_vars
    - when: never
  script:
    - echo $MY_VAR
  environment:
    name: $MY_VAR

Which is valid passing it to the ci/lint GitLab API validator:

$ jq --null-input --arg yaml "$(gitlab-ci-local --merge)" '.content=$yaml' \
  | curl "https://scm.clubmed.com/api/v4/projects/${GITLAB_PROJECT_ID}/ci/lint?include_merged_yaml=true" \
    --header "Authorization: Bearer ${GITLAB_TOKEN}" \
    --header 'Content-Type: application/json' \
    --data @- \
  | jq

{
  "valid": true,
  "errors": [],
  "warnings": [],
  "merged_yaml": "---\nstages:\n- \".pre\"\n- build\n- test\n- deploy\n- \".post\"\nmy_job:\n  rules:\n  - if: \"$CI_COMMIT_TAG\"\n    variables:\n      MY_VAR: my global var from .anchored_vars\n  - when: never\n  script:\n  - echo $MY_VAR\n  environment:\n    name: \"$MY_VAR\"\n",
  "includes": []
}

NOTES

I'm a newbie in TS dev so do not hesitate for any change request ;)
The name of the option can really be challenged as I'm not sure about "merge", which is actually what it does but has a special meaning in a VCS environment...

Running the test suite to validate my dev I faced a single failure but I don't think it was due to it (please let me know):


Summary of all failing tests
FAIL  tests/test-cases/artifacts-shell/integration.artifacts-shell.test.ts (6.889 s)
● artifacts-shell <consume> --needs --shell-isolation

expect(received).toEqual(expected) // deep equality

Expected: 1
Received: 2

  23 |         return l.match(regex) !== null;
  24 |     });
> 25 |     expect(found.length).toEqual(1);
     |                          ^
  26 |     expect(await fs.pathExists("tests/test-cases/artifacts-shell/path/file1")).toEqual(true);
  27 | });
  28 |

  at tests/test-cases/artifacts-shell/integration.artifacts-shell.test.ts:25:26

Test Suites: 1 failed, 109 passed, 110 total Tests: 1 failed, 260 passed, 261 total Snapshots: 0 total Time: 55.212 s, estimated 60 s

@firecow I'm working for a french tourism company (https://github.com/ClubMediterranee) and this feature and my two other fixes (#1138 and #1140 ) are quite urgent for us as we would love to use your tool in our internal CI development validation (we have a CE Gitlab factory and are on the way to increase our developments robustness). I'm totally open for discussion, help, question etc.

And many thanks for your great work ! The gitlab-ci-local tool is just amazing ;)

other fixes (https://github.com/firecow/gitlab-ci-local/pull/1138 and https://github.com/firecow/gitlab-ci-local/pull/1140 ) are quite urgent for us as we would love to use your tool in our internal CI development validation

@e-picas by the way, It might take some time to wait for the review and creation of a new release.

Instead, it'll probably be much quicker for you to create the executable from your forked branch

Using the following:

npm run build
npm run pkg-all

$ tree bin                                                                        
├── linux
│   └── gitlab-ci-local
├── linux.gz
├── macos
│   └── gitlab-ci-local
├── macos.gz
├── win
│   └── gitlab-ci-local.exe
└── win.gz

Copy the respective executable to your $PATH

Hello @ANGkeith , thank you for the advice, that's exactly what I did ;)

@e-picas Let's just fix how --preview is working instead of adding a new option

I find it perfectly reasonable that --preview output should be able to pass https://gitlab.com/ci/lint, and I'm sure the other collaborators do too.

@e-picas I'll be publishing a new version tomorrow :+1:

firecow / gitlab-ci-local