glab pipeline ci lint returns invalid for a valid yml configuration

[dacieri]

Dear all, I have just discovered this interesting project, which looks very promising to me. I was trying the CI linter option on one of my project, and I get this invalid error:

➜ glab pipeline ci lint
Getting contents in .gitlab-ci.yml
Validating...
.gitlab-ci.yml is invalid
1 Local file `/.before_script.yml` does not have project!

Naturally the yml is correct, but maybe the linter has some problems with the included ymls? In my case, I include the CI configuration first from another gitlab project, where all stages are defined, and then in my local files I extend the pre-defined stages. If I paste the yml on the gitlab online CI linter, I don't get an unvalid response, in comparison.

• Version used: glab version 1.15.0-27-gf702529 (2021-02-24)
• Operating System and version: Pop 20.10 groovy

[profclems]

Thanks @dacieri for reporting this issue... glab heavily relies on the GitLab API. I have found two issues on GitLab that are related to this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/217018 https://gitlab.com/gitlab-org/gitlab/-/issues/31971

I believe this issue should be resolved for repos hosted on GitLab.com and self hosted instances running v13.6+.

Is your project hosted on GitLab.com or a self-hosted instance?

[dacieri]

My host is gitlab.cern.ch, we should be running GitLab Enterprise Edition 13.7.6-ee

[profclems]

I think this is an issue with the GitLab CI Lint API: https://gitlab.com/gitlab-org/gitlab/-/issues/320778

The existing API endpoints allow for linting of CI scripts that only include scripts on the default branch, but not newly added ones.

[profclems]

Can you share a few insensitive part of your CI yaml?

[dacieri]

sure, that's the .gitlab-ci.yml

include:
    - project: 'hog/Hog'
      file: '/hog.yml'
      ref: 'Hog2021.1-1'
    - local: '/.before_script.yml'
    - local: "/.user_jobs.yml"
    - local: '/.devel_to_master.yml'
    #---------------------------------------------
    #         HW
    #---------------------------------------------
    # - local: '/Top/profpga_duo_test/gitlab-ci.yml'
    # - local: '/Top/profpga_duo_csf_plus_pt.0/gitlab-ci.yml'
    # - local: '/Top/profpga_duo_pt.0/gitlab-ci.yml'
    # - local: '/Top/hal/gitlab-ci.yml'
    # - local: '/Top/l0mdt/gitlab-ci.yml'
    #---------------------------------------------
    #         Modules
    #---------------------------------------------
    - local: '/Top/csf.0/gitlab-ci.yml'
    - local: '/Top/csf.1/gitlab-ci.yml'
    - local: '/Top/ptc_mpi/gitlab-ci.yml'
    - local: '/Top/lsf/gitlab-ci.yml'
    - local: '/Top/upt/gitlab-ci.yml'
    - local: '/Top/ucm/gitlab-ci.yml'
    - local: '/Top/hp/gitlab-ci.yml'
    - local: '/Top/heg/gitlab-ci.yml'
    - local: '/Top/hps_csf/gitlab-ci.yml'
    - local: '/Top/hps_lsf/gitlab-ci.yml'
    # - local: '/Top/daq/gitlab-ci.yml'
    - local: '/Top/mpl/gitlab-ci.yml'
    # - local: '/Top/tdc/gitlab-ci.yml'
    # - local: '/Top/mtc/gitlab-ci.yml'
    #---------------------------------------------
    #         Builds
    #---------------------------------------------
    # - local: '/Top/ult/gitlab-ci.yml'
    - local: '/Top/ult_tp_dev/gitlab-ci.yml'
    - local: '/Top/ult_tp_ucm/gitlab-ci.yml'
    - local: '/Top/ult_tp_csf/gitlab-ci.yml'
    - local: '/Top/ult_tp_csf_ptmpi/gitlab-ci.yml'
    - local: '/Top/ult_tp_lsf/gitlab-ci.yml'
    # - local: '/Top/ult_tp_lsf_questa/gitlab-ci.yml'
    # - local: '/Top/ult_tp_lsf_xsim/gitlab-ci.yml'
    # - local: '/Top/ult_daq/gitlab-ci.yml'
    # - local: '/Top/csf_plus_pt/gitlab-ci.yml'

we make an extensive use of the include feature, so I don't know if that would impact. The main yaml that is included is this https://gitlab.cern.ch/hog/Hog/-/blob/master/hog.yml, which includes other three yml in the reference repository. These yamls include both template jobs that are later defined in the locally included yml (i.e. - local: '/Top/csf.0/gitlab-ci.yml') and normal jobs that are executed by the CI (https://gitlab.cern.ch/hog/Hog/-/blob/master/YAML/hog-main.yml).

Maybe the project is too complex? :)

[profclems]

Thanks a lot @dacieri... I finally figured this is an issue from our part. We use the general CI validation endpoint, which doesn’t have any namespace specific context so it sees the local include as belonging to a different unknown project.

We should use the Project-specific CI validation endpoint, which is quite young (i.e. introduced in GitLab v13.6), but it uses all namespace specific data available, including variables, local includes, and so on.

I will submit a PR with a fix. This means we would have to compromise users on self-hosted GitLab instances using versions older than v13.6

[zemzale]

What if we just hit both endpoints at the same time? We can do that concurrently. If the Project endpoint just returns a 404, we can just ignore that, and fallback to the general CI validation endpoint

[profclems]

What if we just hit both endpoints at the same time? We can do that concurrently. If the Project endpoint just returns a 404, we can just ignore that, and fallback to the general CI validation endpoint

Nice one... I think we should go with that

[clemsbot]

This issue has been automatically marked as stale because it has not had recent activity. We haven't had the time to address it yet, but we want to keep it open. This message is just a reminder for us to help triage issues.

[hoelzro]

I looked into this a bit - I tried using the Validate a CI YAML configuration with a namespace API, which seems to work, but it has an issue where includes are resolved from the project's default branch, and you don't seem to be able to override that :frowning_face:

[clemsbot]

This issue has been automatically marked as stale because it has not had recent activity. We haven't had the time to address it yet, but we want to keep it open. This message is just a reminder for us to help triage issues.