Gitlab CI trigger includes

Bruceforce commented 3 years ago

What would you like Renovate to be able to do?

It would be great if renovate would support Gitlab CI triggered Pipeline includes. See https://docs.gitlab.com/ee/ci/yaml/#trigger-syntax-for-child-pipeline

For local files this would look like: .gitlab-ci.yml

stages:
  - trigger
  - release

Trigger Backend:
  stage: trigger
  trigger:
    include: .gitlab-ci-backend.yml
    strategy: depend

Release:
  image: node:16.11.0-alpine@sha256:6d5ecd68b7d28e63fbec26ae4d05fa679a7003325d8e4ea72e6dc318d9869899
  stage: release
  before_script:
    - apk add git
    - npm install -g semantic-release @semantic-release/gitlab @semantic-release/git @semantic-release/changelog
  script:
    - semantic-release
  rules:
    - if: $CI_COMMIT_BRANCH =~ "main"

.gitlab-ci-backend.yml

stages:
  - prepare

Prepare:Build Environment:
  stage: prepare
  image: alpine
  tags:
    - docker
  script:
    - ls -la

[...]

Note: There is also the possibility to include "remote files". This would look like:

stages:
  - trigger

Trigger Child:
  stage: trigger
  trigger:
    include:
      - project: 'my-group/my-pipeline-library'
        ref: 'main'
        file: '/path/to/child-pipeline.yml'

If you have any ideas on how this should be implemented, please tell us here.

Currently renovate is only looking for a structure like

include:
- project: ABC/XYZ
  file: /template.yaml
  ref: 1.0.0
-   project: ABC/XYZ
    file: /template.yaml
    ref: master

defined in https://github.com/renovatebot/renovate/blob/main/lib/manager/gitlabci-include/extract.ts to allow "general" includes. The triggered includes should be quite similar to set up if I got it right.

Is this a feature you are interested in implementing yourself?

No

viceice commented 3 years ago

Can you please extend the description with a concrete sample, i don't see anything there wich should be updated by renovate.

Bruceforce commented 3 years ago

@viceice I updated the description with a concrete example for the "local files" scenario. As you can see renovate added the digest to the node:16.11.0 image (I did not do that manually). However the included file .gitlab-ci-backend.yml was not touched by renovate. The alpine image is not touched.

I assumed that the "gitlabci-include" is only looking for include.file in the yml, which we do not have in my example (https://github.com/renovatebot/renovate/blob/main/lib/manager/gitlabci-include/extract.ts#L45) I may got this wrong.

If you would need a complete "how to reproduce" repository please let me know. I'll try to provide this in the next days.

github-actions[bot] commented 3 years ago

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

Bruceforce commented 3 years ago

Reproduction repository: https://gitlab.com/1O/renovate-12178-reproduction/-/tree/master

See that renovate created 2 branches in this repo:

These branches just change the file https://gitlab.com/1O/renovate-12178-reproduction/-/blob/renovate/digests-pin/.gitlab-ci.yml.

However the file hosting the triggered Pipeline https://gitlab.com/1O/renovate-12178-reproduction/-/blob/master/.gitlab-ci-child.yml is not touched at all. However my understanding would be that the exact same Branches should be created for .gitlab-ci-child.yml (since the content is basically the same, the alpine image should also be digested and updated)

I hope this made my point clear. If not please let me know.

aarongoldenthal commented 3 years ago

@Bruceforce I think the problem you have is with the file match and renovate is simply not looking at that file, I have several projects where other triggered pipeline files are managed by renovate. See the docs here for details on the default match, which is expecting files follow what is typically the GitLab naming standard of <function>.gitlab-ci.yml, and links to create a custom match for your specific file name.

Bruceforce commented 3 years ago

@aarongoldenthal Thanks for the hint.

However I think that would only be a workaround for the problem. If renaming or adjusting the is filematch of the gitlabci-manager would be the "official solution", then you also would not need the gitlabci-include manager.

This particular feature request is to support a check for triggered pipelines like the gitlabci-include manager does for includes. The naming of the file should not matter in this case. It could also be that you trigger a Pipeline which is not in the same repository (same as with includes). That's why I think this feature request is still valid.

I'm sorry if I missed something or got you wrong.

viceice commented 3 years ago

I think it's a good time to merge both gitlab managers, as they do some same things

aarongoldenthal commented 3 years ago

Sorry @Bruceforce, I misunderstood, but think I understand now and agree after a quick test. Let me see if I can summarize:

If you have a .gitlab-ci.yml file with the following content:

include:
  - project: "gitlab-ci-utils/gitlab-ci-templates"
    ref: 1.4.0
    file:
      - "/collections/All-Projects-Slim.gitlab-ci.yml"

trigger_include:
  stage: test
  trigger:
    include:
      - project: "gitlab-ci-utils/gitlab-ci-templates"
        ref: 1.4.0
        file:
          - "/collections/All-Projects-Slim.gitlab-ci.yml"

...and assuming the latest release is 2.0.0 for gitlab-ci-utils/gitlab-ci-templates (which it currently is), then renovate identifies one gitlabci-include dependency for the top level include and tries to update from 1.4.0 to 2.0.0, but doesn't identify include under the trigger as a dependency it should track and doesn't try to update it (as tested here, and is obviously a contrived example).

Even if it did, I do still think you'll have a file match issue since the gitlabci-include file match default is files ending in .gitlab-ci.yml (as of https://github.com/renovatebot/renovate/pull/12380).

Bruceforce commented 3 years ago

@aarongoldenthal no worries, but this is still not what I meant in particular. However this is a very interesting use case / feature request by itself in my opinion!

What I meant is (for your given example):

Open .gitlab-ci.yml
Look for "trigger.include (can be with OR without .file - see my example repo from above)
Traverse to the included pipeline
check that pipeline for Docker images (in your case All-Projects-Slim.gitlab-ci.yml)
create PR for Docker image updates

I've also given an example in https://github.com/renovatebot/renovate/issues/12178#issuecomment-947825496.

Of course this might not always be intended, for example If you explicitly not want to update images in the triggered pipeline because you do not own it, so this behavior should be configurable.

rarkins commented 3 years ago

I think that being able to "crawl" for files which aren't otherwise found is a good idea, because you can't always find everything with fileMatch - or now would it be efficient for people to check every single .yml file just in case, although we don't stop people from doing that.

There are already abilities to ignore files if people need, such as ignorePaths so ideally we'd need to think of ways to reuse or expose that so that we don't go parsing files the user has explicitly asked to ignore.

renovatebot / renovate