Convert jobs from bundler to pdk

[cdenneen]

Currently all CI templates still utilize bundler. I would be helpful for these to be converted to using pdk.

[DavidS]

The https://github.com/puppetlabs/pdk-docker images could help with that. We currently do not have this as one of our priorities, but would accept a community contribution.

[DavidS]

(no idea what was going on with the label there)

[cdenneen]

Just submitted puppetlabs/pdk-docker#17 to hopefully allow this to be done. I've got a working set of jobs for .gitlab-ci.yml I'm testing that could be used to replace the existing ones with pdk if I can get past the litmus limitation using the puppet/pdk container.

[logicminds]

I have one of these for gitlab-ci. I can submit a PR soon.

[cdenneen]

I have one of these for gitlab-ci. I can submit a PR soon.

I do too except for I needed a custom pdk image that contains docker binary for litmus. Be interested if you found way around that

[logicminds]

I actually used the shell provider for acceptance testing. I was having a hard time using dind to run litmus tests. When I did try I had a custom image with pdk and docker. There were many things hard coded I needed to get around. Also the provision provider I used was docker_exp which was a bit better.

There were some other hacks involved sed -i 's/rake (12.3.3)/rake (13.0.1)/' Gemfile.lock # hack to get around rake dependency issue

Litmus has a rake task so you can control the install modules with a few parameters. However, the parameters don't work on the CLI because true always becomes "true" which breaks things. Below is what made it work.

pdk bundle exec ruby -r 'puppet_litmus/rake_tasks' -e "Rake::Task['litmus:install_modules_from_directory'].invoke(nil,nil,nil,true)"

IMHO litmus should have used a CLI framework like GLI or TTY on top instead of rake.

The first stage here is to download a bundle which is a tar file of modules so I didn't use internet to download everything.

Regarding submitting this to pdk-templates. I am not sure how nor have time to transform this to a erb file because it looks really complex.

---
stages:
  - download
  - syntax
  - unit
  - acceptance

image: 
  name: puppet/pdk
  entrypoint: [""]

before_script:
  - mkdir -p ~/.ssh && ssh-keyscan -t rsa gitlab.com >> ~/.ssh/known_hosts
  - mkdir -p spec/fixtures/

download_bundle:
  artifacts:
      paths:
        - puppet_module_bundle.tar.gz
        - spec/fixtures/modules
  cache:
    paths:
      - puppet_module_bundle.tar.gz
      - spec/fixtures/modules
  image: litmusimage/centos:8
  stage: download
  script:
    - "curl -H \"JOB-TOKEN: ${CI_JOB_TOKEN}\" -o puppet_module_bundle.tar.gz \"${CI_API_V4_URL}/projects/xxx/packages/generic/module_bundle/0.0.1/puppet_module_bundle.tar.gz\""
    - tar -zxf puppet_module_bundle.tar.gz -C spec/fixtures
    - ls -l spec/fixtures/modules

.validate_template: &validate_config
  allow_failure: true
  stage: syntax
  script:
    - pdk validate

.spec_template: &spec_config
  allow_failure: true
  stage: unit
  script:
    - pdk test unit --parallel

pdk_validate ruby 2.5 ~> puppet 6:
  <<: *validate_config
  variables:
    PDK_PUPPET_VERSION: '6'

pdk_validate ruby 2.7 ~> puppet 7:
  <<: *validate_config
  variables:
    PDK_PUPPET_VERSION: '7'

pdk_unit_ruby_2.5 ~> puppet 6:
  <<: *spec_config
  variables:
    PDK_PUPPET_VERSION: '6'

pdk_unit_ruby_2.7 ~> puppet 7:
  <<: *spec_config
  variables:
    PDK_PUPPET_VERSION: '7'

# because of the rake dependency issue we have to break out the tests.  This is duplicated code which 
# only differs with one command

.litmus_test_puppet6: &litmus_test_puppet6
  allow_failure: true
  stage: acceptance
  tags:
    - docker-shell
  after_script:
    - pdk bundle exec rake litmus:tear_down
  before_script:
    -  pdk bundle exec rake 'litmus:provision_list[docker]'
    - "pdk bundle exec bolt command run 'puppet --version' -t docker_nodes --inventoryfile spec/fixtures/litmus_inventory.yaml"
    - ls -l spec/fixtures/modules
    - pdk bundle exec ruby -r 'puppet_litmus/rake_tasks' -e "Rake::Task['litmus:install_modules_from_directory'].invoke(nil,nil,nil,true)"
  script:
    - "pdk bundle exec rake litmus:acceptance:parallel"

.litmus_test_puppet7: &litmus_test_puppet7
  allow_failure: true
  stage: acceptance
  tags:
    - docker-shell
  after_script:
    - pdk bundle exec rake litmus:tear_down
  before_script:
    - "pdk bundle exec rake spec_prep"
    - sed -i 's/rake (12.3.3)/rake (13.0.1)/' Gemfile.lock # hack to get around rake dependency issue
    - pdk bundle exec rake 'litmus:provision_list[docker]'
    - "pdk bundle exec bolt command run 'puppet --version' -t ssh_nodes --inventoryfile spec/fixtures/litmus_inventory.yaml"
    - ls -l spec/fixtures/modules
    - pdk bundle exec ruby -r 'puppet_litmus/rake_tasks' -e "Rake::Task['litmus:install_modules_from_directory'].invoke(nil,nil,nil,true)"
  script:
    - "pdk bundle exec rake litmus:acceptance:parallel"

acceptance_test puppet6:
 variables:
   PDK_PUPPET_VERSION: 6
   WGET_INSTALL_CMD: "yum install wget -y"
 <<: *litmus_test_puppet6

# provision.yml
# ---
# In order to test for puppet7 we will need to create images with puppet7 installed
# Once we create the images we have to add to our provision.yaml
# docker:
#   provisioner: docker_exp
#   images: 
#     - 'litmusimage/centos:7'
#     - 'litmusimage/centos:8'

[cdenneen]

Yeah here is mine... you did a bunch of anchors... yes I did a custom pdk with docker-ce binary for litmus... works pretty well:

---
stages:
  - syntax
  - unit
  - acceptance
  - build
  - release
variables:
  CI_REGISTRY: 'privateart.jfrog.io'

default:
  image: privateart.jfrog.io/docker/ap/pdk-docker:latest

  cache:
    paths:
      - /root/.pdk/cache

pdk validate -Puppet ~> 7:
  stage: syntax
  script:
  - pdk validate
  variables:
    PDK_PUPPET_VERSION: '7'
  only:
    refs:
    - branches
  except:
    variables:
    - "$CI_COMMIT_MESSAGE =~ /\\[skip[ _-]tests?\\]/i"
pdk test unit-Puppet ~> 6:
  stage: unit
  script:
  - pdk test unit
  variables:
    PDK_PUPPET_VERSION: '6'
  only:
    refs:
    - branches
  except:
    variables:
    - "$CI_COMMIT_MESSAGE =~ /\\[skip[ _-]tests?\\]/i"
pdk test unit-Puppet ~> 7:
  stage: unit
  script:
  - pdk test unit
  variables:
    PDK_PUPPET_VERSION: '7'
  only:
    refs:
    - branches
  except:
    variables:
    - "$CI_COMMIT_MESSAGE =~ /\\[skip[ _-]tests?\\]/i"
litmus-centos7-puppet6:
  stage: acceptance
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: 1
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
    PDK_PUPPET_VERSION: '6'
  services:
  - name: docker:19.03.13-dind
    command:
    - "--registry-mirror"
    - "https://privateart.jfrog.io"
  before_script:
  - until docker info; do sleep 1; done
  - mkdir -p $HOME/.docker; echo $DOCKER_AUTH_CONFIG > $HOME/.docker/config.json
  - pdk bundle install
  - pdk bundle exec rake 'litmus:provision_list[centos7]'
  - pdk bundle exec rake 'litmus:install_agent[puppet6]'
  - pdk bundle exec rake litmus:install_module
  script:
  - pdk bundle exec rake litmus:acceptance:parallel
  only:
    refs:
    - branches
  except:
    variables:
    - "$CI_COMMIT_MESSAGE =~ /\\[skip[ _-]tests?\\]/i"
litmus-centos7-puppet7:
  stage: acceptance
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: 1
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
    PDK_PUPPET_VERSION: '7'
  services:
  - name: docker:19.03.13-dind
    command:
    - "--registry-mirror"
    - "https://privateart.jfrog.io"
  before_script:
  - until docker info; do sleep 1; done
  - mkdir -p $HOME/.docker; echo $DOCKER_AUTH_CONFIG > $HOME/.docker/config.json
  - pdk bundle install
  - pdk bundle exec rake 'litmus:provision_list[centos7]'
  - pdk bundle exec rake 'litmus:install_agent[puppet7]'
  - pdk bundle exec rake litmus:install_module
  script:
  - pdk bundle exec rake litmus:acceptance:parallel
  only:
    refs:
    - branches
  except:
    variables:
    - "$CI_COMMIT_MESSAGE =~ /\\[skip[ _-]tests?\\]/i"
build-package:
  stage: build
  allow_failure: false
  only:
  - tags
  script:
  - pdk release --skip-changelog --skip-publish --force --version=${CI_COMMIT_TAG:1}
  artifacts:
    paths:
    - pkg/*.tar.gz
push-package:
  stage: release
  allow_failure: false
  before_script:
  - echo $CI_BUILD_NAME
  only:
  - tags
  image: releases-docker.jfrog.io/jfrog/jfrog-cli-v2
  variables:
    JFROG_CLI_OFFER_CONFIG: 'false'
    CI: 'true'
  script:
  - jfrog rt upload --url https://privateart.jfrog.io --user ${GITLAB_CI_USER}
    --password ${GITLAB_CI_TOKEN} --flat pkg/*-${CI_COMMIT_TAG:1}.tar.gz artifactory/puppet/ap/${CI_PROJECT_NAME#puppet-}/

So I definitely like the idea of your anchor templating to reduce the code. Not sure I fully understand the download_bundle part though instead of building a package? I build it and then push it to my internal JFrog forge. All of my gitlab-ci is done in a gitlab-runner in Kubernetes so I don't have docker-shell to use for litmus.

I could probably use the custom pdk-docker image for just the litmus runs and the public pdk for the other jobs, since it's a smaller image but since the image has to get pulled anyway the less images the better after that initial pull is complete I would think.

[cdenneen]

So I tried putting this together in .sync.yaml to work with the anchor templates but it's doing something weird with the only: and except: blocks:

Here is the template before convert/update:

      .validate_template: &validate_config
        stage: syntax
        script:
          - pdk validate
        only:
          refs:
          - branches
        except:
          variables:
          - $CI_COMMIT_MESSAGE =~ /\[skip[ _-]tests?\]/i

      .spec_template: &spec_config
        stage: unit
        script:
          - pdk test unit --parallel
        only:
          refs:
          - branches
        except:
          variables:
          - $CI_COMMIT_MESSAGE =~ /\[skip[ _-]tests?\]/i

      pdk validate -Puppet ~> 7:
        <<: *validate_config
        variables:
          PDK_PUPPET_VERSION: '7'

      pdk test unit-Puppet ~> 6:
        <<: *spec_config
        variables:
          PDK_PUPPET_VERSION: '6'

      pdk test unit-Puppet ~> 7:
        <<: *spec_config
        variables:
          PDK_PUPPET_VERSION: '7'

but after the pdk update/convert it basically takes the template section and does this:

-  only:
+  only: &1
     refs:
     - branches
-  except:
+  except: &2
     variables:
     - "$CI_COMMIT_MESSAGE =~ /\\[skip[ _-]tests?\\]/i"

and the <<: * where it's referenced it adds only: and exclude: as such, instead of just leaving the <<: * desired syntax:

+pdk validate -Puppet ~> 7:
+  stage: syntax
+  script:
+  - pdk validate
+  only: *1
+  except: *2
+  variables:
+    PDK_PUPPET_VERSION: '7'
+pdk test unit-Puppet ~> 6:
+  stage: unit
+  script:
+  - pdk test unit --parallel
+  only: *3
+  except: *4
+  variables:
+    PDK_PUPPET_VERSION: '6'

Does the <<: *spec_config need to be escaped or something?

[cdenneen]

FWIW I tried to have the pdk-docker image publish a pdk and pdk-dind image so there would be one publically available for litmus dind testing but that didn't get approved by the team: https://github.com/puppetlabs/pdk-docker/pull/25

Not sure if @DavidS or @da-ar have any suggestions for how to get around this limitation. Right now the limitation is litmus trying to do dind for spinning up a container... but maybe there is a different approach to accomplish this for CI systems that doesn't require DinD?

[logicminds]

I am wondering if we can come up with several ci configs and use the include feature of gitlab ci.

1. prep work ( in my case bundle download)
2. validate
3. unit
4. acceptance
5. deploy?
6. publish?

Then the pdk template would be minimal and we can either host include files via public url or host internally that ship with the pdk. Allow the user to select which include templates they want via .sync.yml

include:
  - 'https://gitlab.com/puppetlabs/pdk-templates/module_root/raw/main/.prep_work.yml'
  - 'https://gitlab.com/puppetlabs/pdk-templates/module_root/raw/main/.validate.yml'
  - 'https://gitlab.com/puppetlabs/pdk-templates/module_root/raw/main/.unit'
  - 'https://gitlab.com/puppetlabs/pdk-templates/module_root/raw/main/.acceptance-litmus'
  - 'https://gitlab.com/puppetlabs/pdk-templates/module_root/raw/main/.deploy.yml'

Really like this idea as I type this comment.

@cdenneen btw the download stage downloaded a pre-bundled tar of modules. I took a puppetfile with all the modules we used, used librarian puppet to create a directory of all the files, then created a tar file. That tar file is downloaded by the puppet ci download job then cached for other jobs so that downloading all the fixtures became unnecessary for each job. This was primary done due to airgapped environment but also sped up ci jobs considerably.