Gitly CI/CD Considerations

[shiipou]

Gitly CI/CD design

Hi, I'm here to talk about CI/CD integration for Gitly.

I use daily CI/CD on both GitLab and GitHub, and I see the power of the both technology.

GitLab has highly customisable and powerful pipelines. It also has parallels runs.

GitHub as simple workflow and highly community oriented workflow.

So I try to keep the powerful GitLab pipelines with the comminuty based workflow of GitHub Actions.

I think It's powerful enough to do anything.

Here my proposal of a YAML for Gitly CI/CD file.

# filename -> .gitly/flow/

# Name of the flow pipeline
name: Build MyApp

# Define the differents steps of the pipeline flow.
## Steps are launched in the order of the list.
## Names can be customized by developer.
steps:
  - prepare
  - build
  - test
  - publish

# Define each jobs run by the flow
jobs:
  # Set the id of the job to be selected after.
  docker-prepare:
    # Set the display name of the jobs. (default will be the id)
    name: Prepare the Docker environment
    # Set the contraints of the job
    on:
      # Link this job into the `prepare` steps.
      ## This will display it in the first column of the flow view. (because prepare is the first)
      steps:
      - prepare
      # run this job only on push event.
      events:
      - push
      # Run this jobs only on event triggered on `main` and `staging` branch...
      branches:
      - main
      - staging
      # ...or Run it on tags like `v1.0`
      tags:
      - v(.*)
    # Define all actions that job will run.
    actions:
      # Get the source code of the repository.
      - name: Checkout
         ## This can be the same like this : gitly.org/gitly/flow/checkout@master
         ## We can use @<branch_name> or @<tag_name>. 
         ## Any @<tag_name> must be cached to not recompile it at any use.
         ## So using @<tag_name> will speedup your flow.
         use: checkout@v1
         args:
           # (Optional)
           ref: ${gitly.event.ref}
      # define another action that run a user custom flow by selecting his public repository. (Can be on other git server)
      - name: Get environment docker image
         use: gitly.org/newuser/myflow-pull.git@master
      # define another action that run a user custom flow by selecting his public repository. (Can be on other git server)
      - name: Publish Environment Docker image
         use: gitly.org/newuser/myflow-push.git@master
         args:
           repo: ${gitly.server.docker_repository}
           tag: ${gitly.repository.url}/android:${gitly.event.branch-or-tag}
       - name: Display image size
         # This `use` is optional, it allow to choose a prefered runner and/or a docker image.
         use: ubuntu
         scripts:
         # Ouput is defined by the flow repository (Here by gitly.org/newuser/myflow-push.git@master)
         - echo "${gitly.jobs.docker-prepare.output.docker_image_size}"

  # Run another job (Other job will not contain the same folder, but can get it from )
  display-env:
    on:
      steps:
      - prepare
      # Without this, jobs are in parallels. You can access to output of jobs only if his id are listed here.
      after:
      - docker-prepare
    actions:
      - name: Display android sdkmanager version
         # This use docker image
         use: ${gitly.server.docker_repository}/${gitly.repository.url}
         scripts:
         - sdkmanager --version

  display-env:
    on:
      steps:
      - prepare
    actions:
      - name: Display flutter version
         # This use docker image
         use: ${gitly.server.docker_repository}/${gitly.repository.url}
         scripts:
         - flutter --version

  build_job:
    # Link to docker-prepare to get output
    ## But I didn't define any steps, so it will not be runned.
    ## That allow to use it as template and shortcut other actions or jobs.
    on:
      after:
      - docker-prepare
    actions:
    - id: build-setup
      use: ${gitly.server.docker_repository}/${gitly.repository.url}
      before_script:
      - echo "${gitly.secrets.signing_jks_file_hex}" | xxd -r -p - > android-signing-keystore.jks
      - "export VERSION_CODE=${gitly.flow.id} && echo $VERSION_CODE"
      - "export VERSION_SHA=`echo ${gitly.event.commit_sha:0:8}` && echo $VERSION_SHA"
      - cp ./fastlane/metadata/android/en-GB/changelogs/CURRENT_VERSION.txt "./fastlane/metadata/android/en-GB/changelogs/$VERSION_CODE.txt"
      - chmod +x ./fastlane/create-changelog-mr.sh
      - ./fastlane/create-changelog-mr.sh || true
      - git checkout "${gitly.event.commit_sha}"
      after_script:
      - rm -f android-signing-keystore.jks || true
    - id: upload-artifact
      name: Upload Artifact
      use: set_artifact@v1
      args:
        id: app_bundle
        name: App Bundle
        paths: app/build/outputs

# Run job in build steps
  build-debug:
    on:
      steps:
      - build
    actions:
      - name: Checkout
         use: checkout@v1
      # This extends will keep everything in the actions build_job.build-setup (scripts, before-scripts, after-scripts, env, ...) 
      ## But you can append those elements here.
      ## If you fix an id on the action, you can replace the action by re-set it here with the fixed id too.
      - extends: build_job.build-setup
         name: Build debug
         # This use docker image
         use: ${gitly.server.docker_repository}/${gitly.repository.url}
         scripts:
         - bundle exec fastlane buildDebug

  build-release:
    on:
      steps:
      - build
    actions:
      - name: Checkout
         use: checkout@v1
      # Like you can see, when you use a docker image, the current workspace will be copied into the image.
      ## So if you use checkout here, the docker image will get the repository.
      - extends: build_job.build-setup
         name: Build release
         use: ${gitly.server.docker_repository}/${gitly.repository.url}
         scripts:
         - bundle exec fastlane buildRelease

  test-debug:
    on:
      after: 
      - build-debug
      steps:
      - test
    actions:
      - name: Checkout
         use: checkout@v1
      - name: Test Debug
         use: ${gitly.server.docker_repository}/${gitly.repository.url}
         scripts:
         - bundle exec fastlane testDebug

  test-release:
    on:
      after: 
      - build-release
      steps:
      - test
    actions:
      - name: Checkout
         use: checkout@v1
      - name: Test Release
         use: ${gitly.server.docker_repository}/${gitly.repository.url}
         scripts:
         - bundle exec fastlane testDebug

  publish-internal:
    on:
      after: 
      # Here it will wait for test to finish, because it run on steps publish, and all `publish` steps will always be run after `test` steps
      - build-debug
      steps:
      - publish
    actions:
      - name: Downlaod Artifact
         use: get_artifact@v1
         args:
           artifact: ${gitly.jobs.build-release.output.app_bundle.artifact_url}
           path: app/build/outputs
      - name: Publish to Playstore in Internal
         use: ${gitly.server.docker_repository}/${gitly.repository.url}
         scripts:
         - bundle exec fastlane internal

It allow to get a Flow like the GitLab Pipelines with the power of Actions like GitHub Action.

[muuvmuuv]

If this is taken seriously caching should be mandatory and prio #1. GitLab is very bad in caching and GitHub makes it kinda ez. Would love to see some MinIO (or V native Object Storage) or Redis built-in solution. No file caching for security and enhanced methods to do dind builds and caching artifacts.

Especially with projects having node_modules or Angular prjects (which stores compiled files in node_modules) caching is quite a pain to setup and use properly.

[Cons-Cat]

Does CI need to be a YAML document? I know it's fairly standard in other forges, but I always struggle to follow what's happening in these documents and the syntax seems needlessly complicated to me. We already have v run and $tmpl files. I feel like it should be possible to express CI pipeline scripts purely with V.

[medvednikov]

You're 100% right @Cons-Cat.

Gitly will be using V scripts for CI.

YAML is terrible.

[muuvmuuv]

@medvednikov YAML is kind of cool because it is ez to read and write, but missing schema and weird alias/extending syntax makes it hard for this type of target solution.

I think TOML would be a good choice, or even JSON (but that even harder to read); both have schema support.

[JalonSolov]

@medvednikov YAML is kind of cool because it is ez to read and write, but missing schema and weird alias/extending syntax makes it hard for this type of target solution.

YAML is "ez" to read and write... and just as "ez" to get wrong.

[shiipou]

For now, really quick way to get CI/CD is to allow to edit Git hooks of the remote repo. I think that can be done easily and it will allow most part of our needs.

With that we can run V scripts or use the act (or other CI/CD tools) for people who want it.