Speedup CI run to execute only necessary steps

[Garados007]

Currently we are executing all steps even if they are not required because their data haven't been changed. For example if I update the documentation I don't have to spin up all Igor Pro instances to test the UTF code.

We can optimize it to use a Gitlab feature to run a step only if some files in certain directories have been changed (see StackOverflow-Post).

[t-b]

That is tricky to get right. The current solution uses rules 1, the predecessor only did not work that well when we tried it.

Setting up a local environment is a bit of work. See https://gitlab.byte-physics.de:4001/internal/docker-sphinx/-/blob/main/Dockerfile#L31 for the list of python packages. But unbeatable fast ;)

An alternative to a local setup is to get the docker image we are using for CI running locally. Nobody stepped up to try that so far ;)

[Garados007]

I don't want to change something on the docker scripts or run the CI locally.

I have used the only keyword in my CI this year a lot and it worked pretty well. But Gitlab seems to have deprecated it in favor of rules.

What I want is to extend the existing rules to start the step only if certain files have been changed. For example:

• Start the Igor Pro instances only for changed files in:
  • procedures/**/*
  • tests/**/*
  • xop/**/*
• Documentation only for changed files in:
  • docu/**/*
  • procedures/**/*

This won't run for example Igor Pro if I change something in the user documentation.

This change will shorten the pipeline runtime and safe expensive energy. ;)

[t-b]

@Garados007 Fine, git it a shot.

[Garados007]

Okay, I got it somewhat working. My first intention was that it will check for changes between the last and current run and then decide what to do. This is not easy to implement, especially because for this is a Merge Request needed which doesn't exists in our setup.

With the new setup it will check the changes between the current commit and main and decide for this range what to do. This is most times a larger range but in some cases it can reduce the amount of steps that need to be done.

Some examples:

• No changes to UTF itself and docs: https://gitlab.byte-physics.de:4001/mirror/igor-unit-testing-framework/-/pipelines/7109
• Some changes in /docu: https://gitlab.byte-physics.de:4001/mirror/igor-unit-testing-framework/-/pipelines/7114
• Some changes in /tests: https://gitlab.byte-physics.de:4001/mirror/igor-unit-testing-framework/-/pipelines/7115
• Some changes in /procedures: https://gitlab.byte-physics.de:4001/mirror/igor-unit-testing-framework/-/pipelines/7116

Most time it will be like the last case. But sometimes we can improve the runtime.

[Garados007]

If you like the results I will cleanup and rebase the PR and make it ready for review.

[t-b]

Hmppf.

• Is it possible to rerurn the same pipeline again? Dpes it work the same way again?
• No shortcuts (rules/if/changes) when we are on the main branch please. We always want to be able to rerun a pipeline and it should run completely.
• I don't like that the deployment job is touched. In the current setup it will also not deploy IIUC, see https://gitlab.byte-physics.de:4001/mirror/igor-unit-testing-framework/-/ci/editor?branch_name=main&tab=2.
• Do we have to duplicate the statements? Why not do

  rules:
  - if:
    changes:
      compare_to: main
      paths:
        - procedures/**/*
        - tests/**/*
        - xop/**/*
  - if: '$ARCHITECTURE == "x64"'
    variables:
      IGORTARGET: ""
      IGOREXTPATH: "Igor Extensions (64-bit)"
  - if: '$ARCHITECTURE == "x86"'
    variables:
      IGORTARGET: "--32bit"
      IGOREXTPATH: "Igor Extensions"

?

• If you depend in xml_verification optionally on documentation, you also don't need the if/changes in xml_verification.