Refactor and add test actions

[endast]

This pr waits for this pr before merging to make sure all pipelines work: https://github.com/PMBio/deeprvat/pull/92

What

This PR makes sure we have pipeline tests for all pipelines in the repo (the annotation pipeline only has as smoke test). It also adds reusable workflows for the pipeline and pytests workflows. This way we don't need to duplicate the code to run the tests. Also the pipelines run in parallel now so we speed up the time to run all the tests.

So now a new pipeline can be added like this (with both smoke and full tests:

  # Association Testing Pretrained Pipeline
  Smoke-Association-Testing-Pretrained:
    uses: ./.github/workflows/run-pipeline.yml
    with:
      pipeline_file: ./pipelines/association_testing_pretrained.snakefile
      environment_file: ./deeprvat_env_no_gpu.yml
      prerun_cmd: cd ./example && ln -s ../pretrained_models

  Pipeline-Tests-Training-Association-Testing:
    needs: Smoke-Association-Testing-Pretrained
    uses: ./.github/workflows/run-pipeline.yml
    with:
      pipeline_file: ./pipelines/association_testing_pretrained.snakefile
      environment_file: ./deeprvat_env_no_gpu.yml
      prerun_cmd: cd ./example && ln -s ../pretrained_models
      dry_run: false

Instead of this:

jobs:
  DeepRVAT-Pipeline-Smoke-Tests:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository code
        uses: actions/checkout@v4
      - uses: mamba-org/setup-micromamba@v1.8.1
        with:
          environment-name: deeprvat-gh-action
          environment-file: ${{ github.workspace }}/deeprvat_env_no_gpu.yml
          cache-environment: true
          cache-downloads: true
      - name: Link pretrained models
        run: cd ${{ github.workspace }}/example && ln -s ../pretrained_models
        shell: bash -el {0}
      - name: Smoketest association_testing_pretrained pipeline
        run: |
          python -m snakemake -n -j 2 --directory ${{ github.workspace }}/example \
          --snakefile ${{ github.workspace }}/pipelines/association_testing_pretrained.snakefile --show-failed-logs
        shell: micromamba-shell {0}
      - name: Copy seed gene discovery snakemake config
        run: cd ${{ github.workspace }}/example && cp ../deeprvat/seed_gene_discovery/config.yaml .
        shell: bash -el {0}
      - name: Smoketest seed_gene_discovery pipeline
        run: |
          python -m snakemake -n -j 2 --directory ${{ github.workspace }}/example \
          --snakefile ${{ github.workspace }}/pipelines/seed_gene_discovery.snakefile --show-failed-logs
        shell: micromamba-shell {0}

  DeepRVAT-Pipeline-Tests:
    runs-on: ubuntu-latest
    needs: DeepRVAT-Pipeline-Smoke-Tests
    steps:
      - name: Check out repository code
        uses: actions/checkout@v4
      - uses: mamba-org/setup-micromamba@v1.8.1
        with:
          environment-name: deeprvat-gh-action
          environment-file: ${{ github.workspace }}/deeprvat_env_no_gpu.yml
          cache-environment: true
          cache-downloads: true
      - name: Install DeepRVAT
        run: pip install -e ${{ github.workspace }}
        shell: micromamba-shell {0}
      # There are no GPUs on the gh worker, so we disable it in the config
      - name: Update config to use no gpus
        run: "sed -i 's/gpus: 1/gpus: 0/' ${{ github.workspace }}/example/config.yaml"
        shell: bash -el {0}
      - name: Link pretrained models
        run: cd ${{ github.workspace }}/example && ln -s ../pretrained_models
        shell: bash -el {0}
      - name: Run association_testing_pretrained pipeline
        run: |
          python -m snakemake -j 2 --directory ${{ github.workspace }}/example \
          --snakefile ${{ github.workspace }}/pipelines/association_testing_pretrained.snakefile --show-failed-logs
        shell: micromamba-shell {0}

And new code tests (pytest) can now be added like this:

  DeepRVAT-Tests-Runner-Preprocessing:
    uses: ./.github/workflows/run-pytest.yml
    with:
      environment_file: ./deeprvat_preprocessing_env.yml
      test_path: ./tests/preprocessing

Testing

All the tests should run in github actions, so proof reading the yaml files should be enough if the pipelines pass in actions.