cannot turn on release testing

[XSven]

I wanted to convince the called workflow cpan-test.yml@main to execute my release tests. According to some perl consensus release test should run if the evnvironment variable RELEASE_TESTING is true. I am unable to use the env keyword in a way that does not break my workflow. This

name: CI

on:
  push:
    branches: [ development ]
  workflow_dispatch:

jobs:
  test:
    env:
      RELEASE_TESTING: 1
    uses: PerlToolsTeam/github_workflows/.github/workflows/cpan-test.yml@main
    with:
      perl_version: "[ 'latest' ]"

doesn't work. Any help to set this up properly are very much appreciated.

[davorg]

I guess the called workflow needs a release_tests parameter that is then used to set the environment inside the workflow. I'll take a look at it.

Thanks

[XSven]

Note that other types of tests may exist too. The environment variables NONINTERACTIVE_TESTING, EXTENDED_TESTING, RELEASE_TESTING, AUTHOR_TESTING were described in the 2013 Lancaster consensus! Each variable refers to a so called "testing context". Maybe the parameter should be named testing_context and its value should be a list. Example

testing_context: "[ 'noninteractive', 'author' ]"

[davorg]

I have a first draft for this. See https://github.com/PerlToolsTeam/github_workflows/blob/main/.github/workflows/cpan-test-new.yml

However, I've just realised that it uses Bash features, so it's not going to work on Windows containers. If you have any suggestions for working around that, I'd love to hear them.

[XSven]

I will have a look into that today. I am against the use of any non-POSIX shell complient features! I usually don't care about Windows.

PS: I haven't read your GitHub Actions Essentials book yet. Question: How to write tests for GitHub Actions workflows? Which testing framework supports writing such tests?

[XSven]

I think the testing context should not be set when installing dependencies: The "Set testing contexts" step should be run after the "Install modules" step. This is my suggestion of a POSIX shell compatible "Set testing contexts" step

- name: Set testing contexts
  run: |
    printf '%s\n' '${{ inputs.testing_contexts }}' | \
    jq -r '.[]' | \
    tr '[:lower:]' '[:upper:]' | \
    while read -r item; do
      printf '%s_TESTING=1\n' "${item}" >> "${GITHUB_ENV}"
    done

The shell part was tested but the workflow step itself not. I am waiting for your feedback related to the point of framework supported GitHub Actions workflow tests.

[davorg]

I am against the use of any non-POSIX shell complient features!

I'm wondering if I can move the logic from bash into the actual workflow YAML. But that's a deliberately cut-down language.

I usually don't care about Windows.

I'm producing something that will (hopefully) be used by everyone. And that includes people who want to run their code on Windows.

How to write tests for GitHub Actions workflows? Which testing framework supports writing such tests?

I don't know of any testing frameworks for GitHub Actions. It's all currently boringly manual.

I think the testing context should not be set when installing dependencies: The "Set testing contexts" step should be run after the "Install modules" step.

Ok, That makes sense.

This is my suggestion of a POSIX shell compatible "Set testing contexts" step

I don't think making it POSIX compliant really gains anything. What I already have runs successfully on the GitHub Actions runners. Oh, unless PowerShell is POSIX compliant. I'll investigate further.

[XSven]

implemented for non-windows OS in https://github.com/PerlToolsTeam/github_workflows/pull/11