Licensing fails inconsistently during test step

Bug description

Occasionally, test runs will fail due to a supposedly inactive license, despite that license properly being activated in a previous step. For example, on a recent run, the license initially activated fine at the very start of the test step of the job:

Requesting activation
Unity Editor version:    2022.3.10f1 (ff3792e53c62)
Branch:                  2022.3/staging
Build type:              Release
Batch mode:              YES
System name:             Linux
Distro version:          #66~20.04.2-Ubuntu SMP Wed Feb 28 12:36:21 UTC 2024
Kernel version:          5.15.0-[105](https://github.com/chris-janidlo/redacted-game-name/actions/runs/8396020107/job/22996520370#step:5:106)8-azure
Architecture:            x86_64
Available memory:        7940 MB
CreateDirectory '/root/.cache/unity3d' failed:  (current dir: /github/workspace)
User *** logged in successfully
[Licensing::Module] Trying to connect to existing licensing client channel...
[Licensing::IpcConnector] Connection attempt to the License Client on channel: "LicenseClient-root" failed because channel doesn't exist; code: "0x80000002"
[Licensing::Module] Successfully launched the LicensingClient (PId: 32)
[Licensing::IpcConnector] Successfully connected to the License Client on channel: "LicenseClient-root" at "2024-03-22T20:29:42.855163Z"
[SignatureVerifier] Application signature verification not supported on this platform.
[Licensing::Client] Handshaking with LicensingClient:
  Version:                 1.13.1+ca5f299
  Session Id:              1827b69d172c441fb45709d1a36f853a
  Correlation Id:          c489e40c0f286181eb50439423fe2633
  External correlation Id: 8884835463932669826
  Machine Id:              D7nTUnjNAmtsUMcnoyrqkgIbYdM=
[Licensing::Module] Successfully connected to LicensingClient on channel: "LicenseClient-root" (connect: 0.67s, validation: 0.11s, handshake: 0.00s)
[Licensing::IpcConnector] Successfully connected to the License Notification on channel: "LicenseClient-root-notifications" at "2024-03-22T20:29:42.966237Z"
[Licensing::Module] Connected to LicensingClient (PId: 32, launch time: 0.00, total connection time: 0.78s)
[Licensing::Client] Successfully updated the access token
[Licensing::Module] Successfully updated the access token krjlMrWbFX...
[Licensing::Client] Successfully processed license management request
[Licensing::Module] Successfully activated the entitlement license
[Licensing::Client] Successfully queried for the EntitlementGroupsDetails
[Licensing::Client] Successfully activated ULF license
[Licensing::Module] Serial number assigned to (masked): "***"
[Licensing::Client] Successfully resolved entitlements
[Licensing::Module] Serial number assigned to: "***"
Pro License: YES

But then when the runner actually gets to running tests, license activation fails. Here's the entire block of log output from Testing in playmode onward:

###########################
#   Testing in playmode  #
###########################

xvfb-run: error: problem while cleaning up temporary directory
Unity Editor version:    2022.3.10f1 (ff3792e53c62)
Branch:                  2022.3/staging
Build type:              Release
Batch mode:              YES
System name:             Linux
Distro version:          #66~20.04.2-Ubuntu SMP Wed Feb 28 12:36:21 UTC 2024
Kernel version:          5.15.0-1058-azure
Architecture:            x86_64
Available memory:        7940 MB
[Licensing::Module] Trying to connect to existing licensing client channel...
[Licensing::IpcConnector] Connection attempt to the License Client on channel: "LicenseClient-root" failed because channel doesn't exist; code: "0x80000002"
[Licensing::Module] Successfully launched the LicensingClient (PId: 1226)
[Licensing::IpcConnector] Successfully connected to the License Client on channel: "LicenseClient-root" at "2024-03-22T20:30:29.057961Z"
[SignatureVerifier] Application signature verification not supported on this platform.
[Licensing::Client] Handshaking with LicensingClient:
  Version:                 1.13.1+ca5f299
  Session Id:              d72c738b57ae4050800d3c9b8ce8c98f
  Correlation Id:          0c00f06b418e458aa49c4118f30b02da
  External correlation Id: 7646962551635602403
  Machine Id:              D7nTUnjNAmtsUMcnoyrqkgIbYdM=
[Licensing::Module] Successfully connected to LicensingClient on channel: "LicenseClient-root" (connect: 0.60s, validation: 0.12s, handshake: 0.00s)
[Licensing::IpcConnector] Successfully connected to the License Notification on channel: "LicenseClient-root-notifications" at "2024-03-22T20:30:29.174676Z"
[Licensing::Module] Connected to LicensingClient (PId: 1226, launch time: 0.00, total connection time: 0.72s)
[Licensing::Module] Error: Access token is unavailable; failed to update
[Licensing::Client] Successfully updated license
[Licensing::Client] Successfully resolved entitlements
[Licensing::Module] Error: License is not active (com.unity.editor.headless). HasEntitlements will fail.
Pro License: NO
Desktop is [1280](https://github.com/chris-janidlo/redacted-game-name/actions/runs/8396020107/job/22996520370#step:5:1281) x 1024 @ 60 Hz
Initialize udev device monitor.
No valid Unity Editor license found. Please activate your license.
Checking for leaked weakptr:
  Found no leaked weakptrs.
Memory Statistics:
[ALLOC_TEMP_TLS] TLS Allocator
  StackAllocators : 
    [ALLOC_TEMP_CoreBusinessMetricsCache]
      Initial Block Size 64.0 KB
      Current Block Size 64.0 KB
      Peak Allocated Bytes 0 B
      Overflow Count 0
    [ALLOC_TEMP_Profiler.Dispatcher]
      Initial Block Size 64.0 KB
      Current Block Size 64.0 KB
      Peak Allocated Bytes 0 B
      Overflow Count 0
    [ALLOC_TEMP_AssetGarbageCollectorHelper]
      Initial Block Size 64.0 KB
      Current Block Size 64.0 KB
      Peak Allocated Bytes 0 B
      Overflow Count 0
[ALLOC_MEMORYPROFILER]
  Requested Block Size 1.0 MB
  Peak Block count 1
  Peak Allocated memory 4.6 KB
  Peak Large allocation bytes 0 B
##utp:{"type":"MemoryLeaks","version":2,"phase":"Immediate","time":1711139430308,"processId":1222,"allocatedMemory":91743,"memoryLabels":[{"Default":730},{"Thread":64},{"Manager":6810},{"GfxDevice":408},{"Serialization":52},{"BaseObject":12376},{"String":5603},{"DynamicArray":640},{"HashMap":17034},{"Utility":1024},{"GI":2920},{"VR":80},{"EditorUtility":2872},{"RestService":1022},{"License":3608},{"UnityConnect":25432},{"Collab":25},{"LocalIPC":212},{"ProfilerEditor":10831}]}
Unexpected exit code 5

How to reproduce

Happens inconsistently, but frequently enough to be a nuisance - maybe around 30% of the time. I originally thought it was due to running multiple test runs in parallel, as per this Discord discussion, but after trying to fix that I still see the error.

Expected behavior

Test runs should not fail due to an "inactive license" when given a valid license.

Additional details

My workflow file:

on:
  push:
    branches:
      - main
      - bug/**
      - feature/**
  pull_request:
  workflow_dispatch:
    inputs:
      branch:
        description: 'Build from branch:'
        type: string
        default: main
      deploy:
        description: 'After build, deploy to:'
        type: choice
        options:
          - <nowhere>
          - live
          - beta

env:
  UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
  UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
  UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}

jobs:
  metadata:
    runs-on: ubuntu-20.04
    steps:
      - name: Log Inputs
        run: echo "${{ toJson(github.event.inputs) }}"

  test:
    runs-on: ubuntu-20.04

    steps:
      - name: Free Disk Space
        uses: jlumbroso/free-disk-space@v1.3.0

      - name: Checkout code
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.inputs.branch }}
          lfs: true
          submodules: recursive

      - name: Cache
        uses: actions/cache@v3
        with:
          path: Library
          key: Library-redacted-game-name-

      - name: Run tests
        uses: game-ci/unity-test-runner@v4
        with:
          testMode: PlayMode

      - name: Upload test report
        if: ${{ success() || failure() }}
        uses: actions/upload-artifact@v3
        with:
          name: test-report
          path: artifacts

  build:
    runs-on: ubuntu-20.04

    strategy:
      fail-fast: false
      matrix:
        targetPlatform:
          - WebGL
          - StandaloneWindows
          - Android

    steps:
      - name: Free Disk Space
        uses: jlumbroso/free-disk-space@v1.3.0

      - name: Checkout code
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.inputs.branch }}
          lfs: true
          submodules: recursive

      - name: Cache
        uses: actions/cache@v3
        with:
          path: Library
          key: Library-redacted-game-name-${{ matrix.targetPlatform }}
          restore-keys: |
            Library-redacted-game-name-

      - name: Build
        uses: game-ci/unity-builder@v3
        with:
          targetPlatform: ${{ matrix.targetPlatform }}
          buildName: redacted-game-name-${{ matrix.targetPlatform }}

      - name: Group artifacts into a folder
        if: ${{ matrix.targetPlatform != 'WebGL' }} # webgl artifacts are already grouped
        run: |
          mkdir tmp
          sudo mv build/${{ matrix.targetPlatform }}/* tmp/
          sudo mkdir build/${{ matrix.targetPlatform }}/redacted-game-name-${{ matrix.targetPlatform }}
          sudo mv tmp/* build/${{ matrix.targetPlatform }}/redacted-game-name-${{ matrix.targetPlatform }}

      - name: Upload build artifact to github cache
        uses: actions/upload-artifact@v3
        with:
          name: redacted-game-name-${{ matrix.targetPlatform }}
          path: build/${{ matrix.targetPlatform }}
          retention-days: 1

  release:
    needs: [test, build]
    if: ${{ github.ref == 'refs/heads/main' || (inputs.deploy && inputs.deploy != '<nowhere>') }}
    runs-on: ubuntu-20.04
    env:
      # only deploy to live if specifically requested
      ITCH_GAME: ${{ inputs.deploy == 'live' && 'redacted-game-name' || 'redacted-game-name-secret-version' }}
      ITCH_USER: crass-sandwich
      BUTLER_CREDENTIALS: ${{ secrets.BUTLER_CREDENTIALS }}

    strategy:
      fail-fast: true
      matrix:
        include:
          - builder-name: WebGL
            itch-name: webgl
          - builder-name: StandaloneWindows
            itch-name: win
          - builder-name: Android
            itch-name: android

    # only allow one release per itch game per platform to run at a time
    # itch game has to be duplicated here because concurrency doesn't have access to env context
    concurrency: release-${{ matrix.itch-name }}-${{ inputs.deploy == 'live' && 'redacted-game-name' || 'redacted-game-name-secret-version' }}

    steps:
      - name: Download build artifact
        uses: actions/download-artifact@v3
        with:
          name: redacted-game-name-${{ matrix.builder-name }}

      - name: Push to itch.io
        uses: josephbmanley/butler-publish-itchio-action@v1.0.3
        env:
          PACKAGE: redacted-game-name-${{ matrix.builder-name }}
          CHANNEL: ${{ matrix.itch-name }}

game-ci / unity-test-runner

Licensing fails inconsistently during test step #265