rewrite cpp build workflows

[dstroy0]

I want to make them more maintainable, a lot of code in them is reused. I think moving the fqbn matrices to one file is a start, there needs to be one matrix for platformio fqbn and one for arduino fqbn because they're different between the platforms.

[dstroy0]

 name: build
 on: push
 jobs:
   job1:
     runs-on: ubuntu-latest
     outputs:
        matrix: ${{ steps.setmatrix.outputs.matrix }}
     steps:
     - name: checkout source code
       uses: actions/checkout@v1
     - id: setmatrix
       run: |
         matrixArray=$(find ./lambdas -name '*.js') # Creates array of all files .js withing lambdas
         # Start Generate Json String
         echo "$matrixArray" | \
         jq --slurp --raw-input 'split("\n")[:-1]' | \
         jq  "{\"filepath\": .[] }" | \
         jq -sc "{ \"include\": . }" > tmp
         cat ./tmp
         # End Generate Json String
         matrixStringifiedObject=$(cat ./tmp) # Use this as jq @sh wasn't cooperating
         echo "::set-output name=matrix::$matrixStringifiedObject"
   job2:
     needs: job1
     runs-on: ubuntu-latest
     strategy:
       matrix: ${{fromJson(needs.job1.outputs.matrix)}}
     steps:
      - name: checkout source code
        uses: actions/checkout@v1
      - run: echo ${{ matrix.filepath }}

[dstroy0]

I'm going to do it like that but I'll get the matrix from the supported boards file

[2bndy5]

echo "::set-output name=matrix::$matrixStringifiedObject"

This needs to change because github will be retiring the set-output command (starting March 2023) in favor of using an env file. Instead use:

echo "matrix=$matrixStringifiedObject" >> $GITHUB_OUTPUT

---

~I'm curious to see if it is possible to set a matrix from within the job that is supposed to use the matrix. Seems like you're putting the chicken before the egg there.~ EDIT: I read that wrong... seems fine.

[dstroy0]

I just grabbed that from SO, I need to modify it so it opens the fqbn file for pio or arduino, probably using grep so I can get them by platform out of two "supported boards" files. Then to add a platform to build coverage the fqbn can be added to those two files when the board mfr makes the board configuration available to pio and arduino.

[dstroy0]

name: Adafruit platforms (dynamic)

# Controls when the workflow will run
on:
  push:
    branches:
      - main
    paths:
      - "**/examples/all_platforms/basic/GetCommandFromStream/**"
      - "**/examples/all_platforms/advanced/GetCommandFromStream/**"
      - "**/examples/all_platforms/advanced/MultiFile/**"
      - "**/examples/all_platforms/advanced/NestedCommands/**"
      - "**/examples/all_platforms/basic/WildcardCommands/**"
      - "**/interfaces/test_output_cli/**"
  workflow_dispatch:
    branches:
      - main
    paths:
      - "**/examples/all_platforms/basic/GetCommandFromStream/**"
      - "**/examples/all_platforms/advanced/GetCommandFromStream/**"
      - "**/examples/all_platforms/advanced/MultiFile/**"
      - "**/examples/all_platforms/advanced/NestedCommands/**"
      - "**/examples/all_platforms/basic/WildcardCommands/**"
      - "**/interfaces/test_output_cli/**"
  release:
    branches:
      - release
    paths:
      - "**/examples/all_platforms/basic/GetCommandFromStream/**"
      - "**/examples/all_platforms/advanced/GetCommandFromStream/**"
      - "**/examples/all_platforms/advanced/MultiFile/**"
      - "**/examples/all_platforms/advanced/NestedCommands/**"
      - "**/examples/all_platforms/basic/WildcardCommands/**"
      - "**/interfaces/test_output_cli/**"

jobs:
  generate-job-matrices:
    runs-on: ubuntu-latest
    outputs:
      arduino-build-matrix: ${{ steps.set-arduino-matrix.outputs.arduino-build-matrix }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
      - id: set-arduino-matrix
        run: |
          echo -n "arduino-build-matrix=" >> $GITHUB_OUTPUT
          python supported_boards/generate_workflow_matrix.py -c arduino -p adafruit >> $GITHUB_OUTPUT
  output-generated-matrix:
    needs: generate-job-matrices
    runs-on: ubuntu-latest
    #strategy:
    #  matrix: ${{ fromJSON(needs.generate-job-matrices.outputs.arduino-build-matrix) }}
    steps:
      - run: |
          echo ${{ needs.generate-job-matrices.outputs.arduino-build-matrix }}

This works to make the workflow matrices, I wrote a small script to generate the boards json from the supported_boards files.

[dstroy0]

jobs:
  generate-job-matrices:
    runs-on: ubuntu-latest
    outputs:
      arduino-build-matrix: ${{ steps.set-arduino-matrix.outputs.arduino-build-matrix }}
      platformio-build-matrix: ${{ steps.set-platformio-matrix.outputs.platformio-build-matrix }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
      - id: set-arduino-matrix
        run: |
          echo -n "arduino-build-matrix=" >> $GITHUB_OUTPUT
          python supported_boards/generate_workflow_matrix.py -c arduino -p adafruit >> $GITHUB_OUTPUT
      - id: set-platformio-matrix
        run: |
          echo -n "platformio-build-matrix=" >> $GITHUB_OUTPUT
          python supported_boards/generate_workflow_matrix.py -c platformio -p adafruit >> $GITHUB_OUTPUT

  output-generated-arduino-matrix:
    needs: generate-job-matrices
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{ fromJSON(needs.generate-job-matrices.outputs.arduino-build-matrix) }}
    steps:
      - name: test
        env:
          fqbn-list: ${{ matrix.boards }}
        run: |
          echo "arduino: $fqbn-list"

  output-generated-platformio-matrix:
    needs: generate-job-matrices
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{ fromJSON(needs.generate-job-matrices.outputs.platformio-build-matrix) }}
    steps:
      - name: test
        env:
          fqbn-list: ${{ matrix.boards }}
        run: |
          echo "platformio $fqbn-list"

This spawns the correct number of jobs, fqbn-list contains one fqbn per job. Now I can write the reusable workflows.

[2bndy5]

If you're using a python script to generate values for $GITHUB_OUTPUT, then you can use the $GITHUB_OUTPUT env var directly in your python code:

import os

GITHUB_OUTPUT = os.environ.get("GITHUB_OUTPUT", None)

if GITHUB_OUTPUT is not None:  # using a CI runner
    with open(GITHUB_OUTPUT, "a", encoding="utf-8") as gh_out:
        gh_out.write(
            "{}={}\n".format(  # the '\n' is important!
                "variable_name", "variable_value"
            )
        )

Now, you don't need to pipe the script's stdout in the yml.

[dstroy0]

That's really handy!

This is working:

build-platformio:
    needs: generate-job-matrices
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{ fromJSON(needs.generate-job-matrices.outputs.platformio-build-matrix) }}        
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
      - name: Cache pip
        uses: actions/cache@v3
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
          restore-keys: ${{ runner.os }}-pip-
      - name: Cache PlatformIO
        uses: actions/cache@v3
        with:
          path: ~/.platformio
          key: ${{ runner.os }}-${{ hashFiles('**/lockfiles') }}
      - name: Install PlatformIO
        run: |
          python -m pip install --upgrade pip
          pip install --upgrade platformio    
          ls
      - name: Run PlatformIO
        run: pio ci --lib="." --board="${{matrix.boards}}"
        env:
          PLATFORMIO_CI_SRC: ${{ matrix.examples }}

Now I need to whittle down the platforms list so that we only compile once per manufacturer for each different microcontroller so it doesn't spawn more than the 256 job limit.

[dstroy0]

Holy moly I was on the struggle bus for while with this. On the plus side I got a bit better with bash string manipulation. Now I have to figure out reusable workflows.

[dstroy0]

I ran into problems with trying to have multiple examples or test_cli paths, and went off on a tangent.

[dstroy0]

Ok this is finally working, I'll be rewriting the workflows today maybe and I hopefully wont forget to remove the extra workflow triggers.

now it goes platform input->generate build sequences->build library examples and test cli->build success checkpoint

[dstroy0]

this is complete