Matrix multi-arch build scenario not supported

[thehedgefrog]

I have an action that does multi-architecture builds. While your action has no issues with the common dependencies, any try to install architecture-specific deps in the action results in failed builds as they can't find the packages.

Here is the current snippet:

      - name: Setup dependencies
        run: |
          sudo dpkg --add-architecture armhf
          sudo dpkg --add-architecture arm64
          echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list
          echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
          echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu jammy-security main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
          echo "deb [arch=armhf,arm64] http://ports.ubuntu.com/ubuntu-ports jammy main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
          echo "deb [arch=armhf,arm64] http://ports.ubuntu.com/ubuntu-ports jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
          echo "deb [arch=armhf,arm64] http://ports.ubuntu.com/ubuntu-ports jammy-security main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
          sudo apt-get update

      - name: Install common dependencies
        uses: awalsh128/cache-apt-pkgs-action@v1.4.2
        with:
          packages: gcc g++ cmake make libasio-dev libncurses-dev libssl-dev

      - name: Install dependencies
        run: |
          if [ "${{ matrix.arch }}" == "arm64" ]; then
            sudo apt-get install -y g++-aarch64-linux-gnu gcc-aarch64-linux-gnu libasio-dev:arm64 libncurses-dev:arm64 libssl-dev:arm64
          elif [ "${{ matrix.arch }}" == "arm" ] || [ "${{ matrix.arch }}" == "armhf" ]; then
            sudo apt-get install -y g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf libasio-dev:armhf libncurses-dev:armhf libssl-dev:armhf
          fi

Any try to leverage conditional runs of the action for arm64, arm or armhf packages run fine, but the build never succeeds.

As far as I have been able to find, this isn't related to the non-file dependencies limitation. Am I running into a different limitation?