Running github actions locally & retrieving results.

mdeweerd commented 3 years ago

I am examinging the CI/CD workflow but I have to run the actions locally and there is a tool for that: act.

That essentially works, but:

the files from the artifact server are not readily available,
the files do not seem to be available "locally" (maybe I am not generating any result at this moment)
INTI-CMNB/KiBot@v1 is cloned on every call, which seems to stall/take a long time when done repeatedly (maybe github is throthling this). could this not be in the image?

I installed the latest version which allows running an artifact server locally. I also add -b to use the local path rather than copying the files, hoping that I can get some result files.

act -b --artifact-server-path e:\help

I added an 'env' variable for the project name, this is ok according to the output. This prepares the path to making this a file to include from anohter yaml.

name: "KiCad CI/CD"

env:
  PROJECT: TEST

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
  push:
    paths:
      - '*.sch'
      - '*.kicad_pcb'
      - '*.kibot.yaml'
      - .github/workflows/${{ env.PROJECT }}.yml
  pull_request:
    paths:
      - '*.sch'
      - '*.kicad_pcb'
      - '*.kibot.yaml'
      - .github/workflows/${{ env.PROJECT }}.yml

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  Fabrication:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: INTI-CMNB/KiBot@v1
      with:
        # Required - kibot config file
        config: ${{ env.PROJECT }}.yaml
        # optional - prefix to output defined in config
        dir: ${{ env.PROJECT }}.yaml
        # optional - schematic file
        schema: '${{ env.PROJECT }}.sch'
        # optional - PCB design file
        board: '${{ env.PROJECT }}.kicad_pcb'
    - name: Retrieve results
      if: ${{ always() }}
      uses: actions/upload-artifact@v2
      with:
        name: ${{ env.PROJECT }}
        path: |
          ${{ env.PROJECT }}
        if-no-files-found: warn

I have to determine why I do not get any result yet.

Act is usefull when uploads to github/gitlab are not available options or to do the flow locally for other reasons.

mdeweerd commented 3 years ago

I added drc checking to the configuration file and I got the log further below. I wonder if [KiCad CI/CD/Fabrication] | > FileExistsError: [Errno 17] File exists: '/mnt/d/mdeweerd/KiCAD/TEST/TEST.yaml' is causign the failure.

I am going to revert running docker directly as per my by calling kibot with the yaml configuration file as parameter.

[KiCad CI/CD/Fabrication] 🚀  Start image=node:12-buster-slim
[KiCad CI/CD/Fabrication]   🐳  docker pull image=node:12-buster-slim platform= username= forcePull=false
[KiCad CI/CD/Fabrication]   🐳  docker create image=node:12-buster-slim platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[KiCad CI/CD/Fabrication]   🐳  docker run image=node:12-buster-slim platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[KiCad CI/CD/Fabrication]   🐳  docker exec cmd=[mkdir -m 0777 -p /var/run/act] user=root workdir=
[KiCad CI/CD/Fabrication] ⭐  Run actions/checkout@v2
[KiCad CI/CD/Fabrication]   ✅  Success - actions/checkout@v2
[KiCad CI/CD/Fabrication] ⭐  Run INTI-CMNB/KiBot@v1
[KiCad CI/CD/Fabrication]   🐳  docker pull image=act-inti-cmnb-kibot-v1-dockeraction:latest platform= username= forcePull=false
[KiCad CI/CD/Fabrication]   🐳  docker create image=act-inti-cmnb-kibot-v1-dockeraction:latest platform= entrypoint=[] cmd=["-c TEST.yaml" "-d TEST.yaml" "-b TEST.kicad_pcb" "-e TEST.sch" "-v 2"]
[KiCad CI/CD/Fabrication]   🐳  docker run image=act-inti-cmnb-kibot-v1-dockeraction:latest platform= entrypoint=[] cmd=["-c TEST.yaml" "-d TEST.yaml" "-b TEST.kicad_pcb" "-e TEST.sch" "-v 2"]
[KiCad CI/CD/Fabrication]   | DEBUG:Detected KiCad v5.1.9 (5.1.9+dfsg1-1~bpo10+1 5001009) (kibot - __main__.py:238)
[KiCad CI/CD/Fabrication]   | DEBUG:KiCad config path /root/.config/kicad (kibot - __main__.py:273)
[KiCad CI/CD/Fabrication]   | DEBUG:Importing from /usr/lib/python3/dist-packages/kibot (kibot.kibot.kiplot - kiplot.py:53)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_pdf_pcb_print (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_step (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_pcbdraw (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_ps (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_kicost (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_gerb_drill (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_compress (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_dxf (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_position (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_sch_variant (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_hpgl (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_kibom (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_pdf (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_any_drill (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_svg (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_any_layer (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_pdf_sch_print (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_base (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_excellon (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_boardview (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_svg_sch_print (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_ibom (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_bom (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing out_gerber (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing pre_update_xml (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing pre_drc (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing pre_ignore_unconnected (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing pre_erc (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing pre_base (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing pre_filters (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing pre_erc_warnings (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing pre_check_zone_fills (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing var_kicost (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing var_kibom (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing var_base (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing var_ibom (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing fil_rot_footprint (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing fil_subparts (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing fil_var_rename (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing fil_var_rename_kicost (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing fil_base (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing fil_field_rename (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing fil_generic (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:- Importing globals (kibot.kibot.kiplot - kiplot.py:60)
[KiCad CI/CD/Fabrication]   | DEBUG:Deactivating macros (kibot.kibot.kiplot - kiplot.py:83)
[KiCad CI/CD/Fabrication]   | DEBUG:Using configuration file: `TEST.yaml` (kibot - __main__.py:164)
[KiCad CI/CD/Fabrication]   | DEBUG:Parsing preflight options: {'check_zone_fills': True, 'erc_warnings': True, 'ignore_unconnected': False, 'run_drc': True, 'run_erc': True} (kibot.kibot.config_reader - config_reader.py:147)
[KiCad CI/CD/Fabrication]   | DEBUG:Parsing preflight check_zone_fills (kibot.kibot.config_reader - config_reader.py:155)
[KiCad CI/CD/Fabrication]   | DEBUG:Parsing preflight erc_warnings (kibot.kibot.config_reader - config_reader.py:155)
[KiCad CI/CD/Fabrication]   | DEBUG:Parsing preflight ignore_unconnected (kibot.kibot.config_reader - config_reader.py:155)
[KiCad CI/CD/Fabrication]   | DEBUG:Parsing preflight run_drc (kibot.kibot.config_reader - config_reader.py:155)
[KiCad CI/CD/Fabrication]   | DEBUG:Parsing preflight run_erc (kibot.kibot.config_reader - config_reader.py:155)
[KiCad CI/CD/Fabrication]   | DEBUG:Using schematic: `/mnt/d/mdeweerd/KiCAD/TEST/TEST.sch` (kibot - __main__.py:141)
[KiCad CI/CD/Fabrication]   | DEBUG:Using PCB: `TEST.kicad_pcb` (kibot - __main__.py:191)
[KiCad CI/CD/Fabrication]   | DEBUG:Starting outputs for board /mnt/d/mdeweerd/KiCAD/TEST/TEST.kicad_pcb (kibot.kibot.kiplot - kiplot.py:348)
[KiCad CI/CD/Fabrication]   | DEBUG:Preflight checks (kibot.kibot.kiplot - kiplot.py:285)
[KiCad CI/CD/Fabrication]   | DEBUG:Preflight apply check_zone_fills (kibot.kibot.pre_base - pre_base.py:54)
[KiCad CI/CD/Fabrication]   | DEBUG:Preflight apply erc_warnings (kibot.kibot.pre_base - pre_base.py:54)
[KiCad CI/CD/Fabrication]   | DEBUG:Preflight apply run_drc (kibot.kibot.pre_base - pre_base.py:54)
[KiCad CI/CD/Fabrication]   | DEBUG:Preflight apply run_erc (kibot.kibot.pre_base - pre_base.py:54)
[KiCad CI/CD/Fabrication]   | DEBUG:Preflight run check_zone_fills (kibot.kibot.pre_base - pre_base.py:58)
[KiCad CI/CD/Fabrication]   | DEBUG:Preflight run erc_warnings (kibot.kibot.pre_base - pre_base.py:58)
[KiCad CI/CD/Fabrication]   | DEBUG:Preflight run run_drc (kibot.kibot.pre_base - pre_base.py:58)
[KiCad CI/CD/Fabrication]   | DEBUG:Running: ['pcbnew_do', '--version'] (kibot.kibot.kiplot - kiplot.py:92)
[KiCad CI/CD/Fabrication]   | DEBUG:Board loaded (kibot.kibot.kiplot - kiplot.py:217)
[KiCad CI/CD/Fabrication]   | DEBUG:PCB title: `STAR_LED` (kibot.kibot.gs - gs.py:127)
[KiCad CI/CD/Fabrication]   | DEBUG:PCB date: `2019-11-08` (kibot.kibot.gs - gs.py:128)
[KiCad CI/CD/Fabrication]   | DEBUG:PCB revision: `02` (kibot.kibot.gs - gs.py:129)
[KiCad CI/CD/Fabrication]   | DEBUG:PCB company: `AVADY POOL sas - 9 Chaussée Jules César - Bâtiment 4 - Hall 406 - 95520 OSNY - France` (kibot.kibot.gs - gs.py:130)
[KiCad CI/CD/Fabrication]   | DEBUG:DRC report: /mnt/d/mdeweerd/KiCAD/TEST/TEST.yaml/TEST-drc.txt (kibot.kibot.pre_drc - pre_drc.py:42)
[KiCad CI/CD/Fabrication]   | - Running the DRC
[KiCad CI/CD/Fabrication]   | DEBUG:Executing: ['pcbnew_do', '-r', '-vv', 'run_drc', '-o', '/mnt/d/mdeweerd/KiCAD/TEST/TEST.yaml/TEST-drc.txt', '/mnt/d/mdeweerd/KiCAD/TEST/TEST.kicad_pcb', '/mnt/d/mdeweerd/KiCAD/TEST/TEST.yaml'] (kibot.kibot.kiplot - kiplot.py:164)
[KiCad CI/CD/Fabrication]   | DEBUG:Failed with error 1, retrying ... (kibot.kibot.kiplot - kiplot.py:171)
[KiCad CI/CD/Fabrication]   | WARNING:(W058) Missing KiCad main config file /root/.config/kicad/kicad_common (pcbnew_do - misc.py:161) (kibot.kibot.kiplot - log.py:82)
[KiCad CI/CD/Fabrication]   | DEBUG:Output from command:
[KiCad CI/CD/Fabrication]   | > DEBUG:Detected KiCad v5.1.9 (5.1.9+dfsg1-1~bpo10+1 5001009) (pcbnew_do - misc.py:129)
[KiCad CI/CD/Fabrication]   | > DEBUG:Config path /root/.config/kicad (pcbnew_do - misc.py:142)
[KiCad CI/CD/Fabrication]   | > WARNING:Missing KiCad main config file /root/.config/kicad/kicad_common (pcbnew_do - misc.py:161)
[KiCad CI/CD/Fabrication]   | > DEBUG:Select a filename -> Select a filename (pcbnew_do - pcbnew_do:619)
[KiCad CI/CD/Fabrication]   | > DEBUG:Print -> Print (pcbnew_do - pcbnew_do:620)
[KiCad CI/CD/Fabrication]   | > DEBUG:Current pcb (/mnt/d/mdeweerd/KiCAD/TEST/TEST.kicad_pcb) size: 422478 date: Thu Nov 11 13:04:46 2021 (pcbnew_do - pcbnew_do:447)
[KiCad CI/CD/Fabrication]   | > DEBUG:PCBnew config: /root/.config/kicad/pcbnew (pcbnew_do.kiauto.file_util - file_util.py:168)
[KiCad CI/CD/Fabrication]   | > DEBUG:Moving current config to /root/.config/kicad/pcbnew.pre_script (pcbnew_do.kiauto.file_util - file_util.py:175)
[KiCad CI/CD/Fabrication]   | > Traceback (most recent call last):
[KiCad CI/CD/Fabrication]   | >   File "/usr/bin/pcbnew_do", line 681, in <module>
[KiCad CI/CD/Fabrication]   | >     os.makedirs(output_dir, exist_ok=True)
[KiCad CI/CD/Fabrication]   | >   File "/usr/lib/python3.7/os.py", line 221, in makedirs
[KiCad CI/CD/Fabrication]   | >     mkdir(name, mode)
[KiCad CI/CD/Fabrication]   | > FileExistsError: [Errno 17] File exists: '/mnt/d/mdeweerd/KiCAD/TEST/TEST.yaml'
[KiCad CI/CD/Fabrication]   | > DEBUG:Restoring old pcbnew config (pcbnew_do.kiauto.file_util - file_util.py:152)
[KiCad CI/CD/Fabrication]   | > DEBUG:Checking if /mnt/d/mdeweerd/KiCAD/TEST/TEST.pro was modified (pcbnew_do.kiauto.file_util - file_util.py:242)
[KiCad CI/CD/Fabrication]   | >  (kibot.kibot.kiplot - kiplot.py:175)
[KiCad CI/CD/Fabrication]   | ERROR:DRC returned 1 (kibot.kibot.pre_drc - pre_drc.py:63)
[KiCad CI/CD/Fabrication]   ❌  Failure - INTI-CMNB/KiBot@v1

mdeweerd commented 3 years ago

Running

kibot -c TEST.yaml -b TEST.kicad_pcb -e TEST.sch -d TEST -v 0

where kibot runs in the docker container gives better results ;-). That's good enough for me.

set-soft commented 3 years ago

INTI-CMNB/KiBot@v1 is cloned on every call, which seems to stall/take a long time when done repeatedly (maybe github is throthling this). could this not be in the image?

You don't need to do that, just use the setsoft/kicad_auto:latest docker image.

Using the GitHub action is good if you are running the workflow at GitHub servers. This is efficient because is easier to create a new docker image than downloading one from Docker Hub. But if you run it locally is a real waste of resources because docker keeps a copy of downloaded images and you don't need to download anything. GitHub actions are for GitHub, and only if you like using them. The examples referenced in the README.md doesn't use it (https://github.com/INTI-CMNB/kicad_ci_test/blob/master/.github/workflows/test1.yml)

About the error: You are setting the output dir as /mnt/d/mdeweerd/KiCAD/TEST/TEST.yaml, which is a file and which already exist. This generates the problem.

mdeweerd commented 3 years ago

Thanks for your feedback - I've tried many ways to run KiBot over the few past days.

I started from the example to run the CI/CD.

Good spot for a file defined as a directory! Sorry for that.

I run kibot locally now using the kibot wrapper I made for Windows.

I updated the workflow, changed a few other things in the configuration file, and could complete the workflow locally using

act -b --artifact-server-path 'e:\\help' | tee act.log

INTI-CMNB / KiBot

Running github actions locally & retrieving results. #97