[BUG] kicad8 erc crashed with json.decoder.JSONDecodeError

[vermut]

# docker run --rm -ti -v .:/app -w /app/Hardware/OpenMowerMainboard ghcr.io/inti-cmnb/kicad_auto_full:dev_k8 kibot -v
<...snip...>
- Running the ERC
DEBUG:Executing: /usr/bin/eeschema_do -r -v run_erc -o erc_reporta6auszr0.rpt -g 50 -f /app/Hardware/OpenMowerMainboard/kibot_errors.filter /app/Hardw
are/OpenMowerMainboard/OpenMowerMainboard.kicad_sch /tmp (kibot - kiplot.py:175)
DEBUG:Failed with error 1, retrying ... (kibot - kiplot.py:185)
DEBUG:Output from command:
> Traceback (most recent call last):
>   File "/usr/bin/eeschema_do", line 928, in <module>
>     cfg = Config(logger, args.schematic, args)
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/kiauto/misc.py", line 194, in __init__
>     self.load_kicad_environment(logger)
>   File "/usr/lib/python3/dist-packages/kiauto/misc.py", line 294, in load_kicad_environment
>     env = self.get_config_vars_json(self.conf_kicad)
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/kiauto/misc.py", line 307, in get_config_vars_json
>     data = json.load(f)
>            ^^^^^^^^^^^^
>   File "/usr/lib/python3.11/json/__init__.py", line 293, in load
>     return loads(fp.read(),
>            ^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
>     return _default_decoder.decode(s)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
>     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
>                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
>     raise JSONDecodeError("Expecting value", s, err.value) from None
> json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
>  (kibot - kiplot.py:189)
DEBUG:Removing temporal files (kibot.pre_base - pre_base.py:194)

Files are at https://github.com/ClemensElflein/OpenMower/pull/83

Worked in V7 (main branch).

You've helped me already. This is V6 imported into V7 imported into V8. So there might be weird things inside. I just can't decipher where to look at - I thought there are no JSONs in play.

[set-soft]

Hi @vermut ! The message seems to be related with some problem loading the kicad_common.json. You also mention the project is imported in v8, but you are running the v7 docker image (v2_k7). So it looks like some mess with versions

[vermut]

We don't have kicad_common.json in our repo. Should I add empty one?

Yes, v2_k7 is still in the GH config, because PR was created before v2_k8 was released. Anyway, I am testing that locally on my PC using ghcr.io/inti-cmnb/kicad_auto_full:dev_k8 (see first line of the this issue).

[set-soft]

We don't have kicad_common.json in our repo. Should I add empty one?

This is in the docker image, are you sure you want a fresh image? In a fresh image I get:

# ls -la /root/.config/kicad/8.0/kicad_common.json 
-rw-r--r-- 1 root root 1927 Mar 31 15:28 /root/.config/kicad/8.0/kicad_common.json

And the file contains valid JSON. Just try:

docker run --rm -ti -v .:/app -w /app/Hardware/OpenMowerMainboard ghcr.io/inti-cmnb/kicad_auto_full:dev_k8  ls -la /root/.config/kicad/8.0/kicad_common.json 

Yes, v2_k7 is still in the GH config, because PR was created before v2_k8 was released. Anyway, I am testing that locally on my PC using ghcr.io/inti-cmnb/kicad_auto_full:dev_k8 (see first line of the this issue).

I tried using the same command you put here:

docker run --rm -ti -v .:/app -w /app/Hardware/OpenMowerMainboard ghcr.io/inti-cmnb/kicad_auto_full:dev_k8 kibot -v

And didn't get your error. I got an error related to the fact that I just downloaded a .ZIP file and didn't clone the repo.

[vermut]

I see file present in docker, but the command still failing for me. I'll try tomorrow on ZIP, and after that on Linux - maybe Windows is messing up some important permission.

[set-soft]

Hi @vermut ! I added more debug information when we hit a parser exception reading the kicad_common config. The KiCad 7/8 dev images now contain such a feature. So if you refresh the images and try again you'll get more information about the problem.

[vermut]

I ran it manually (from docker bash) and kicad_common is present BEFORE I run kibot but empty AFTER I run it. What's going on?..

root@d4924a93d90e:/app/Hardware/OpenMowerMainboard# cat /root/.config/kicad/8.0/kicad_common.json  | python3 -c "import json; import sys;  print(json.load(sys.stdin).keys())"
dict_keys(['appearance', 'auto_backup', 'do_not_show_again', 'environment', 'git', 'graphics', 'input', 'meta', 'netclass_panel', 'package_manager', 'session', 'system'])

root@d4924a93d90e:/app/Hardware/OpenMowerMainboard# kibot   
Using config file: OpenMowerMainboard.kibot.yaml
WARNING:(W044) More than one SCH file found in `.`.
  Using ./OpenMowerMainboard.kicad_sch if you want to use another use -e option. (kibot - kiplot.py:833)
- Running the DRC
- Running the ERC
ERROR:Corrupted KiCad config file `/root/.config/kicad/8.0/kicad_common.json`:
 (eeschema_do - misc.py:312) (kibot - kiplot.py:124)
ERROR:ERC returned 20 (kibot.gs - gs.py:814)

root@d4924a93d90e:/app/Hardware/OpenMowerMainboard# ls -al /root/.config/kicad/8.0/kicad_common.json
-rw-r--r-- 1 root root 0 Apr  2 14:59 /root/.config/kicad/8.0/kicad_common.json

[vermut]

I disabled DRC - ERC worked, but next step broke with same error. Something between steps is doing something.

Using config file: OpenMowerMainboard.kibot.yaml
WARNING:(W044) More than one SCH file found in `.`.
  Using ./OpenMowerMainboard.kicad_sch if you want to use another use -e option. (kibot - kiplot.py:833)
- Running the ERC
WARNING:(W058) Ignoring (pin_to_pin) Pins of type Power output and Power output are connected
    ; error
<skip 1000000 warnings>

- Updating BoM in XML format
ERROR:Corrupted KiCad config file `/root/.config/kicad/8.0/kicad_common.json`:
 (eeschema_do - misc.py:312) (kibot - kiplot.py:124)
ERROR:/usr/local/bin/eeschema_do returned 20 (kibot.gs - gs.py:816)

[set-soft]

You can try running with high verbosity levels (i.e. -vvvv) You'll see how KiBot calls eeschema_do and pcbnew_do and call this commands manually. You can also run the commands with -vvvv to see messages about "Restoring old %s config".

[set-soft]

I tried:

ls -la ~/.config/kicad/8.0/kicad_common.json ; eeschema_do run_erc OpenMowerMainboard.kicad_sch .; ls -la ~/.config/kicad/8.0/kicad_common.json

And all worked well

[set-soft]

Also tried it as root using your command line:

root@1fb97f0a89f1:/app/Hardware/OpenMowerMainboard# ls -la ~/.config/kicad/8.0/kicad_common.json ; eeschema_do run_erc OpenMowerMainboard.kicad_sch .; ls -la ~/.config/kicad/8.0/kicad_common.json

And got:

-rw-r--r-- 1 root root 1927 Mar 31 15:28 /root/.config/kicad/8.0/kicad_common.json

Before and after running the ERC.

Note that KiAuto will move away the KiCad config renaming to use a reproducible config, and then will rename it again to the original name. Something must be really wrong to truncate a file during a rename operation.

[vermut]

Kicad commands doesn't harm the json file.

root@56d6ea1a6e43:/app/Hardware/OpenMowerMainboard# kicad-cli pcb drc -o /app/Hardware/OpenMowerMainboard/OpenMowerMainboard-drc.txt --format json --severity-all --units mm /app/Hardware/OpenMowerMainboard/OpenMowerMainboard.kicad_pcb
Loading board
Running DRC...
Found 162 violations
Found 0 unconnected items
Saved DRC Report to /app/Hardware/OpenMowerMainboard/OpenMowerMainboard-drc.txt

root@56d6ea1a6e43:/app/Hardware/OpenMowerMainboard# ls -al ~root/.config/kicad/8.0/ -al
total 64                                   
drwxr-xr-x 3 root root  4096 Apr  2 15:15 .
drwxr-xr-x 3 root root  4096 Apr  2 15:15 ..
-rw-r--r-- 1 root root  6912 Apr  2 15:15 3d_viewer.json
drwxr-xr-x 2 root root  4096 Apr  2 15:15 colors
-rw-r--r-- 1 root root  7265 Apr  2 15:15 cvpcb.json
-rw-r--r-- 1 root root  6019 Apr  2 15:15 fpedit.json
-rw-r--r-- 1 root root  4465 Apr  2 15:15 kicad.json
-rw-r--r-- 1 root root  1959 Apr  2 15:15 kicad_common.json
-rw-r--r-- 1 root root 15363 Apr  2 15:15 pcbnew.json

[set-soft]

Of course, the problem must be in an operation done by eeschema_do and pcbnew_do But looks like some bizarre problem in your OS.

[vermut]

Yes, this works on usual Linux machine. Under pure WSL docker it gave me this:

- Running the DRC
- Running the ERC
ERROR:KiCad common config back-up found (/root/.config/kicad/8.0/kicad_common.json.pre_script) (eeschema_do.kiauto.file_util - file_util.py:210) (kibot - kiplot.py:124)
ERROR:It could contain your kicad common configuration, rename it to /root/.config/kicad/8.0/kicad_common.json or discard it. (eeschema_do.kiauto.file_util - file_util.py:211) (kibot - kiplot.py:124)
ERROR:ERC returned 3 (kibot.gs - gs.py:814)

So this time it didn't rename back. Possibly a race condition, but otherwise I had enough of this, we can close it with "windows being windows" resolution.

[set-soft]

I pushed a patch to mess with config files only when using the GUI. This should workaround the problem.

[set-soft]

Now run_drc and run_erc are deprecated in favor of drc and erc that uses KiCad CLI directly. So this issue, even when looks like a Windows glitch, shouldn't be a problem.

[vermut]

Confirm, works flawlessly even on my devil's spawn OS.