search

INTI-CMNB / KiBot

KiCad automation utility
GNU Affero General Public License v3.0
575 stars 68 forks source link

PDF files generation fails for sym-lib-table error #42

Closed eeintech closed 3 years ago

eeintech commented 3 years ago

I have a very simple project setup to test KiBot, that you can download here. The KiBot configuration was mostly generated with kibot --example command (I commented out some of the preflight section + some formatting fixes).

However, I'm consistently getting this error when trying to generate PDF outputs:

$ kibot pdf_sch_print_example
Using config file: example.kibot.yaml
Using SCH file: test_project.sch
- Running the ERC
- Updating BoM in XML format
ERROR:At line 11 of `sym-lib-table`: Unknown symbol table entry (kibot.kibot.kiplot - kiplot.py:186)
ERROR:Line content: `
` (kibot.kibot.kiplot - kiplot.py:187)

Running pdf_example gave me this output:

$ kibot pdf_example
Using config file: example.kibot.yaml
Using SCH file: test_project.sch
- Running the ERC
- Updating BoM in XML format
- 'Exports the PCB to the most common exhange format. Suitable for printing.' (pdf_example) [pdf]
WARNING:(W009) KiCad config without EnvironmentVariables section (kibot.kibot.kicad.config - log.py:82)
WARNING:(W010) Unable to find KiCad libraries (kibot.kibot.kicad.config - log.py:82)
ERROR:Unable to expand `KICAD_SYMBOL_DIR` in `${KICAD_SYMBOL_DIR}/Modules.lib` (kibot.kibot.kicad.config - config.py:63)
ERROR:At line 11 of `sym-lib-table`: Unknown symbol table entry (kibot.kibot.kiplot - kiplot.py:186)
ERROR:Line content: `
` (kibot.kibot.kiplot - kiplot.py:187)

So I set the KICAD_SYMBOL_DIR environmental variable the same way as in my KiCad paths ("Preferences>Configure Paths..."), there was some progress but it is still unsuccessful to generate any PDF output:

$ kibot pdf_example
Using config file: example.kibot.yaml
Using SCH file: test_project.sch
- Running the ERC
- Updating BoM in XML format
- 'Exports the PCB to the most common exhange format. Suitable for printing.' (pdf_example) [pdf]
WARNING:(W009) KiCad config without EnvironmentVariables section (kibot.kibot.kicad.config - log.py:82)
ERROR:At line 11 of `sym-lib-table`: Unknown symbol table entry (kibot.kibot.kiplot - kiplot.py:186)
ERROR:Line content: `
` (kibot.kibot.kiplot - kiplot.py:187)

What am I missing? Hopefully you can reproduce on your side.

Also, how do I tell KiBot where my KiCad environmental variables are set?

set-soft commented 3 years ago
eeintech commented 3 years ago

Oh okay so I figured it out, KiBot was not reading the right sym-lib-table file... It's also not using the right kicad_common file, how can I specify the location of it? I have like 7 of those:

$ find / -name kicad_common 2>/dev/null 
/home/francois/Desktop/KiCad/tools/KiBot/tests/data/kicad_err_1/kicad_common
/home/francois/Desktop/KiCad/tools/KiBot/tests/data/kicad_err_2/kicad_common
/home/francois/Desktop/KiCad/tools/KiBot/tests/data/kicad/kicad_common
/home/francois/Desktop/KiCad/tools/KiBot/tests/data/kicad_ok/kicad_common
/home/francois/Desktop/KiCad/other-libraries/lib_ci_testing/Lib/kicad_common
/home/francois/.config/kicad/kicad_common
/home/francois/.config/kicad5/kicad_common

But none of those contain my path configuration:

image

After setting the KICAD_CONFIG_HOME environmental variable, I get the following output:

$ kibot pdf_sch_print_example
Using config file: example.kibot.yaml
Using SCH file: test_project.sch
- Running the DRC
{'current_version_release': '0.0.4.6'}
- Running the ERC
- Updating BoM in XML format
WARNING:(W042) Missing doc-lib entry for Power_Ports:PWR_FLAG (kibot.kibot.kicad.v5_sch - log.py:82)
- 'Exports the PCB to the most common exhange format. Suitable for printing.' (pdf_sch_print_example) [pdf_sch_print]
Traceback (most recent call last):
  File "/home/francois/.local/bin/eeschema_do", line 593, in <module>
    eeschema_plot_schematic(cfg)
  File "/home/francois/.local/bin/eeschema_do", line 218, in eeschema_plot_schematic
    wait_for_file_created_by_process(cfg.eeschema_pid, cfg.output_file)
  File "/home/francois/.local/lib/python3.6/site-packages/kiauto/file_util.py", line 57, in wait_for_file_created_by_process
    raise RuntimeError('Timed out waiting for creation of %s' % file)
RuntimeError: Timed out waiting for creation of /home/francois/Desktop/test/Example/pdf_sch_print_dir/test_project.pdf
Traceback (most recent call last):
  File "/home/francois/.local/bin/eeschema_do", line 593, in <module>
    eeschema_plot_schematic(cfg)
  File "/home/francois/.local/bin/eeschema_do", line 218, in eeschema_plot_schematic
    wait_for_file_created_by_process(cfg.eeschema_pid, cfg.output_file)
  File "/home/francois/.local/lib/python3.6/site-packages/kiauto/file_util.py", line 57, in wait_for_file_created_by_process
    raise RuntimeError('Timed out waiting for creation of %s' % file)
RuntimeError: Timed out waiting for creation of /home/francois/Desktop/test/Example/pdf_sch_print_dir/test_project.pdf
ERROR:eeschema_do returned 1 (kibot.kibot.out_pdf_sch_print - out_pdf_sch_print.py:48)

The Example/pdf_sch_print_dir folder now contains this a test_project.plt file but nothing else (no PDF).

eeintech commented 3 years ago

Ok so it seems like it is definitely using ~/.config/kicad/kicad_common because I get this error when moving it elsewhere:

WARNING:(W008) Unable to find KiCad configuration file (/home/francois/.config/kicad/kicad_common) (kibot.kibot.kicad.config - log.py:82)

Now I edited that file to match my KiCad GUI paths:

[EnvironmentVariables]
KICAD_CONFIG_HOME=/home/francois/Desktop/KiCad/kicad-library/SETUP
KICAD_SYMBOL_DIR=/home/francois/Desktop/KiCad/kicad-library/SYMBOLS
KICAD_TEMPLATE_DIR=/usr/share/kicad/template
KICAD_USER_TEMPLATE_DIR=/home/francois/kicad/template
KISYS3DMOD=/home/francois/Desktop/KiCad/kicad-library/3D_MODELS
KISYSMOD=/home/francois/Desktop/KiCad/kicad-library/FOOTPRINTS

but, it still not finding my libraries:

$ kibot -s all pdf_sch_print_example
Using config file: example.kibot.yaml
Using SCH file: test_project.sch
WARNING:(W041) Missing library `Resistors` (kibot.kibot.kicad.v5_sch - log.py:82)
WARNING:(W041) Missing library `Diodes` (kibot.kibot.kicad.v5_sch - log.py:82)
WARNING:(W041) Missing library `Power_Ports` (kibot.kibot.kicad.v5_sch - log.py:82)
- 'Exports the PCB to the most common exhange format. Suitable for printing.' (pdf_sch_print_example) [pdf_sch_print]
WARNING:(W043) Missing component `Power_Ports:PWR_FLAG` (kibot.kibot.kicad.v5_sch - log.py:82)
WARNING:(W043) Missing component `Power_Ports:3.3V` (kibot.kibot.kicad.v5_sch - log.py:82)
WARNING:(W043) Missing component `Power_Ports:GND` (kibot.kibot.kicad.v5_sch - log.py:82)
WARNING:(W043) Missing component `Diodes:DIO-000062-00` (kibot.kibot.kicad.v5_sch - log.py:82)
WARNING:(W043) Missing component `Resistors:RES-000007-00` (kibot.kibot.kicad.v5_sch - log.py:82)
Found 8 unique warning/s (13 total)

Only setting the two KICAD_CONFIG_HOME and KICAD_SYMBOL_DIR environmental variables manually seem to work and gives the RuntimeError from my previous comment.

eeintech commented 3 years ago

@set-soft For the RuntimeError, I ran kibot with the -vv argument and this is the captured video and plot file: export_eeschema_screencast.zip

So it looks like it was plotting with the HPGL and not the PDF output format... I went to KiCad and initiated a print/plot with the PDF option enabled, re-run the command and now it works! Yay!

Result: test_project-schematic.pdf

So now I still do not understand why my configuration is not correct...

set-soft commented 3 years ago

Yes, KICAD_CONFIG_HOME is the environment variable used by KiCad v5 and v6.

The ~/.config/kicad/ is the default place. Are you using ~/.config/kicad5/? Which exact version of KiCad are you using? This is very important because you say you are setting KICAD_CONFIG_HOME to ~/.config/kicad/ but KiCad seems to be reading its config from other place.

The problem is with the configuration, KiAuto is failing to set the correct configuration. Looking at KiAuto code I see it isn't using KICAD_CONFIG_HOME. I'll fix it.

eeintech commented 3 years ago

@set-soft I'm using KiCad 5.1.9

Yes it does look like KiAuto/KiBot isn't looking into my kicad_common, I see it being written when KiBot runs and ends-up the same everytime.

No I don't think KiBot is consuming data from ~/.config/kicad5/, you can disregard this.

Thanks for your support, hopefully with your fix my configuration will be fine :smiley:

set-soft commented 3 years ago

@set-soft I'm using KiCad 5.1.9

Ok.

Yes it does look like KiAuto/KiBot isn't looking into my kicad_common, I see it being written when KiBot runs and ends-up the same everytime.

KiBot always tries to restore any changes to the configuration.

No I don't think KiBot is consuming data from ~/.config/kicad5/, you can disregard this.

I think the problem is the other way around: your KiCad is reading from ~/.config/kicad5/ and you are configuring KiBot to read from ~/.config/kicad/, so they get out of sync.

I'm making a patch that will hopefully solve this problem.

Thanks for your support, hopefully with your fix my configuration will be fine

eeintech commented 3 years ago

@set-soft No, it is not reading from the ~/.config/kicad5, I've tested it and also proved above it is definitely using ~/.config/kicad. I think the problem lies in the fact that the KICAD_CONFIG_HOME and KICAD_SYMBOL_DIR env vars are not read, because when set in ~/.config/kicad it does not find my libraries, but it does when I set them in the prompt using export.

Ok so it seems like it is definitely using ~/.config/kicad/kicad_common because I get this error when moving it elsewhere:

WARNING:(W008) Unable to find KiCad configuration file (/home/francois/.config/kicad/kicad_common) (kibot.kibot.kicad.config - log.py:82)
set-soft commented 3 years ago

@set-soft No, it is not reading from the ~/.config/kicad5,

Are you sure your KiCad (not KiBot) isn't using ~/.config/kicad5?

eeintech commented 3 years ago

@set-soft Yes, because when I move the ~/.config/kicad5 folder to another location, KiCad still loads my path and settings the same way. KiCad is configured to use the KICAD_CONFIG_HOME path, which is not ~/.config/kicad5.

set-soft commented 3 years ago

The current git code asks KiCad API what's the config path.

You must use both, KiBot and KiAuto, from git.

set-soft commented 3 years ago

@eeintech also try running KiBot with -vv this will tell you where is the KiCad config according to KiCad API.

eeintech commented 3 years ago

The current git code asks KiCad API what's the config path.

You must use both, KiBot and KiAuto, from git.

Cool, I may wait this goes into a release as I don't think I'll have the time and energy to set this up :smiley: I'll rely on manually setting those env vars for the time being.

@eeintech also try running KiBot with -vv this will tell you where is the KiCad config according to KiCad API.

Outputs for command kibot -s all -d Generated -g variant=PROD -vv pdf_sch_print_example

  1. Both KICAD_CONFIG_HOME and KICAD_SYMBOL_DIR unset: log_missing_env_var.txt
DEBUG:Reading KiCad config from `/home/francois/.config/kicad/kicad_common` (kibot.kibot.kicad.config - config.py:224)
WARNING:(W009) KiCad config without EnvironmentVariables section (kibot.kibot.kicad.config - log.py:82)
WARNING:(W010) Unable to find KiCad libraries (kibot.kibot.kicad.config - log.py:82)
DEBUG:Loading symbols lib table `/home/francois/.config/kicad/sym-lib-table` (kibot.kibot.kicad.config - config.py:254)
  1. KICAD_CONFIG_HOME set and KICAD_SYMBOL_DIR unset: log_missing_symbol_dir.txt
DEBUG:Reading KiCad config from `/home/francois/Desktop/KiCad/kicad-library/SETUP/kicad_common` (kibot.kibot.kicad.config - config.py:224)
DEBUG:- KiCad var: KICAD_TEMPLATE_DIR="/usr/share/kicad/template" (kibot.kibot.kicad.config - config.py:238)
DEBUG:- KiCad var: KICAD_USER_TEMPLATE_DIR="/home/francois/kicad/template" (kibot.kibot.kicad.config - config.py:238)
DEBUG:- KiCad var: KISYS3DMOD="/usr/share/kicad/modules/packages3d/" (kibot.kibot.kicad.config - config.py:238)
DEBUG:- KiCad var: KISYSMOD="/usr/share/kicad/modules" (kibot.kibot.kicad.config - config.py:238)
WARNING:(W010) Unable to find KiCad libraries (kibot.kibot.kicad.config - log.py:82)
DEBUG:Loading symbols lib table `/home/francois/Desktop/KiCad/kicad-library/SETUP/sym-lib-table` (kibot.kibot.kicad.config - config.py:254)
ERROR:Unable to expand `KICAD_SYMBOL_DIR` in `${KICAD_SYMBOL_DIR}/Antennas_Filters.lib` (kibot.kibot.kicad.config - config.py:63)
  1. Both KICAD_CONFIG_HOME and KICAD_SYMBOL_DIR set: log_good.txt
DEBUG:Reading KiCad config from `/home/francois/Desktop/KiCad/kicad-library/SETUP/kicad_common` (kibot.kibot.kicad.config - config.py:224)
DEBUG:- KiCad var: KICAD_TEMPLATE_DIR="/usr/share/kicad/template" (kibot.kibot.kicad.config - config.py:238)
DEBUG:- KiCad var: KICAD_USER_TEMPLATE_DIR="/home/francois/kicad/template" (kibot.kibot.kicad.config - config.py:238)
DEBUG:- KiCad var: KISYS3DMOD="/usr/share/kicad/modules/packages3d/" (kibot.kibot.kicad.config - config.py:238)
DEBUG:- KiCad var: KISYSMOD="/usr/share/kicad/modules" (kibot.kibot.kicad.config - config.py:238)
DEBUG:Detected KICAD_SYMBOL_DIR="/home/francois/Desktop/KiCad/kicad-library/SYMBOLS" (kibot.kibot.kicad.config - config.py:247)
DEBUG:Loading symbols lib table `/home/francois/Desktop/KiCad/kicad-library/SETUP/sym-lib-table` (kibot.kibot.kicad.config - config.py:254)
DEBUG:- Adding lib alias `Antennas_Filters` -> `/home/francois/Desktop/KiCad/kicad-library/SYMBOLS/Antennas_Filters.lib` (kibot.kibot.kicad.config - config.py:266)

EDIT: If I have KICAD_SYMBOL_DIR set in /home/francois/Desktop/KiCad/kicad-library/SETUP/kicad_common file, then I just need to set the KICAD_CONFIG_HOME variable :+1: 

DEBUG:Reading KiCad config from `/home/francois/Desktop/KiCad/kicad-library/SETUP/kicad_common` (kibot.kibot.kicad.config - config.py:224)
DEBUG:- KiCad var: KICAD_CONFIG_HOME="/home/francois/Desktop/KiCad/kicad-library/SETUP" (kibot.kibot.kicad.config - config.py:238)
...
DEBUG:Loading symbols lib table `/home/francois/Desktop/KiCad/kicad-library/SETUP/sym-lib-table` (kibot.kibot.kicad.config - config.py:254)
DEBUG:- Adding lib alias `Antennas_Filters` -> `/home/francois/Desktop/KiCad/kicad-library/SYMBOLS/Antennas_Filters.lib` (kibot.kibot.kicad.config - config.py:266)
set-soft commented 3 years ago

Hi @eeintech !

Cool, I may wait this goes into a release as I don't think I'll have the time and energy to set this up I'll rely on manually setting those env vars for the time being.

If you alredy installed using pip this is not that hard, you just need to clone the repos and then pip install them.

git clone https://github.com/INTI-CMNB/KiAuto.git
git clone https://github.com/INTI-CMNB/KiBot.git
cd KiAuto
pip install --user .
cd ../KiBot
pip install --no-compile --user .

If you need .deb packages I can create them.

If the patch works I will create new releases ASAP, but if nobody tests the patch I'll wait until I'm more confident.

eeintech commented 3 years ago

@set-soft I followed the steps you sent to install the latest from KiAuto/KiBot.

Then I did:

$ unset KICAD_CONFIG_HOME && unset KICAD_SYMBOL_DIR
$ kibot -s all -d Generated -vv pdf_sch_print_example
...
DEBUG:Reading KiCad config from `/home/francois/.config/kicad/kicad_common` (kibot.kibot.kicad.config - config.py:181)
DEBUG:- KiCad var: KICAD_CONFIG_HOME="/home/francois/Desktop/KiCad/kicad-library/SETUP" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_SYMBOL_DIR="/usr/share/kicad/library" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_TEMPLATE_DIR="/usr/share/kicad/template" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_USER_TEMPLATE_DIR="/home/francois/kicad/template" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KISYS3DMOD="/usr/share/kicad/modules/packages3d/" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KISYSMOD="/usr/share/kicad/modules" (kibot.kibot.kicad.config - config.py:195)
DEBUG:Loading symbols lib table `/home/francois/.config/kicad/sym-lib-table` (kibot.kibot.kicad.config - config.py:211)
...
$ export KICAD_CONFIG_HOME=/home/francois/Desktop/KiCad/kicad-library/SETUP
$ kibot -s all -d Generated -vv pdf_sch_print_example
...
DEBUG:Reading KiCad config from `/home/francois/Desktop/KiCad/kicad-library/SETUP/kicad_common` (kibot.kibot.kicad.config - config.py:181)
DEBUG:- KiCad var: KICAD_CONFIG_HOME="/home/francois/Desktop/KiCad/kicad-library/SETUP" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_SYMBOL_DIR="/home/francois/Desktop/KiCad/kicad-library/SYMBOLS" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_TEMPLATE_DIR="/usr/share/kicad/template" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_USER_TEMPLATE_DIR="/home/francois/kicad/template" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KISYS3DMOD="/home/francois/Desktop/KiCad/kicad-library/3D_MODELS" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KISYSMOD="/home/francois/Desktop/KiCad/kicad-library/FOOTPRINTS" (kibot.kibot.kicad.config - config.py:195)
DEBUG:Loading symbols lib table `/home/francois/Desktop/KiCad/kicad-library/SETUP/sym-lib-table` (kibot.kibot.kicad.config - config.py:211)
...

As you can see, when KICAD_CONFIG_HOME is unset, the kicad_common path is not updated and uses the default path: /home/francois/.config/kicad/kicad_common when the KICAD_CONFIG_HOME is set to /home/francois/Desktop/KiCad/kicad-library/SETUP in the same file. After setting it manually, it shows the correct path: /home/francois/Desktop/KiCad/kicad-library/SETUP/kicad_common

So I don't think your patch quite works... Hopefully you can figure out why!

eeintech commented 3 years ago

I also set all the paths in ~/.config/kicad/kicad_common file:

DEBUG:Reading KiCad config from `/home/francois/.config/kicad/kicad_common` (kibot.kibot.kicad.config - config.py:181)
DEBUG:- KiCad var: KICAD_CONFIG_HOME="/home/francois/Desktop/KiCad/kicad-library/SETUP" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_SYMBOL_DIR="/home/francois/Desktop/KiCad/kicad-library/SYMBOLS" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_TEMPLATE_DIR="/usr/share/kicad/template" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_USER_TEMPLATE_DIR="/home/francois/kicad/template" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KISYS3DMOD="/home/francois/Desktop/KiCad/kicad-library/3D_MODELS" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KISYSMOD="/home/francois/Desktop/KiCad/kicad-library/FOOTPRINTS" (kibot.kibot.kicad.config - config.py:195)
DEBUG:Loading symbols lib table `/home/francois/.config/kicad/sym-lib-table` (kibot.kibot.kicad.config - config.py:211)

But it's still using the wrong sym-lib-table path... (should be /home/francois/Desktop/KiCad/kicad-library/SETUP/sym-lib-table, when KICAD_CONFIG_HOME env var is set manually it is updated).

set-soft commented 3 years ago

Hi @eeintech !

So I don't think your patch quite works... Hopefully you can figure out why!

I'm using KiCad API now. But I think I found where the problem is.

Your KiCad is using ~/.config/kicad/kicad_common but in this file the KICAD_CONFIG_HOME points to /home/francois/Desktop/KiCad/kicad-library/SETUP. You don't have a real environment variable, just a KiCad configuration setting. Is this right?

Also: You don't have home/francois/Desktop/KiCad/kicad-library/SETUP/kicad_common file? The output of find says this. Is this right?

But you say:

I also set all the paths in ~/.config/kicad/kicad_common file:

Where are they defined? The values in the dialog you captured must be stored somewhere, and this place should be ~/.config/kicad/kicad_common. Can you grep both directories?

I tried to reproduce your setup and found that KiCad can store its config in ~/.config/kicad/kicad_common but then use KICAD_CONFIG_HOME for all the rest. But it looks a little bit incosistent with what you describe.

set-soft commented 3 years ago

Hi @eeintech !

New patch for both tools. They now should handle this dual situation.

I'm also trying to figure out if your setup is the KiCad intended behavior or just a bug. KiCad Forum

eeintech commented 3 years ago

@set-soft Nice, thanks for the follow-up, I just tested and seems like paths are set nicely now :smiley: 

DEBUG:Reading KiCad config from `/home/francois/.config/kicad/kicad_common` (kibot.kibot.kicad.config - config.py:181)
DEBUG:- KiCad var: KICAD_CONFIG_HOME="/home/francois/Desktop/KiCad/kicad-library/SETUP" (kibot.kibot.kicad.config - config.py:195)
DEBUG:- KiCad var: KICAD_SYMBOL_DIR="/home/francois/Desktop/KiCad/kicad-library/SYMBOLS" (kibot.kibot.kicad.config - config.py:195)
DEBUG:Redirecting symbols lib table to /home/francois/Desktop/KiCad/kicad-library/SETUP (kibot.kibot.kicad.config - config.py:239)
eeintech commented 3 years ago

@set-soft So it turns out that our setup is quirky and is not supposed to be supported, as explained by others to me in the forum thread. Sorry it made your head spin for no good reason. Your latest patch seem to be working great however, don't give it up if you like it!

set-soft commented 3 years ago

The current git code behaves as discussed in KiCad's forum. It also prints information about the paths when using -vv.

If it works for you we can close this issue.

eeintech commented 3 years ago

@set-soft

The current git code

Is it different than the one from 3 days ago? Do I have to re-test it to be sure?

set-soft commented 3 years ago

I think the functionality is the same, just more debug info.

I'm closing it.

Even when your setup is particular KiCad didn't complain about it, so I think the problem is in KiCad.

eeintech commented 3 years ago

@set-soft Ok more debug is good. Thanks a lot for your support!