search

INTI-CMNB / KiBot

KiCad automation utility
GNU Affero General Public License v3.0
573 stars 68 forks source link

[FEATURE] export kicad project with libs #491

Closed hpopols closed 11 months ago

hpopols commented 1 year ago

Is your feature request related to a problem? Please describe. I want to export the whole project including schematics, pcb and project used libs (schematic and PCB, 3D could be nice but not mandatory). In a word : export a minimal working project in a zip file.

My actual use case is for release (client deliverable, peer review, archiving...) where it is not possible/desirable/practical/relevant to share my company whole libraries.

Describe the solution you'd like Actually, I am able to realize this process manually using only the kicad gui following this post.

It would be very nice to have it done auto-magically in CI/CD through kiauto/kibot.

Describe alternatives you've considered Continue to do it manually ^^'

hpopols commented 1 year ago

I am willing to work on it but, before commit to it, I am wondering:

set-soft commented 1 year ago

Hi @hpopols !

When using KiCad 7 the schematic and PCB files are "self-contained", the only real problem are 3D models (not very well handled by KiCad).

The copy_files output has a special mode to copy the 3D models and save a modified PCB that can use the copied files.

This should solve your problem. The only detail is that running the ERC/DRC will generate errors about the missing libs (KiCad 7 likes to compare the embedded symbols/footprints with the copy in the lib, and using a weak method).

Is that what you need?

hpopols commented 1 year ago

Hi @set-soft !

Thank you again for your responsiveness!

Regarding personal use, I largely concur with your perspective. It seems that there is no essential 'information' missing in the 'copy_file' output. Additionally, manually generating the necessary library from the schematic and pcb files doesn't pose significant challenges once you grasp the inner workings of KiCad and how it stores information.

However, the inconsistency between the original KiCad project and its exported counterpart remains a concern for me. I firmly believe that users should reasonably expect the released (exported) version to exhibit the same 'out-of-the-box' behavior as the one available in the repository version through the CI/CD pipeline. One could argue that the exported version should be responsible for generating outputs (ERC, DRC, PDF, Gerbers, etc.) within the CI/CD pipeline, while the 'exported library' could function as a preflight option.

In my specific scenario, releasing a KiCad project that triggers library errors and fails to pass ERC/DRC inspections, despite the CI/CD reports and my own assessments stating otherwise, is simply not viable. This inconsistency issue carries significant practical consequences, as it raises valid questions and concerns among my clients and peer reviewers. Furthermore, these individuals may perceive the delivered package as incomplete or subpar, which is entirely unacceptable.

I firmly believe that implementing this functionality would be advantageous for everyone. If you share this belief and consider the implementation to be reasonably feasible, I would like to give it a try.

What do you think?

set-soft commented 1 year ago

Hi @hpopols ! The main problem here is how KiCad tries to compare a symbol/footprint with its lib counterpart. The comparison assumes things that complicates all the process. I'm not sure about it. You'll get a couple of libs (one with symbols and the other with footprints) whose only purpose is to allow people to run all the ERC/DRC tests. IMHO is much more sensitive to create a project with these checks disabled. So you can run ERC/DRC, but using the generated project. For KiCad 5 creating the libs (like the cache lib) is needed, but now all the symbols and footprints are inside the files. Generating a copy of them is only needed to allow KiCad to tell you that they are the same. I don't know, it looks like a silly workaround.

hpopols commented 1 year ago

Hi @set-soft !

I appreciate your prompt response and valuable insights.

I'd like to begin by extending my apologies for any confusion that may have arisen from my previous message. Upon further reflection, I've come to realize that there isn't a fundamental flaw in the way things work; rather, the issue lies in the omission of the original libraries in the exported project. When the original libraries are included, the issues are effectively resolved.

Nevertheless, I maintain the belief that relying on an exported library for the release isn't as cumbersome as it may initially appear. It's analogous to distributing a minimal library virtual environment alongside the project, a practice that can be considered beneficial in certain scenarios.

The present situation results in broken links within the KiCad files, which, in my view, is less than ideal for a release. Disabling the ERC/DRC checks to circumvent this issue doesn't seem to be a desirable solution.

To illustrate my idea, I've been contemplating a potential CI/CD build pipeline for this kind of project (It would involve multiple Kibot jobs):

I'm interested in hearing your thoughts on implementing such a feature in Kibot. Additionally and in any case, if you could provide some insight into the level of complexity involved and suggest where I should begin (e.g., modifying the PCB/SCH source or working with Kiauto), it would be greatly appreciated.

Once again, thank you for your time and assistance.

Xavier

set-soft commented 1 year ago

I'm interested in hearing your thoughts on implementing such a feature in Kibot.

It can be done without exporting any lib. If the generated project just disables the "lib parity checks" (as other silly things are disabled ... like checking if each symbol has a SPICE model!) you can check the ERC/DRC of the generated project.

Additionally and in any case, if you could provide some insight into the level of complexity involved and suggest where I should begin (e.g., modifying the PCB/SCH source or working with Kiauto), it would be greatly appreciated.

The KiAuto solution is interesting, but I think will be limited to some KiCad versions. And the code already does some work with libs, specially on KiCad 5 files. So I think adding some functionality to the v5_sch.py and v6_sch.py files will make it simple.

hpopols commented 1 year ago

Hi @set-soft !

It can be done without exporting any lib. If the generated project just disables the "lib parity checks" (as other silly things are disabled ... like checking if each symbol has a SPICE model!) you can check the ERC/DRC of the generated project.

I had a quite relevant experience to share which occurred just this afternoon during a formal design review session with a client. Right at the outset of the session, the client's first step was to meticulously verify the ERC/DRC rules and run both checks on the project I had provided. I was relief to have provided him the project with the exported the libs. I just can say, this practice is standard procedure among my clients and is an integral part of their formal review process.

The KiAuto solution is interesting, but I think will be limited to some KiCad versions. And the code already does some work with libs, specially on KiCad 5 files. So I think adding some functionality to the v5_sch.py and v6_sch.py files will make it simple.

Thank you for this starting point, I will dig into it and I will stop bothering you with this!

Have a great day!

set-soft commented 11 months ago

Hi @hpopols !

The above patch adds a new source_type named project, I think it covers this feature request. Can you test if this works for your projects?

hpopols commented 11 months ago

Hi @set-soft !

Wow this would be amazing! I want definitely check this but I am not sure how to apply this patch inside my container and from which version I should apply it.

Is the last dev kicad7_auto_full container has this feature in it? (https://github.com/inti-cmnb/kicad_auto_test/pkgs/container/kicad7_auto_full/151698440?tag=dev_1.6.4-2bc3bdf_k7.0.9_d12.1_b3.5.1)?

Thank you very much!

set-soft commented 11 months ago

Is the last dev kicad7_auto_full container has this feature in it? (https://github.com/inti-cmnb/kicad_auto_test/pkgs/container/kicad7_auto_full/151698440?tag=dev_1.6.4-2bc3bdf_k7.0.9_d12.1_b3.5.1)?

Yes, if you look at the name it says 2bc3bdf, now you can go to the commits for the dev branch and you'll see this hash is for november 27, and the patch you need is e686d9b from november 24. In general: development patches are applied to the dev branch and they are available in docker images as soon as the patch passes the regression tests.

hpopols commented 11 months ago

Hi @set-soft,

I apologize for the delay but I had to release a board for a client asap. Anyway, I have now more time to test this wonderfull feature! 😃

The setup:

I tested it with a rather large project for me (1000+ components hierarchical design, with db libs, classic project libs, kicad lib, power libs and few components with modified footprint/symbols outside the lib (mainly pin swap, pad trim, silkscreeen adjustements...)).

The image used is: ghcr.io/inti-cmnb/kicad7_auto_full:dev_1.6.4-02a2334_k7.0.9_d12.1_b3.5.1

The relevant part of the kibot config is:

  - name: copy_kicad_project2
    comment: export kicad project source
    type: copy_files
    dir: "./kicad_project"
    options:
      files:
        - source_type: project

I run it with : kibot -c config.kibot.yaml -e xxx.kicad_sch -d ../04-outputs -s all copy_kicad_project2

The outcome:

The process begins and the output file was filling until this error occurred (during the symbol copy process):

Traceback (most recent call last):
  File "/usr/local/bin/kibot", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/kibot/__main__.py", line 536, in main
    generate_outputs(outputs, args.target, args.invert_sel, args.skip_pre, args.cli_order, args.no_priority,
  File "/usr/local/lib/python3.11/dist-packages/kibot/kiplot.py", line 611, in generate_outputs
    _generate_outputs(outputs, targets, invert, skip_pre, cli_order, no_priority, dont_stop)
  File "/usr/local/lib/python3.11/dist-packages/kibot/kiplot.py", line 601, in _generate_outputs
    run_output(out, dont_stop)
  File "/usr/local/lib/python3.11/dist-packages/kibot/kiplot.py", line 502, in run_output
    out.run(get_output_dir(out.dir, out))
  File "/usr/local/lib/python3.11/dist-packages/kibot/out_copy_files.py", line 405, in run
    self.options.run(output_dir)
  File "/usr/local/lib/python3.11/dist-packages/kibot/out_copy_files.py", line 350, in run
    files = self.get_files()
            ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/kibot/out_copy_files.py", line 295, in get_files
    files_list, extra_files = self.get_3d_models(f, mode_project, no_out_run)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/kibot/out_copy_files.py", line 244, in get_3d_models
    extra_files += self.copy_symbols(f.dest, dry)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/kibot/out_copy_files.py", line 181, in copy_symbols
    GS.sch.write_lib(out_lib_base, lib, comps)
  File "/usr/local/lib/python3.11/dist-packages/kibot/kicad/v6_sch.py", line 1907, in write_lib
    lib.extend([self.lib_symbol_names[name+':'+s].write(False), Sep()])
                ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'locally_edited:GND_1'

There are two reference to "GND_1" both in main schematic (renamed xxx.kicad_sch) as:

    (symbol "GND_1" (power) (pin_names (offset 0)) (in_bom yes) (on_board yes)
      (property "Reference" "#PWR" (at 0 -6.35 0)
        (effects (font (size 1.27 1.27)) hide)
      )
      (property "Value" "GND_1" (at 0 -3.81 0)
        (effects (font (size 1.27 1.27)))
      )
      (property "Footprint" "" (at 0 0 0)
        (effects (font (size 1.27 1.27)) hide)
      )
      (property "Datasheet" "" (at 0 0 0)
        (effects (font (size 1.27 1.27)) hide)
      )
      (property "ki_keywords" "global power" (at 0 0 0)
        (effects (font (size 1.27 1.27)) hide)
      )
      (property "ki_description" "Power symbol creates a global label with name \"GND\" , ground" (at 0 0 0)
        (effects (font (size 1.27 1.27)) hide)
      )
      (symbol "GND_1_0_1"
        (polyline
          (pts
            (xy 0 0)
            (xy 0 -1.27)
            (xy 1.27 -1.27)
            (xy 0 -2.54)
            (xy -1.27 -1.27)
            (xy 0 -1.27)
          )
          (stroke (width 0) (type default))
          (fill (type none))
        )
      )
      (symbol "GND_1_1_1"
        (pin power_in line (at 0 0 270) (length 0) hide
          (name "GND" (effects (font (size 1.27 1.27))))
          (number "1" (effects (font (size 1.27 1.27))))
        )
      )
    )

and later in the same file (main schematic)

  (symbol (lib_name "GND_1") (lib_id "power:GND") (at 359.41 48.26 0) (unit 1)
    (in_bom yes) (on_board yes) (dnp no) (fields_autoplaced)
    (uuid f7f85c97-8b4f-4f69-a34e-3abdca30ee56)
    (property "Reference" "#PWR0157" (at 359.41 54.61 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "Value" "GND" (at 359.41 53.34 0)
      (effects (font (size 1.27 1.27)))
    )
    (property "Footprint" "" (at 359.41 48.26 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "Datasheet" "" (at 359.41 48.26 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (pin "1" (uuid 97ff4b28-a829-49fb-9d3c-e7f66ed6f423))
    (instances
      (project "_xxx_"
        (path "/5eefe6a1-faae-4012-9a95-08477ec0e762"
          (reference "#PWR0157") (unit 1)
        )
      )
    )
  )

I did notice the process I mentionned in my first post (through kicad export libs) also has issues with exporting power symbols.

Do not hesitate to ask me further data/tests. I have now more time to work on this and will try to be responsive!

set-soft commented 11 months ago

Hi @hpopols ! The above patch should solve the issue. Note that this is a symbol that was locally edited in the SCH file. It should be reported as mismatch between the SCH and the lib by KiCad. KiBot now skips them when exporting the symbol libs.

hpopols commented 11 months ago

Strangely enough the ERC does not report this GND power symbol at all (but it does report a custom transformer symbol which I pin-swaped coils).

I rolled back the project from the previous state, run the same kibot command and the process succeeded. However I have few things to mentions:

The last point does not bother me as we already planned to be more self contained with all the libraries as you suggest in your documentation.

Note : all the below tests was done using the kicad gui shipped in the container to avoid to many versions/os compatibility problems...

Thank you again

set-soft commented 11 months ago
  • [schematic/pcb] The drawing sheets are not copied and rereferenced (.kicad_wks)

This should be fixed. Will look into it after fixing some QR Lib issues.

  • [schematic] There is brand 7 new ERC warnings. All of them are "symbol not found in library" although only 2 unique ones (namely BAT54S and the 74AHC1G14). Both was not reported at all by the erc before, are from the kicad stock lib and are indeed missing from the exported project libraries. I think these 2 stock kicad lib symbols was changed between kicad bugfix versions upgrade and are no more in sync with the schematic but not reported by the ERC. I think these out-of-sync symbols has to be exported to allows kicad to make the comparison and replicate the initial project behaviors.

I need a small example showing it. From the description I don't know what is the source of the problem.

  • [pcb] Custom rules are not copied and rereferenced (.kicad_dru)

Should be fixed. Will look into it after fixing some QR Lib issues.

  • [pcb] The 3D model path between the lib and the pcb are now out of sync. Indeed the exported lib model path point to the old ${KICAD6_3DMODEL_DIR} and not the new 3DMODEL path.

Strange, should be easy to reproduce, an example could help.

hpopols commented 11 months ago

I will try to produce you a minimal representative example by tomorow

set-soft commented 11 months ago
  • [pcb] The 3D model path between the lib and the pcb are now out of sync. Indeed the exported lib model path point to the old ${KICAD6_3DMODEL_DIR} and not the new 3DMODEL path.

Strange, should be easy to reproduce, an example could help.

I took a look at the test project and I don't get this problem. It could be related to some problem trying to detect KICAD6_3DMODEL_DIR.

hpopols commented 11 months ago

Hi @set-soft,

I think you pinpointed the problem. With a much smaller design the terminal output is much more concise:

- 'export kicad project source' (copy_kicad_project2) [copy_files]
WARNING:(W009) KiCad config without environment.vars section (kibot - config.py:358)
WARNING:(W010) Unable to find KiCad 3D models (kibot - config.py:433)
ERROR:Unable to expand `KICAD6_3DMODEL_DIR` in `${KICAD6_3DMODEL_DIR}/Connector_IDC.3dshapes/IDC-Header_2x05_P2.54mm_Vertical.wrl` (kibot - config.py:113)
ERROR:Unable to expand `KICAD7_3DMODEL_DIR` in `${KICAD7_3DMODEL_DIR}/Connector_USB.3dshapes/USB_C_Receptacle_GCT_USB4105-xx-A_16P_TopMnt_Horizontal.wrl` (kibot - config.py:113)
WARNING:(W098)  2 3D models downloaded or cached (kibot - out_base_3d.py:526)
Found 3 unique warning/s (3 total)

Both "KICAD6_3DMODEL_DIR" and "KICAD7_3DMODEL_DIR" seems to have a problem.

This environment vars seems to be a kicad related. When using the kicad version inside the kicad_auto_full container I get this defined paths: image

The "KICAD7_3DMODEL_DIR" seems to be defined there but not the "KICAD6_3DMODEL_DIR". However, I don't know if kibot is using those definitions or not...

Do you still need the minimal project (I have to work on it a little bit more) as it does not seems to be related to the project, isn't it?

Good evening.

Xavier