Archive project: missing remapping of footprints

[KarlZeilhofer]

Hi,

first, thanks for that great plugin! I think this is very useful in general, but still missing some features.

I think, if a project is called portable, it should be editable after archiving. Usecase could be to do a design in house, and then release it to the public domain. Changes by other people should be easily doable.

What has been achieved so far:

• archiving 3D files
• updating the 3D models of the footprints within the kicad_pcb file
• these footprints can be exported by KiCad PCBnew (File -> Archive Footprints, which updates/creates the project's fp-lib-table)
• all schematic symbols are archived into a xxx-archive.lib
• project's sym-lib-table is updated

What's the difference to stock KiCad?

• symbols are cached anyway by KiCad - no need for an xxx-archive.lib
• 3D models are perfectly archived with this plugin. This was missing in KiCad!

What is still missing:

Remapping footprints

For now all the symbols in the schematic editor still have the link to the original footprint. That means, that a modified schematic (e.g. copy/paste an existing symbol within a schematic sheet) can not be pulled into the PCB, since the symbol refers to an undefined footprint.

Autoarchive Footprints

this is tedious to be done manually. And if remapping footprints is implemented, this also is a step between archiving 3D models combined with updating them in the PCB file, and remapping footprints in the schematic.

Greets, Karl

[MitjaNemec]

Thanks for the feedback Karl.

From the feedback that I've received and from forum.kicad.info I can see that there are as many different workflows as there are users. And I prefer to have the symbols stored in a dedicated library as they can be edited. Having them only in -cache.lib still makes the project portable but can lead to conflicts if user has a different symbol with the same name in the library with same nickname.

As for Autoarchive of footprints, the 5.1.x branch of KiCad does not support this, but it is doable in 5.99 (see #41). Once this is working the footprint remapping should be easy to implement.

But currently there are a couple of serious issues in the way:

• being solo developer I can not support 5.1.x branch and 5.99 at the same time (I try to make some plugins work on both systems, but the code is starting to bloat - see #64)
• even if I had the time and energy I don't have a slightest clue how to release separate versions of the plugins
• I'd probably have to get much more experience using git in order to support development of two branches concurrently (cherry picking, ...)

So at the time, I am focusing my work on 5.1.x branch, once the V6 will be release, I'll first migrate the plugins (which is probably going to take some time as schematics format will change) and only then I'll start adding features.

But if you feel like it, you can fork the repo and start adding the functionality. I can only offer pointers how to proceed.

And I'll grab this opportunity trying to recruit you. If coding is not your forte, the plugin documentation could be significantly improved.

[EeliK]

For 6.0 the schematic and symbol file formats will be changed. The symbols will be embedded into schematic files, like footprints are now. The cache file will be gone. It's hardly worth it to try to keep a version for 5.99 until that happens. But then porting will be easy - the same code should at least in theory work for symbols and footprints.

Also the 3D files will probably use a library table system (which you already know from https://forum.kicad.info/t/cache-embed-3d-model-in-footprint/19371/23?u=eelik).

I agree with Karl that remapping footprints is needed. The "archive footprints" functionality in KiCad is half-baked for making a project standalone because I have to manually edit existing footprint links. I want a self-contained but yet normally editable project so that I can make modifications to a local library and update the board footprints from there instead of a central library. I would like to keep the changes to embedded footprints minimal - I often change the library footprints and update the board, and loose customizations in an embedded footprint if there are any.

(After that I would like to be able to check for updated footprints in the central library vs. in the local library and possibly do the updates between libraries, but that would be another plugin.)

It's possible that after file format changes are made to 5.99, this plugin could be reduced to remapping functionality - or, if proper remapping is implemented into KiCad, to nothing. On the other hand KiCad developers don't like implementing periferal functionality, and I wouldn't oppose seeing all archiving and remapping in a plugin.

[MitjaNemec]

While diverging from the main topic of this issue, I'll discuss what you wrote.

Currently my main effort is in supporting 5.1 branch. If doable, I support the 5.99 branch but only if it requires minor code changes, without modifying flow or program logic. So the schematics part of the plugin is much relevant (only with some workflows, as you can always lean on the -cache.lib otherwise) as V6 is quite some time away. It is true that when V6 is released the schematic part of the plugin will become obsolete, but then the symbols remapping will be required. And I am afraid that eeschema python support will most likely not be released with V6, so it will be hard to implement this.

As for the footprints, I agree that remapping is required in some workflows, but currently I can not really implement this with the python API available in 5.1.4. It is probably doable in 5.99, but I've got other things on my plate, but if anyone is interested, I can offer pointers.

And finally the 3D models. I am currently quite content with current implementation, but I can appreciate the library table solution. The thing I am afraid is that with the change, the python API will not be available immediately.

[paulvdhoeven]

So, if I understand correctly. Pcbnew, even older versions, can easily create an archive of it's footprints with: File / Archive Footprints / ..." but this function is not easily accessible via a Python script?

This step is also easily done manually. The most manual work is to update all the references of Footprints in the schematic. This is also the most important reason I am interested in a script. It is also a necessary step to ensure that if an old project is opened, it's integrity is maintained if the KiCad libraries have changed in between.

In Pcbnew, there is an option to export a Footprint association file. Pcbnew / File / Export / Footprint Association File ...

This file can be imported into Eeschema. So, if there is an easy way to convince Pcbnew that it's exported footprints library is the official source of Footprints, the schematic can be updated via this file. Using a script to modify the "foobar.cmp" file to fix the links may also be relatively easy.

The schematic-cache file is a hack which I do not want to depend on too much. I'm now in the process of cleaning up a project made in KiCad V4 to use it with KiCad V5. Part (or all?) of the schematic symbols now refer to the "-resque" library. I think the -resque lib is a copy of the -cache lib, possibly only for symbols for which the links to the libraries are broken. In KiCad V4 schematic symbols were easy. Just rename the -cache.lib and add it to the top of the library list. Back then the libraries were searched in order. (Which was also not ideal.

Some tips: When archiving a project, always add .pdf (or .svg) version of the schematic. At least you have a human readable backup if you have library troubles. Also never delete the schematic symbols with the big question marks. Instead, repair the the library links in them.

[paulvdhoeven]

If you click on a Footprint in Pcbnew, and then press Edit you get the Footprint Properties. In the Footprint Properties there is an option to change the footprint. (or multiple footprints!) Can this also be easily accessed via scripts? If footprints references are changed here, they can be backported to Eeschema via the Footprint Association file.

[MitjaNemec]

So, if I understand correctly. Pcbnew, even older versions, can easily create an archive of it's footprints with: File / Archive Footprints / ..." but this function is not easily accessible via a Python script?

Yes. The support for this has been added to master branch (5.99), but I am currently only supporting 5.1.x branch

The most manual work is to update all the references of Footprints in the schematic. This is also the most important reason I am interested in a script. It is also a necessary step to ensure that if an old project is opened, it's integrity is maintained if the KiCad libraries have changed in between.

Since V4 you can open a layout file without a problem as layout file contains footprint cache

While I can see the benefits of exporting all of the used footprints to project library mainly for sharing I don't see the reason why should you want to re-link the footprints within the layout to point to project library? I don't know if this is currently even possible as I haven't tried it, but I'd look into the work from Miles Mccoo. I seem to remember that he was doing something with footprints and footprint libraries. And even if you manage to remap them in layout, you'd have to sync these changes back to schematics. Now this might be already implemented with current back annotation support that will be added. But it would be great if this could be triggered within python API. My experience with project archiving is that it has to be fully automatic from start to end. If you require intermediate manual steps, then you have to make sure that they were executed properly (this is a major PITA by itself), as if you don't, sooner or later somebody will note make the intermediate steps as required and it might (will) damage his/hers project.

The schematic-cache file is a hack which I do not want to depend on too much. I'm now in the process of cleaning up a project made in KiCad V4 to use it with KiCad V5. Part (or all?) of the schematic symbols now refer to the "-resque" library. I think the -resque lib is a copy of the -cache lib, possibly only for symbols for which the links to the libraries are broken. In KiCad V4 schematic symbols were easy. Just rename the -cache.lib and add it to the top of the library list. Back then the libraries were searched in order. (Which was also not ideal.

Yes the cache files are a hack, but life is better with them, then without them. And it is true that in V4 the -cache.lib files were pure library file, so you could map them to the project. This changed in V5. this was also the reason why I developed this plugin. And when V 6 comes out, where the cache of symbols will be contained in .sch file I'll get rid of this functionality and I'll keep only the 3D model part of the plugin.

If somebody is willing to develop footprint remapping, I am open to including it in the plugin, but is has to be pretty robust and fully automatic.

[MitjaNemec]

If you click on a Footprint in Pcbnew, and then press Edit you get the Footprint Properties. In the Footprint Properties there is an option to change the footprint. (or multiple footprints!) Can this also be easily accessed via scripts? If footprints references are changed here, they can be backported to Eeschema via the Footprint Association file.

I don't know if this can be accessed by python API. As said, I am interested only in fully automatic conversion. Getting this info back to schematics is also possible via file parsing the way I am remapping the schematics symbols. But as I don't see (e.g. will not use this feature personally) I am not really motivated to implement it. I am wiling to merge it though.

[paulvdhoeven]

While I can see ... I don't see the reason why should you want to re-link the footprints within the layout to point to project library?

Its for project integrity. The footprint field in schematic symbols have a reference to the footprint library that the footprints were originated from. For an archived project you do not want to depend on external libraries. Thus, if you export all footprints to a project specific library in Pcbnew, you create a footprint library independent from KiCad's main libraries.

I don't know if this is currently even possible as I haven't tried it, but ...

Yes, it's possible. I did it by hand (which is error prone)

And even if you manage to remap them in layout, you'd have to sync these changes back to schematics. Now this might be already implemented with current back annotation support that will be added. But it would be great if this could be triggered within python API.

Footprint fields can already be back-anotated via the .cmp file I mentioned earlier.

My experience with project archiving is that it has to be fully automatic from start to end. If you require intermediate manual steps, then you have to make sure that they were executed properly (this is a major PITA by itself), as if you don't, sooner or later somebody will note make the intermediate steps as required and it might (will) damage his/hers project.

Yes, of course, single button is not only much quicker, but eliminates most possibilities of introducing mistakes.

If somebody is willing to develop footprint remapping, I am open to including it in the plugin, but is has to be pretty robust and fully automatic.

I do not have many (old) projects, I am also quite a noob with scripting (-languages). The whole KiCad project is improving at a pretty good rate for years. It went from "Barely usable" to (almost?) surpassing other packages. My strategy is to port the few projects I have manually, and wait for KiCad V6 for most of this stuff. Especially because it apparently significantly overhauls the schematic format.

[EeliK]

Footprint fields can already be back-anotated via the .cmp file I mentioned earlier.

In 5.99 there's this:

[MitjaNemec]

In 5.99 there's this:

Thanks for confirming the back annotation is in 5.99

Its for project integrity. The footprint field in schematic symbols have a reference to the footprint library that the footprints were originated from. For an archived project you do not want to depend on external libraries. Thus, if you export all footprints to a project specific library in Pcbnew, you create a footprint library independent from KiCad's main libraries.

I think we see things differently (and it is alright). The way I see it, exported and re-linked footprints don't really matter as you can edit the footprints used in the layout anyhow. So having the footprints in project specific library does not add anything of value to the project archive. You can always get those footprints from the project. The only use I see for exporting the footprints is that if you get a layout where you see a footprint that you don't have in your library exporting is a way to obtain those footprints so that you can store them in your library.

But maybe I am just too stuck in my point of view that I cannot see things from different perspective.

[EeliK]

There would be two use cases for a project specific library, both involving further editing of the project. As has been said in one way or another, having footprints inside the board file without libraries is enough if it's just for viewing purposes.

1. Edit schematic, add new components identical to some old ones, update PCB.
2. Edit a footprint and update all those footprints on the board.

Neither can be done without a library. Why creating a library and copying footprints from the board to the library should be required to do those things? Why not automate it?

[MitjaNemec]

• Edit schematic, add new components identical to some old ones, update PCB.

• Edit a footprint and update all those footprints on the board.

Aaaah, thanks for opening the eyes for me. Much appreciated.

Currently the issue can not be resolved as python API lacks support. Once V6 will be close to release I'll start supporting the 5.99 and dropping support for 5.1.x (this is the plan at least). Then this might be semi automated (especially if "update schematics from PCB" dialog can be started from python API).

[paulvdhoeven]

I think we see things differently (and it is alright). The way I see it, exported and re-linked footprints don't really matter as ...

The difference between our thinking is not so big. It's just this text typing thing that easily confuses conversations.

I do not really care whether Footprints are in the PCB or in a separate library, but in the schematic there are links with library names to the footprints. To presevere integrity these liks must point to something. Therefore, if those links point to the Footprints exported to a library by Pcbnew, the loop is closed.

For just viewing it is OK, but when you are opening an old and verified project, change the schematic and update the PCB, you do not want existing footprints to be replacec by newer versions from some default library.

It would also be swell to modify the links in eeschema to never import or change the footprints.

I may be nitpicking. Just unchecking the "[ ] Update footprints" goes a long way in preventing errors, but it an all or nothing setting.

[MitjaNemec]

To presevere integrity these liks must point to something. Therefore, if those links point to the Footprints exported to a library by Pcbnew, the loop is closed.

For just viewing it is OK, but when you are opening an old and verified project, change the schematic and update the PCB, you do not want existing footprints to be replacec by newer versions from some default library.

I hadn't considered these cases. So the discussion just bumped the issue a bit higher on my TODO list, but as said, until V6 is ready to come out I'll stay off it. Again, thanks for the discussion