Lib 4.0.7: missing and conflicting environment variable KISYS3DMOD for 3D model's path prefix

[KarlZeilhofer]

Dear Library Maintainers,

I've reported a bug on launchpad: https://bugs.launchpad.net/kicad/+bug/1765740

After talking to @pointhi , he told me, that this should be the correct place to post the issue.

The old libraries V4.0.7, which I've downloaded from http://downloads.kicad-pcb.org/libraries/ have missing path prefixes for the 3D models. It seems, this was implicitly KISYS3DMOD in KiCad 4. But this assumption is incompatible with KiCad 5.

This fact makes the library incompatible with KiCad 5, which really shouldn't be the case.

In fact, I've added a proper prefix to all the model records in the V4 lib, but also renamed that from KISYS3DMOD to KICAD4_3D_DIR, since the new libraries are using again KISYS3DMOD, which would lead to incompatibilities anyway.

To sum it up: There are 2 major problems, which should ideally be fixed upstream:

1. missing prefix in lib 4
2. conflicting prefix in lib 5 with lib 4

Greets, Karl

[poeschlr]

As you already discovered, the majority of footprints in the old repo does include the variable. (Why did you leave this information out of the description here? Do we see a repeat of our last discussion?)

---

The reason we use KISYS3DMOD is that the installers of kicad use that variable. A dual lib setup as you describe it (both v4 and v5 libs active in all projects) is not really possible that way.

I fear that we don't have the manpower to do anything about that. (There is a reason we declared the old libs as legacy.)

[KarlZeilhofer]

Sorry for missing that point about only ~20% of the footprints in the lib v4 are affected. But don't you think, this is more than enough, to publish a new version of that footprint library?

Motivation

Perhaps I have to point out my motivation. I think for us (Team14) it is required, that

1. we can open and edit existing v4 projects in KiCad 4 like we never have installed KiCad 5
2. we can port existing projects to KiCad v5, but without the need to reassign every footprint to a (hopefully) proper one from the new lib v5. That's why a dual-lib setup is needed.
3. we can create new KiCad 5 projects, like we never have been using KiCad 4.

Local solution

I had to invest many hours of thinking, researching and trying to finally find a way for our company, how to set up KiCad and it's libraries, so the upgrade to KiCad 5 won't cost us further problems of incompatibilities with existing, ported and new projects. That time I really would like to save any other professional KiCad users, by fixing the problem in it's root. Here we talk about the library only. Further things have to happen with two sets of configs, as discussed there: https://bugs.launchpad.net/kicad/+bug/1765706

My Help

I could easily provide a patch or pull request for the lib v4 to fix the missing prefixes. What manpower do you need, so that the libraries can coexist by default?

Coexisting Libs v4 and v5

I see 3 ways to realize this (in preferred order):

1. change the default variable for KiCad 5 and lib v5 from KISYS3DMOD to something else
2. patch lib v4, so it won't use KISYS3DMOD any more
3. provide a python script, which can rename all prefixes in the lib v4 footprints. Such a script would be very helpful anyway for managing a proper libraries setup.

ad 3: Such a script could work as simple as this: python3 chenge3dpathvariable <new_variable_name> That script would change all the footprints` model record. The according changes in the .config folder can be done by the user manually.

I think it is mandatory for the release of KiCad 5, that it can coexist with KiCad 4, and that includes both related library versions.

I really hope, we come to a conclusion before releasing KiCad 5, otherwise this will all be a big mess for all the existing KiCad users.

[poeschlr]

We can not touch the v4 libs (change the variable there) because that would break kicad 4 installations that use the github plugin. (which sadly is the default)

We can not change the variable in v5 because we are in feature freeze. (would require the developers to change all installers on all platforms. It would also require changes to the kicad internal 3d files downloader.)

We could in theory publish a "reverse backport" of the old libs such that they use a different variable. Something similar to your option 2 but it is not allowed to touch the master branch of any of the repos.

This is made especially hard as the old libs are in separate repos per lib. (in each we would need to make a separate branch, edit the files and somehow make it possible to download some sort of package that includes all this repos.)

So the only option i can see is to make a new repo that contains all of these files. I can see the benefit but we are already strained enough maintaining one set of libraries. (There is a reason we declared the old libs as archived -> not maintained anymore.)

---

So the least effort solution would be a python script. Just add one to the kicad-library-utils repo.

---

I think it is mandatory for the release of KiCad 5, that it can coexist with KiCad 4, and that includes both related library versions.

That is something that is discussed over at the mailing list. (recently) There are problems in how kicad stores it's setup files. (kicad 4 and 5 use the same files -> conflict) I doubt there will be a solution for kicad 5. (Add your voice over at the mailing list discussion. We librarians have no influence there.)

I am not sure how the path variables setup works on all platforms. At least windows has an installer option where you can select if they are operating system global environment variables or if they are specialized for the currently installed version. (Not sure how this is done in any linux or mac installation. Ask the developers.)

I am convinced that fixing this on the library level is the wrong approach.

The best and safest option is to use a virtual machine where you run the kicad version that you use not so often.

[poeschlr]

Another option would be to go the other way round. Instead of forward porting the v4 libs for kicad 5 we might be better of backporting the new libs for kicad 4. That way all the bugfixes in the new lib will be usable by users who are stuck with kicad 4 for one reason or another. (At least the subset of symbols/footprints that their software supports. At the moment i would guess this is >90% of them.) A backport script for footprints already exists. See https://github.com/KiCad/kicad-library-utils/pull/202 We are only missing one for the symbol libs. (That one needs to "backport" footprint filters and ensure that no pin "number" has more than 4 chars.)

---

To clarify: We do not suggest using kicad 4 libs for new projects. Supporting old projects is fulfilled by the rescue functionality (assuming the cache lib is valid) and by the fact the footprints are embedded inside the pcb_new file.

Your suggestion to change the libs does not help supporting old projects at all. It would help people who wish to create new projects in kicad 5 using the old (buggy) kicad 4 libs. To help people with their old projects you would need to touch the pcb_new files. So a python script might really be the only option.

[KarlZeilhofer]

@poeschlr I've setup a new repository now, which contains the suggested patches and installation instructions: https://github.com/KarlZeilhofer/kicad4-footprints

The rescue and footprint caching mechanisms do not include the 3D models, which this issue is all about.

Back porting of the lib v5 to v4 also doesn't address this issue. It is about opening and editing v4 projects with KiCad 5.

I'm in contact with the devs about the compatibility issues. Using a VM for a CAD system is not an option for me, since the graphical performance is far too weak.

Thanks for your help and suggestions.

[Ratfink]

On Linux, you could install KiCad 4.0.7 as a flatpak, which would remove any conflicts between versions 4 and 5 by keeping all the files for version 4 in a sandbox. I assume there are similar solutions for other OSs.

---

We can not change the variable in v5 because we are in feature freeze.

@poeschlr Oh, has that started already? Might be nice to make an announcement so the others don't have to dig through this issue to find out. 🙂

[poeschlr]

Changing the variable in the libs would not be the problem. But as this variable affects the installers and is also hardcoded in the lib downloader it would affect things that are in feature freeze.

---

Edit for clarification:

The main reason we can not change the variable name is that it is expected that all installations after the feature freeze of the software (happend back in december or even november) are expected to be compatible. Changing the variables used in the lib would mean all previous installations are now incompatible with the newer version of the lib.

So that is why i say that the feature freeze affects us. (I did not mean to imply that there is a feature freeze in the lib.)

[poeschlr]

And as i stated above, changing the 3d path in the libraries will not help with old projects! The footprint is embedded inside the pcb_new file. This means installing the old lib with a different variable name does not fix the outlined problem at all! You need some way of updating the footprints for that to work. So it might be easier to change the KISYS3DMOD variable depending on what project you opened. (Opening an old project, point it to the location where your old 3d files live. Opening a new one, point it to the new location.)

Alternatively you would need some sort of script that updates the pcb_new files of your old project. (Last time i checked the update footprint dialog inside kicad did not update footprints if they only differ in the 3d path settings. So there might really be a need for an external tool.)

---

About performance concerns with regards to running virtual machines: I found that running kicad inside an oracle vmware setup is well enough for everything i can throw at it. It even runs better in there as my guest system is ubuntu which has better support than my host system fedora. Not sure how well the visualization support is if you run windows as your host. (Linux has kernel level visualization support for processors that include visualization extensions.)

[poeschlr]

@KarlZeilhofer this might be of interest to you: https://lists.launchpad.net/kicad-developers/msg35545.html

[KarlZeilhofer]

Configs in parallel

@Ratfink Is the flatpak any better than using AppImage? In my current setup on Linux Mint 18, I've installed V4.0.7 via PPA and the nightly via AppImage. Using the AppImage doesn't separate the .config folder - that is used normally as KiCad nightly would be normally installed.

But with the new feature, which allows a custom XDG_CONFIG_HOME to be set on any operating system, this will be solved anyway.

Update existing projects

@poeschlr Yes, as you said, a script would be necessary to update the 3D models only. I did a "update all footprints" on a larger project, and the side effect is, that also all silk screen labels gets reset to their default position, that's unacceptable. EDIT: refer to https://github.com/KarlZeilhofer/kicad4-footprints/issues/1

What about the python console, which is integrated in PCBnew? I've never used this tool... I'd like to provide a script update_all_3d_models which relays on the local and global fp-lib-table.

[poeschlr]

An update all 3d model script might be useful for other usecases as well. Sadly i never wrote a python plugin.

In kicad 5 you can even write it as an action plugin that allows it to be callable via the tools menu https://forum.kicad.info/t/howto-register-a-python-plugin-inside-pcbnew-tools-menu/5540/13

[KarlZeilhofer]

@poeschlr what do you mean by "plugin"?

I'd like to provide it for KiCad 4 also. Since the integrated python console seem not to be very mature for V4.0.7, I'll write a standalone python script. Is there any standard way how to get the paths to the config folder on all operating systems? Is there a python script available, which parses a kicad_pcb file into pythons standard containers?

[Ratfink]

@KarlZeilhofer Yes, flatpak changes the XDG paths for applications so they don't interfere with one another, as described here.

[KarlZeilhofer]

@Ratfink great to know, but we are looking for an OS independent solution, which seems to be available soon.

[KarlZeilhofer]

I wrote that models updater in May, but forgot to post it here: https://github.com/KarlZeilhofer/kicad-models-updater