Removal of the divers.pretty, discret.pretty and labels.pretty repo breakes kicad stable

[poeschlr]

As mentioned in https://github.com/KiCad/kicad-library/issues/1358#issuecomment-311181098 it seems cvpcb gives an error message because of the missing repos. (I don't think it does crash completely but unexpirienced users might think it is broken.)

Forum entry: https://forum.kicad.info/t/cvpcb-crashes-on-deprecated-footprint-library-files/6872

My suggestion would be to recreate these repos with one dummy footprint in them. Can we disable forks and pull requests for specific repos?

[SchrodingersGat]

Ah, that's very frustrating. Such a silly way of managing libraries. I'll recreate these for now as the path of least resistance.

Thanks for pointing this out.

[SchrodingersGat]

Alright, fixed :)

[tijuca]

Hi,

I was hitting by such renaming and reorder within a stable release too while working on the Debian packages of KiCad between 4.0.4 and 4.0.5. as seen in retrospective. Please don't do that within a stable release cycle as I and the users too don't expect such a behaviour within a stable release. Related to that we have a now a potentially broken 4.0.5 within the Debian Stretch release. 😞 Also some trees haven't a tagged version. By this we packaging the HEAD off the date we collect the libraries which isn't necessary the date a release is tagged or happen.

Taking care on such upstream changes takes much energy on the packaging side which cost at least a lot of time. I guess there is some more policy needed to avoid such issues. Classical libraries are expected to have a stable API between releases. I see not much difference to the footprint libraries.

Sometimes renaming is inevitable but please give it some attention in release notes in the future.

[poeschlr]

Do i understand you correctly that we need to ensure that all 4.x releases stay compatible with the current lib? Just give your users the option to update to the newest stable. Otherwise the lib comes to a standstill.

[poeschlr]

To elaborate more: Yes we know a release cycle for the lib might be a good idea in future. But with the current state that footprint libs are downloaded directly from github, this is not really possible. (It doesn't help either that all footprint libs live in separate repos.)

For this reason, there will always be inconsistent libs as soon as we change anything in the footprint libs. We can only hope that in kicad 5 this will change.

[tijuca]

The problem is not really the dynamic of the libraries, it's more this it's happen "quietly" or at least not visible without a why this was done. The GitHub platform is not the best form for communicate of final information spreading. This should be done (also) on the mailing list in a collected form. KiCad is not the only project I'm involved and it's simply unfortunate to go through all the various GitHub issue trackers to collect the information. Email is for me the better medium to get all the information, simply because the information will be pushed to me and I don't have go over various websites.

I hope the fp-lib-table usability will be more flexible with KiCad 5 and later, the current solution could and should be improved. It's really annoying that I need to adjust my libraries by every new release, users don't want to fight with that within a release. KiCad is a really nice piece of software but useless without the libraries.

[SchrodingersGat]

@tijuca it is very difficult at the moment as there are over 90 .pretty libraries. To prevent "errors" such as the one you have encountered, deprecated libraries must be retained.

This is quickly becoming untenable.

Ideally, all .pretty libs must be placed inside a single repository. This however goes against the concept of "live-loading" library content from GitHub. (Note: I do not know of anyone who uses KiCad in even a semi-professional capacity who uses live GitHub library data. IMO it's actually a drawback to the KiCad software to have such a feature).

In any case, this is the situation until we can convince the devs to drop this feature or work out a compromise.

[pointhi]

I think kicad should go away from the "live-loading" principle as done yet. It slows down KiCad (and requires internet connection all the time).

My Idea would be to stitch the Github wizard, and replace it with native git support (means libraries are only updated via git, and stored on the harddrive with full history. Instead of fetching it every time from github when someone needs a footprint). The same with the schematic library. @SchrodingersGat, I think you agree, splitting the schematic symbols into repos as done for the footprints is not an option. As you suggested, working towards merging all .pretty repos into one big repo, as done for 3d-models would be the prefered way. But that needs support of such a library structure inside KiCad first.

I proposed some idea going in that direction a few days ago: https://forum.kicad.info/t/idea-to-allow-nested-footprint-modules/6912

In any case, this is the situation until we can convince the devs to drop this feature or work out a compromise.

I think, instead of working on compromises on the current system, we should develop in cooperation of the community, library maintainers and devs a new library system. With the new schematic symbol format there would be already some overhaul of the library management system. Such work could be placed into the Kicad 6 Road Map as well.

[SchrodingersGat]

My Idea would be to stitch the Github wizard, and replace it with native git support (means libraries are only updated via git, and stored on the harddrive with full history. Instead of fetching it every time from github when someone needs a footprint).

I have played around with this concept for a while, and sketched out some ideas of how it might work. In the end, I think that it adds a lot of unnecessary complication. If users want to have up-to-date libraries linked from GitHub, there's already a tool for that: git

If users want to use some other revision control tool, they can use that too. Including native git functionality seems to me to be a waste of (limited) dev resources. Would you agree?

I think that there should be "versioned" downloads of the libs (symbol lib, footprint lib, 3d lib) available on the website, along with clear instructions on how to access the up-to-date libraries using git.

[jkriege2]

Hi!

from the POV of a user, I would like to have something along these lines:

1. When I install, KiCAD comes with a current (or current at the time the version was built) set of symbols/footprints/3D models
2. When I work with KiCAD there should be no Internet connection required (also: yes it's very slow at the moment, when GIT is accessed every time) ... also think of a stick install, when going on travel ;-)
3. I would wish for a button somewhere in the menus, where I can say: Please update my libs. Ideally I then get a list with all changes, or at least which libs have changed and can possibly select only a subset (by default I think all changes should be accepted, so clicking OK will start the update process).

This way, installing KiCAD, I can start to work and later on I can also add new repos. Ideally it would also be possible to add other repos for libs (e.g. my private GIT repo) and the same download functionality is given. This way one can alsomanage ones lies somewhere (e.g. a local GIT repo that can be in any branch at any time) and I can get a current trunk-version from that at any time (into a second location). As an alternative it should be possible to add repos by path only, explicitly without the update option.

To implement such a system, one would need some type of downloader. I think there are several options there:

1. simply download a ZIP from Github, but this means that we have to change the software whenever Github decides to change its API, or even have real trouble, if Github should vanish at some point in the future.
2. The IMHO better option is to distribute a basic GIT with KiCAD (or require the user to provide a path to GIT, the Windows installer could e.g. then at the end point to a suitable installer for GIT). This way one could pull or export from ANY git repo (maybe the GIT could be encapsuled as plugin with a basic API, so other people could provide plugins for other version management systems). If the repos are then kept as full GIT-repos, the drawback is additional memory for the history, but the user can easily use them as repo also outside KiCAD, also deleted footpritns/symbols would not remain visible in the lib, as it is at the moment with KiCAD. If they are kept as export (i.e. without history), this saves memory, but makes update a bit harder, as one will have to pull the full repo and then delete the old and replace it with the new contents (to ensure old files do not exist any longer. This may also have another drawback, as possibly user-files are deleted, if the user puts them into the KiCAD main repos (which he shouldn't but I fear many will do).

What do you think? Did you thoughts go along a comparable line of thought?

Best, JAN

[poeschlr]

I agree with oliver here. (Edit: and with jan. I was a bit too slow in answering.) The idea of easily downloadable "versioned" libs is compelling. Github allows for such releases.

To get something like this we only need to get the developers on board to enable merging of all footprint libs into one repo. We also would need support from the documentation team and the webside maintainers. (This does not mean they need to do all the work, but they have experience that could help with this effort.)

I think the lib should also have its own release cycle. that does not necessarily need to be tied to the software cycle. (Example: a new release after a major update like the recent updates to the 3d lib.)

Somehow the lib version should maybe also be stored in a project. So that if you share your work, other users can easily find out with which lib version it has been created.

[jkriege2]

Hi @poeschlr ! I'm not so sure about the lib releases ... usually people will submit something to the libs and expect it to be available in KiCAD, as soon as the contribution is accepted, right? With a release-cycle, that would mean waiting for the next release .... also the parts are usually usable even as a single file, so no need to ensure a certain state of the full lib.

A way to overcome this would be: There are releases and and a hidden file on the GitHUB-Repo (or the KiCAD-website) lists all versions. then KiCAD could provide a method to download such a release (or parts thereof, as I think the full lib will be quite large and user might want to update only part of it to save download capacity). If then a user wants cutting edge, he COULD download by himself ...

But then we are back to handy-work if you want to have current libs ... which I think would be a big drawback.

Best, JAN

[pointhi]

@jkriege2, my idea about git integration would be by adding standalone git support into kicad: https://libgit2.github.com

github releases then can be managed using git tags (which would also work with private repositories). Users can also specifiy which release (or commit version) they want to use for their project, and therfore can work with a specfic version of the library on a project which does not change as long as there is no need for it.

[SchrodingersGat]

+1 to libgit2, this is what I have been looking into.

The best way to tackle this would be to extend the current concept of a "library", enabling it to be tracked and updated. Legacy libraries obviously could not implement this.

However, there is still a problem that I haven't yet been able to work around:

Footprint libs (.pretty) are added individually to the lib-table file. However we want to merge all .pretty libs into a single GitHub repository. How does the user specify that they want to use only a single library from a github repository? Splitting the libraries into separate repos helps here, but that's obviously very messy.

Any thoughts? It seems like a tough problem to solve.

This is what makes me think it it simply easier to have three repositories and users can keep up-to-date with existing git tools if they want to.

[SchrodingersGat]

simply download a ZIP from Github, but this means that we have to change the software whenever Github decides to change its API

It's already a problem - GitHub have changed their API such that the 3Dshapes downloader doesn't work for directories with more than ~500 files.

[pointhi]

How does the user specify that they want to use only a single library from a github repository?

Well, I could think about different solutions. But one thing is the same: the user is forced to clone the complete repo even he only needs one directory. That shouldn't be much of a problem, because footprints require way less memory than 3d-models.

• Support github url's with sub-directories (like git@github.com:KiCad/kicad-footprints.git:Capacitors_SMD.pretty) There is probably no standard for such url's, and it could be going to be messy
• distinguish between the repository and it's included libraries (and allowing to enable/disable libraries inside that repo) I think the better way to handle libraries. Also for distribution

[jkriege2]

@SchrodingersGat ... hhmmm ... I only see two options:

1. We keep separate repos + one repo that only contains the lib-tables ... but the we still have to manage all those separate repos. Having GIT-submodules in that major-repo will also not really help, as they are by default not meant to be edited INSIDE that major repo. The docu says you can change them so they can be edited from within the source-tree, but I doubt there is simple support in Github, so we might loose all non-GIT-experts from contributing ... and make it very hard for the rest ;-) Submodules are more meant to import a special tag/version of another lib into the own source-tree and have easy update of them... as far as I interpret them from a practical POV.
2. We put everything into a single repo and then somehow indicate that this repo has a split-structure (e.g. by a config-file on the repo-root), e.g.:

   ROOT
    |
    +-- SYMBOLS
    |       +-- SymbolLib1.sweet
    |              +-- SUBDIR1
    |                     +-- SymbolFile1
    |                     +-- SymbolFile2
    |              +-- SUBDIR2
    |                     +-- SymbolFile1
    |                     +-- SymbolFile2
    |              +-- SymbolFile1
    |              +-- SymbolFile2
    |       +-- SymbolLib2.sweet
    |              +-- ...
    |       +-- SymbolLib3.sweet
    |              +-- ...
    |
    +-- FOOTPRINTS
    |       +-- FootprintLib1.pretty
    |              +-- SUBDIR1
    |                     +-- FootprintFile1
    |                     +-- FootprintFile2
    |              +-- SUBDIR2
    |                     +-- FootprintFile1
    |                     +-- FootprintFile2
    |              +-- FootprintFile1
    |              +-- FootprintFile2
    |       +-- FootprintLib2.pretty
    |              +-- ...
    |
    +-- 3DSYMBOLS
    |       +-- ...
    |              +-- ...

   Then a download COULD try and download only some subdirs ... This seems to be called "sparse checkout" in GIT (see https://git-scm.com/docs/git-read-tree#_sparse_checkout and https://lakehanne.github.io/git-sparse-checkout ), i.e. in GIT>1.7.0 there is support for a strategy like that, which could be used by the downloader in GIT ... but is this feature supported in libgit2 (I think no: https://github.com/libgit2/libgit2/issues/2263)? So this leaves us to haveing a downloader that does use libgit maybe only to get a directory listing and then downloads by itself ... not good, but might be a solution?

3. We leave repo-structure as it is now and somehow try work with it, which is also not really a good way, especially when that would mean to have separate repos also for 3D symbols and later symbol-libs ... (repo hell ;-)

Best, JAN

[jkriege2]

Some more URLs of interest:

• https://www.atlassian.com/blog/git/handle-big-repositories-git

[poeschlr]

@jkriege2 wrote:

I'm not so sure about the lib releases ... usually people will submit something to the libs and expect it to be available in KiCAD, as soon as the contribution is accepted, right?

This is currently only true for footprints. For everything else the user needs to wait for the next release or download the lib by hand. I'm not against giving the users this option. But users should have the option for a version based release. (Or at least control over when their "local" libs update.) And i think for the vast majority of users, a version based release of the libs might be the better option.

[SchrodingersGat]

And i think for the vast majority of users, a version based release of the libs might be the better option.

I would imagine that it is also what most users expect. Newbie KiCad users only use the GitHub repo because it is what is presented to them by default. AFAIK other major EDA packages distribute library "blobs" (I suppose that's all they can do as they don't support user contribution).

[SchrodingersGat]

@poeschlr @jkriege2 KiCad project lead Wayne has indicated that he does not want the .sweet libs to be hosted on GitHub as separate repositories - he is expecting them to be housed in a single repository.

In addition to this being yet another different way of managing library files (yay!) it means that no matter what happens, distribution via GitHub will require a distinction between a library and a repository.

Honestly I can't see a clear path here that is satisfactory (beyond nixing the whole GitHub plugin concept).

KiCad currently supports multiple library formats:

• Native (.pretty)
• Legacy (.mod)
• Eagle
• GEDA

Note that all these are file format differences and really it doesn't matter how they GET to your computer. (local storage, networked, USB drive, etc, etc).

Then, the GitHub library "format" is present as being equivalent to these, but it's not. It's a retrieval mechanism, one which handicaps the usefulness of the current library scheme.

I imagine there being a few key repositories at https://github.com/KiCad

1. symbols - schematic symbol library
2. footprints - PCB footprint library
3. packages3d - 3D models aligned with footprints
4. templates - project templates
5. wizards - wizard scripts e.g. for footprint generation

If a user wishes to keep the bleeding-edge footprints up to date, download a git client and away you go!

I would really like to see a good way of integrating this into KiCad but honestly I don't see a good option yet.

[suzizecat]

IMHO, the best way would be to have, as @SchrodingersGat said, have a repo per library kind. Then KiCAD should be able to download these libs and work with it.

The must would be to be able to choose between downloading a local version of theses libs then have a sweet button that will re-sync local version with the repos ones or always work with updated ones as for PCBNEW.

In example, right now, you have to clone the repo, have a git client and reconfigure KiCAD properly to have that in eeschema while you can have changes without notice in your footprint set in PCBNEW... Neither of these solution seems right to me...

[bobc]

Note that all these are file format differences and really it doesn't matter how they GET to your computer. (local storage, networked, USB drive, etc, etc).

Then, the GitHub library "format" is present as being equivalent to these, but it's not. It's a retrieval mechanism, one which handicaps the usefulness of the current library scheme.

I think that is a key point. The current github plugin has blurred the distinction, and tied us in to one particular implementation. Implementing over the Github API (instead of git api) was a bad idea, IMO. Perhaps at the time, the github plugin was seen as a "quick and simple" way to achieve a goal.

Unfortunately, that has led us down a blind alley, with no real good options for a simple improvement path. I think we have to take a step and look at all the things we would like to achieve, and the best way to do it, rather than further hacking what we have got.

Personally, I would like to see a more flexible approach to externals tools in KiCad. KiCad does not need to know how to download libraries, this could be handled by an external tool (e.g. python script). All KiCad needs to know is there is an external command to call, and after calling it, files on disk may have changed, so the internal caches need to be refreshed.

The framework of pre-actions / call hook / post-actions could be generally useful in other parts of KiCad, for better integration with third-party tools (there is quite an active and growing set of Python tools).

ETA: Arduino were in a similar situation to KiCad, and they have implemented a neat framework for managing third-party components (cores, libraries, boards). A JSON file in a repo describes the contents. Arduino IDE only needs the link to the JSON file to handle installs, uninstalls and versioned updates. I think it would be a useful model for KiCad.

[suzizecat]

Personally, I would like to see a more flexible approach to externals tools in KiCad. KiCad does not need to know how to download libraries, this could be handled by an external tool (e.g. python script). All KiCad needs to know is there is an external command to call, and after calling it, files on disk may have changed, so the internal caches need to be refreshed.

Well, if you gonna use an external script to update the libs, it should simple enough to add this script into the main KiCAD distrib and add a button to call it.

I think that third party tools like python scripts are great and should be integrated into KiCAD, but for such "basic" function (perform an update of official libs), there should be a (newbie-)user-friendly way to do it without hacking into KiCAD.

Don't forgot that KiCAD is an EDA Tool designed for electrical purpose, and shouldn't need any Python skills to run smoothly

[bobc]

I think that third party tools like python scripts are great and should be integrated into KiCAD, but for such "basic" function (perform an update of official libs), there should be a (newbie-)user-friendly way to do it without hacking into KiCAD.

Of course, I agree with that. But such a framework doesn't stop pre-configured scripts being bundled with KiCad. Indeed, there already are python scripts pre-configured in KiCad, maybe you are not aware of them?

Don't get too hung up on external/third party/builtin, these are implementation details. I'm not suggesting for a second that users need to become Python authors. The point is that if there is an extensible framework, it can be used by offical devs or interested third-parties, and can be implemented much quicker than the current 2 year dev cycle.

In all cases, a smooth and seamless operation should be presented to the user. The "paddling under the water" can, and should, be hidden.

[SchrodingersGat]

ETA: Arduino were in a similar situation to KiCad, and they have implemented a neat framework for managing third-party components (cores, libraries, boards). A JSON file in a repo describes the contents. Arduino IDE only needs the link to the JSON file to handle installs, uninstalls and versioned updates. I think it would be a useful model for KiCad.

Can you provide some more info on this @bobc ?

[tijuca]

I would appreciate if KiCad would first think about a policy for library releases and lifetime handling for those offline data with the release cycle of a KiCad version. That's in the end this bug report is about, or I'm wrong? There was written a lot of various things for handling dynamically downloading bleeding edge stuff from GitHub or elsewhere. This doesn't really helps the distributions and first time users at all. I'm sure that really experienced user know how to add some extra library to KiCad. Try to think like a typical user which will be happy with the libraries which are mostly already provided. Think about updates and fixes for the existing libraries and long waiting pull request for this. I'm quite demotivated by this situation. It's like for all free and open projects, don't let patches and pull request wait to long. Talk to submitters so they feel integrated into the project.

What about one main git tree which includes all the approved stuff for the next release like done in the linux kernel? Some kind of submaintainers for various kind of library types are responsible for their libraries and the pull requests into the main release tree. Instead of tagging over 90 trees for a release it would only be needed in one repository. You could even pull daily zip archives of this tree for online updates.

Also keep in mind that not all users have directly and free access to the internet, they are depending on data provided by setups, no matter which OS they use.

I'm quite sure that it is not that difficult to handle some KICAD_FOO variable for pointing to the root directory of libraries. Also the download of data from the internet is not a real technical problem I think. But please first think about how handle the release of libraries together with KiCad releases. Thanks!

[SchrodingersGat]

@tijuca thanks for the feedback:

Currently the libraries are released in parallel with the KiCad software. For example, here are the releases for the symbol libraries. These get packaged into official software releases. If you want "stable" library data, download one of those release packs - they won't change.

If users wish to have the up-to-the-minute libraries, then they also have to accept that there is a risk that their symbols and footprint data will change. There is no getting around that.

However, a lot of users (especially new users) don't realize this disconnect. The forums are full of comments indicating that they think that the GitHub libs are the "only way" to access library content. I have tried using the GitHub integration once, just to see how bad it could be. Very bad, was the answer.

Personally I track the github libraries and manage them externally with available git tools. This way I can pick and choose when I update the libraries, and can also have my own library parts in there too.

[tijuca]

@SchrodingersGat

Currently the libraries are released in parallel with the KiCad software. For example, here are the releases for the symbol libraries. These get packaged into official software releases. If you want "stable" library data, download one of those release packs - they won't change.

This is no question, my point about stable release policy is simply about allowed name changes and transitions for existing libraries. Between KiCad 4.0.4 and 4.0.5 the following libraries got reordered, mostly renamed (maybe I haven't found all of them).

    Buttons_Switches_ThroughHole.pretty -> Buttons_Switches_THT.pretty
    Capacitors_ThroughHole.pretty       -> Capacitors_THT.pretty
    Connect.pretty                      -> Connectors.pretty
    Diodes_ThroughHole.pretty           -> Diodes_THT.pretty
    Display.pretty                      -> Displays.pretty
    Relays_ThroughHole.pretty           -> Relays_THT.pretty
    Resistors_ThroughHole.pretty        -> Resistors_THT.pretty
    Sockets_BNC.pretty                  -> Connectors_TE-Connectivity.pretty
    Sockets_Mini-Universal.pretty       -> Connectors_Mini-Universal.pretty
    Sockets_WAGO734.pretty              -> Connectors_WAGO.pretty
    Terminal_Blocks.pretty              -> Connectors_Terminal_Blocks.pretty

KiCad 4.0.5 got new libraries.

    Enclosures.pretty
    Housings_BGA.pretty
    Housings_LCC.pretty
    Housings_LGA.pretty
    Housings_PGA.pretty
    Inductors_SMD.pretty
    Inductors_THT.pretty
    Modules.pretty
    Transformers_SMD.pretty
    Transformers_THT.pretty

The latter is no real problem and I'm fully o.k. with that. But the former can be a problem if distribution don't know about such changes. Like said for classical libraries they never should break the API within a release cycle I see this need also for a KiCad version cycle. I'm only a bit disappointed that all these changes are not really communicated on the KiCad website nor on the mailing list, I haven't found some usable information on kicad-lib-commiters. So it was a long puzzling for me to get the correct fixes into the packaging. Thanks to the possibility for real symlinks on Linux systems we have solved the moved libraries by symlinking the old name to the new name. But this is only a workaround. What if a new library with the old name will be created? The fp-lib-table file of the user are still using the old names and it's no real solution to say to the user "Hey, you need to delete that file and do the whole setup again!".

So what to do? I suggest to think twice about renaming libraries! As written, we can mor e or less easily "fix" this by doing some symlinking but as a user I don't expect that libraries are renamed within a release cycle. I haven't seen anything similar in proprietary and professional software releases.

Adding new libraries is no problem, so for existing libraries it's better to copy them into the new library and mark the old library as deprecated and no longer maintained. Maybe a flag that KiCad can use to display the deprecated warning can integrated in the library that are affected. So users will be warned and they should use the new library instead. But they don't need to do a setup from scratch again.

Talk (not only) to the distributions and announce such changes on the mailing list, that's what ML are made for and typically used. Don't expect that packagers are following every new entry in the issue tracker, they are normally "only" packaging the stuff. Also use the possibility to add a announcement about the changes in the libraries on the KiCad website. Distribution happily take a URL to guide users to the upstream information.

To make a clear statement, I really respect and appreciate the work you library developers are doing and I don't want to blame you. I just want to point out the problems distributions may have to package KiCad libraries.

Thank you!

[SchrodingersGat]

@tijuca I can appreciate the problem.

If all footprints were in a single repository then this would not be an issue, I feel.

[SchrodingersGat]

Marking this as solved, we are well on the way to the new footprints repository with integrated fp-lib-table file.