search

KiCad / kicad-symbols

Official KiCad schematic symbol libraries for Kicad 5
https://kicad.github.io/symbols
Other
706 stars 745 forks source link

It seems the new file format is now available. I suggest we start our testing phase. #2697

Open poeschlr opened 4 years ago

poeschlr commented 4 years ago

https://forum.kicad.info/t/new-schematic-and-symbol-library-file-formats-are-now-the-default/22655

I would like for us librarians to experiment with the new file format. The first thing would be to try and convert our library over and report any issues that we notice (For now just in a local copy of the lib as i supsect there will be bugs)

Once we are confident that the conversion works we might want to branch off a version 5 library and use the master branch for version 6 development. The v5 branch should then only receive major fixes. If we assume we need only 2 month for testing then we should be able to have a v6 lib ready for the planned v6 announcements at KiCon.

I also suspect that we will need to adapt the KLC to the new file format.

And we will need to redevelop our scripts.

chschlue commented 4 years ago

FYI, I'm in the process of pushing converted libs to https://gitlab.com/chschlue/kicad-symbols/-/tree/v6 and am planning to play around with them afterwards.

stambaughw commented 4 years ago

FYI, you can load both versions of the library by adding a new entry in either your global or project library tables by just giving the library a new library nickname to the new symbol library file format. I used a simple "-sexpr" suffix during development so I could easily tell whether I was working with a new or legacy file formats. In the short term, you will be able to "Save As.." between file formats but as soon as we add a new feature to the symbol library, saving to the legacy format will be deprecated and you will have to use the 5.1 branch to edit legacy symbol library files. Thanks in advance for the help testing.

chschlue commented 4 years ago

FWIW, my current setup is having my "normal" repo in, say, ~/kicad-symbols and an additional v6 worktree in ~/kicad-symbols-v6 with 5.99 KICAD_SYMBOL_DIR set to ~/kicad-symbols-v6. Don't know if that's a sensible way to be able to work with both in parallel yet.

poeschlr commented 4 years ago

I personally have a project with the libs added as project specific but with a prefix to the nickname (i use prefix 00dev_). Works well for footprints. Have not yet made such a setup for symbols.

chschlue commented 4 years ago

Should be working. Anyone care to pull and check for errors?

chschlue commented 4 years ago

@stambaughw is there any kind of central discussion thread on format changes? I have only been able to find bits and pieces on GL, LP, the forum and so on so far.

chschlue commented 4 years ago

To clarify: The symbol spec at https://docs.google.com/document/d/1lyL_8FWZRouMkwqLiIt84rd2Htg4v1vz8_2MzRKHRkc/edit seems to be well beyond current UI capabilities, for example, and I couldn't find anything regarding footprint library changes (FPs are already S-expr, of course)

stambaughw commented 4 years ago

@chschlue The symbol file format discussion was done a very lone time ago and the symbol library file format is based on that discussion with a few minor changes to add things that were missed.

The new file format features for which there are no UI access in symbol library editor will be implemented as time allows. The initial goal is to get the current feature set stable and start adding the new features from there.

chschlue commented 4 years ago

FYI, I'm in the process of pushing converted libs to https://gitlab.com/chschlue/kicad-symbols/-/tree/v6 and am planning to play around with them afterwards.

It looks like GL doesn't allow me to make that repo publicly accessible as long as it's a fork, so I re-pushed to https://gitlab.com/chschlue/kicad-symbols-v6.

BTW, are there plans to actually give write access to the libs on Gitlab to mortal librarians?

poeschlr commented 4 years ago

It does not make sense to transition to gitlab until we have CI running over there. So i guess we will stay on github for a while

chschlue commented 4 years ago

Well, being able to tinker around over there, set up demos like the one above, get used to the UI and such does make some sense to me.

Or do you prefer doing everything on your own?

poeschlr commented 4 years ago

You asked when we will open the official gitlab versions and the above is the answer to that question. (we can not start before CI is working)

Edit: the reason we can not really write to that repo until that point is that it will otherwise get quite hard to sync github and gitlab. And we really want to do this only once to be honest.

chschlue commented 4 years ago

We don't have to actually use the future "production" repos. The ones we use for testing won't even have to be publicly accessible if that helps to avoid confusion. But it looks to me like we'd need some actual data for v6 scripts to work and be tested on (like the converted libs I provided above) and I don't get why everybody should do every step all over again, so why aren't we going to do it under the KiCad org's roof over at GL?

poeschlr commented 4 years ago

I am prepared to go along with a move to gitlab, but we really need a benefit that makes it worth the effort. (Github and gitlab basically have the same feature set. So the benefit to our workflow is questionable. It will however definitely require considerable effort on our part.)

Currently, i feel we don't really have the resources to handle our normal workload. I struggle to see a way to handle a transition to gitlab on top of preparing for v6 and still handling pull requests. Especially if we want to invest time into creating a ruleset for the new more powerful file format (and i guess that it would mean manual work if we want to make use of new features).

We also still have work left over from the v5 reorg. I would for example love to see the logic libs, relay and switch libs being cleaned up (getting rid of invisible power pins would be the minimum). If there is time converted into fully specified libs (such that they finally fulfil the current KLC). The QFN/DFN and SO libs also still have a lot of work to be done (would be awesome if we would have all of such footprints scripted). Both these libs might need to be split to make them future proof (requires touching quite a lot of footprint fields which i feel can really only happen with a major release). The capacitor lib has a similar issue.

Also consider that synchronization between gitlab and github is just a bot that needs to be triggered manually for a single sync point. Originally i thought that there would be full synchronization of pull requests, comments and the repo state (i basically misunderstood @sethhillbrand). Without full sync we can not really do a slow transition. Instead, we would realistically need to shut down github on the same day we transition.

For reference: The v5 release preparation was about 2 to 3 man-years (split between all librarians). All of that was on top of handling normal pull requests. The v6 release will not be that much work as we do not need to do another full reorganization (the lib has a decent structure already for the most part). But i still suspect it will be about half of what the v5 release was (at least if we want to at least finish what was started with v5 as well as using the new file format to its full potential).

poeschlr commented 4 years ago

@chschlue i also don't understand why you should not have access to the library repos on gitlab. You are a member of the librarian group and that group is a member of the library repos. I elevated you to maintainer in the librarian group maybe this helps

chschlue commented 4 years ago

I don't know all the details of Gitlabs permission system. What I know is that I wasn't able to open / close / edit MRs before or anything like fiddling around with Gitlab CI. But that's not important anymore if we finally aren't going to transition to GL for v6.

I just figured there can't be any real gradual transistion to new formats anyway, so we were going to freeze v5 libs at some point and start only accepting v6 contributions. In that case, open PRs would have to be converted and re-pushed anyway so we might as well ask to push to another repo at the same time (provided CI is up and running by then.) At least that's how I understood team discussions until now.

Anyway, I think we need to lay out a more detailed plan than just "start our testing phase" pretty soon if we're supposed to have v6 preview libs ready by September.

Are we going to freeze v5 after 5.1.6's release and only provide bugfixes for 5.1.7 so we can mainly work on v6 in the meantime? If not, I don't think we'll have remotely enough time to get anything up to speed by autumn.

chschlue commented 4 years ago

What I think needs to be done on the CI sid: (correct me if I'm wrong)

  1. Be able to parse S-expr symbols (mods are already S-expr)
  2. Adapt rules and scripts to fit v6 capabilities
  3. Transfer from Travis to GL CI

Two appears to be the most time consuming by far, which is why I don't really get that 3 should be the part that's going to break our necks. I'm not keen on transferring to Gitlab by hook or by crook, it just doesn't look like it gets any smoother if we're going to do it at some arbitrary point between v6 releases.

poeschlr commented 4 years ago

I doubt we will freeze right after 5.1.6 but do not believe we can wait till 5.1.7.

Tasks that i think need to be done before that:

poeschlr commented 4 years ago

I'm not keen on transferring to Gitlab by hook or by crook, it just doesn't look like it gets any smoother if we're going to do it at some arbitrary point between v6 releases.

If we can get it done then i am ok with it. I would however give it a quite low priority.

Also symbols are only half the picture. There is no reason why footprint or 3d model pull requests would need to be interrupted. So a transition to gitlab would definetly negatively impact this part of the library.

And yes until recently i had planned that we combine the v6 release with the transition. However, i really want to prioritize getting another massive spike in library quality for our users. We can do changes during such a major release preparation that we can not do during the maintenance phase (we should use this opportunity as best we can).

chschlue commented 4 years ago
  • [ ] find out if we have missing features to do our work (are there some symbols that can not be represented in the new system)

That's the main reason why I pushed a simple conversion to https://gitlab.com/chschlue/kicad-symbols-v6. It would be great if people could just pull that, skim over it, try to use it with their own project they happen to have lying around on a 5.99 nightly, do whatever, just too weed out obvious issues as a first step. I can also push that to GH if that makes it easier, there's just no point in having everyone waste time by converting the stuff.

BTW, v6 sym format allows for pin renaming, swapping, adding, deleting as well as generally overriding or deleting properties for derived symbols (formerly known as aliases), so it's pretty flexible, there's just no UI in the editor as of now.

poeschlr commented 4 years ago

It seems there are now two new flags supported in the file format. There is now a "don't include in BOM" and "don't include in board". I assume we will use these for at least logos and similar pure pictographic stuff and possibly also for power symbols. See for example https://forum.kicad.info/t/schematic-symbol-exclude-from-board/23324/