Add metadata to libraries

[kuro68k]

Libraries should have consistent metadata added to each component, to make BOM generate easier and more consistent.

Apologies for not submitting a pull request, but it seems that you can't for wiki entries.

There is some discussion here: https://forum.kicad.info/t/default-manufacturers-part-number-field-in-kicad-libraries

I propose adding the following to the wiki page on library conventions. This will allow BOM generation plug-ins to be much more compatible and uniform, and save users time editing metadata manually. The goal would be to have the BOM generated directly from the schematic with minimal editing afterwards, and to facilitate tools like automatic order generation at suppliers.

To this end, metadata must be consistent and predictable. Normalization is key.

5. Symbol metadata

1. Value should be ~blank~ R/C/D/etc. for passives like resistors and capacitors, or the primary differentiating factor for ICs (e.g. 32k for a 32k variant of a microcontroller).
2. Add an ~"MFP"~ "ManufacturerPart" hidden field with the manufacturer's part number.
3. Add an ~"MFN"~ "ManufacturerName" hidden field with the manufacturer's name.
4. Add a "fit" hidden field, set to "true" for things that require parts and "false" for things like jumpers, solder pads, markings, edge connectors etc. that don't generate any items on the BOM.
5. Add an optional "data" hidden field with relevant data on precision, voltages/current limits etc. Should be omitted for generic parts.
6. Add an optional "notes" hidden field with any relevant textual notes, e.g. why pins were swapped.
7. Empty fields should have a single underscore ( _ ) to prevent them being stripped out.

[ghost]

@kuro68k: "Value should be blank for passives (...)" If you left Value field blank, it won't be possible fast editing with just a double click on schematic. You will not see then where there a field is!

As regards to the demands in 2, 3. Such an approach requires a change in thinking and the transition from universal symbols for specific symbols. At the moment, it is not rather possible to push through. (I've tried).

[TotalKrill]

Good initiative!

Might just suggest that a default value for MFP and MFN is decided upon. Maybe just leave blank or something for components where this is not as important or for people who do not wish to use them. (Resistors, capacitors etc). So that people who contribute is not forced to fill these fields, but they are there to be filled correctly by somebody else at a later time.

Also, do not leave passives value empty. (shortcut key V is a quick acccess to the value field)

[kuro68k]

@keruseykaryu: Good point. Okay, revise values for passives to be as they are now, e.g. just R for resistors, C for capacitors etc.

@TotalKrill: I think blank for generic parts would be a good idea. You could go with "generic" or something but I don't see much benefit to that. Being blank would allow BOM generators to fill in with, say, "none" as required.

Generally speaking I think it's best to avoid language specific terms where possible. Again, BOM generators can handle that, e.g. translating the fit flag into "fit/no fit" for English, 「ある・なし」 for Japanese etc.

[kuro68k]

JoanSparky suggested filling blank fields with a single underscore ( ) to prevent the component editor from stripping them out.

[jkriege2]

Hhmmm ... I'm not so sure how good an idea this proposal is ... Please consider the following points:

1. Who is supposed to provide the data? And if it is there ... who is supposed to check it?
2. Who tracks changes?
3. Which part numbers/manufacturers should be put in? E.g. LM741 is available from TI, Intersil, ...
4. How do you guarantee consistency of the manufacturer names? Say someone write "TI" someone else "Texas Instrument", a third person write "Texas Instruments Incorporated"? This calls for a relational database scheme, which seems a bit out of the scope of KiCAD and its libs (currently I would say there are more pressing things in the libs to do). ...

Just a first set of thoughts on that ...

Best, JAN

[jkriege2]

One more thing: IMHO, the names should be more readable, not just an abbreviation, but full words, which makes it easier for newbies and people not using this system to identify what these fields are, e.g. ManufacturerName instead of the rather cryptic MFN.

JAN

[SchrodingersGat]

While I appreciate the desire to have these extra data present in the default libraries, I see some potential issues.

Consistency

In addition to the good points made in the forum and above, there are some further concerns regarding part consistency. e.g. many ICs have dozens of part-name variants (within a single manufacturer, and many more if you consider alternative parts from other manufacturers). Some variants are functional, some are ratings (e.g. Automotive Temperature) and some are non-functional such as PbF, packaging options, etc. So for a single IC you either specify many, exact MPN values or you specify a somewhat generic version which is essentially what the libs have now.

Further, point 4) that @jkriege2 made above is a good one.

Many companies keep track of internal part numbers instead of MPN values, and so would this be of any benefit to them?

Aliases

The current ALIAS functionality of the libraries would not work with this scheme, as the part fields are stored in the SYMBOL (.lib file) and not the PART (.dcm file). Any ALIASed parts would take on the extra fields of their master symbol, and so many of the existing library components would break.

Library Format Upgrade

The symbol libraries are going to be updated to the .sweet file format soon^. From what I have seen, this will provide much greater flexibility in terms of the ability to decouple the graphical symbol from the part description. This is a must-have feature for adding custom part fields. Perhaps this idea should be held until that is implemented?

^For certain values of "soon"

DNF Field

I like the idea of a DNF field but I think it should be extended beyond True/False, and allow configuration management by specifying the board configs in which a part should be marked as fit/dnf. This is how I manage this problem in my BoM Tool. The DNF field should really be a default field in KiCAD software rather than an "extra" field in the part lib. Hopefully this is addressed in the upcoming .sweet file format.

Data Field

To start, I think that the "Data" identifier is very generic, it would be better labelled as "Rating" or similar. However, the purpose of this field is still pretty opaque given it has to cover all parts. A "simple" capacitor has voltage rating, temperature coefficient, ESR, etc. FETs have various voltage ratings, capacitance values, etc, etc. How do the librarians decide which params are important enough to go here?

Ideally, I would like to see the (new) library format contain a "Ratings" section which is a list of key-value pairs, each of which could then be searchable within KiCAD. They could be compressed together into a single field for the purpose of BOM generation. However this is a "wishlist" item and outside the scope of this issue.

[SchrodingersGat]

Check out the final page of this document (https://lists.launchpad.net/kicad-developers/binkLP4zB3dYN.bin) for a peer into the future, which looks like it should be much more suited to these data.

(I had to rename this to a .odt to open it, YMMV)

[kuro68k]

@SchrodingersGat, I would suggest generic part names. The user can fill out the specifics themselves. The goal isn't to have the defaults produce a working BOM first time, it's just to a) establish a standard that both libraries and users can make use of and b) lessen the workload a little.

More over, I think the way the libraries are arranged now with aliases isn't optimal. For example, you have a random microcontroller that comes in five more or less identical variants, the only difference being the memory size. At the moment there are five different schematic symbols, but really it should be one symbol and just put the memory size in the metadata. Aside from anything else, then you don't have to delete the whole symbol just to switch to a different variant, in the same way you wouldn't delete and replace a resistor just to change it's value.

You are right about the data field, but it's not actually all that important I think. Libraries won't really make much use of it, it's more for users. They can put what matters to them and their BOM in there. For example I tend to have a default capacitor rating of "ceramic X5R/X7R 16V", and only bother to add metadata to parts that are different. The BOM then has a note says "all ceramic X5R/X7R unless otherwise stated".

[jkriege2]

@kuro68k : Your point on aliases I don't really understand: Currently Aliases do just that (except the fields, but that's due to the Implementation in KiCAD): You have one symbol, but define several ALIASES that only consist of an entry in the ALIASES and their specific description+keywords+datasheet (in the DCM-file) ... Deleteing an alias is therefore lightweight.

About the data field: I see the problem in "They can put what matters to them" ... The question there is: so what should we put in in the libs? Should it be left blank? How should we check bew symbols that are proposed for the lib? Which data should we require and which reject? Should we enforce any format, or just accept anything? In the latter case: What is the use to other users?

About names: Could you provide an example (take e.g. the 741 or 555, which are available from many sources and in many packages, or a simple 74LS00 ...) of how you think the names should be?

Best, JAN

[kuro68k]

@jkriege2: What I mean is, say I have a schematic with a resistor and an MCU. I want to change the resistor's value, I just edit the value. I run out of code space and want to change to a larger memory MCU, I have to delete the symbol, replace it, copy the old reference if assigned, set any metadata and hidden/shown stuff, re-do any field moves etc.

They should work the same way. The alias system doesn't let you swap aliases easily as far as I can see, and seems kinda pointless where the only difference is some numeric value that can be written in.

As for the "data" field, the libs would mostly be blank. They are there for the user to fill in. In fact I'm struggling to think of an example where a library would use it.

I'm not sure we need a specific format for it. It's there for BOM generation, so the two use cases are 1) the user wants relevant information on the BOM (e.g. use 1% resistor) and 2) the BOM generator wants to auto-select a part from a supplier in which case just matching space separated strings should be enough (e.g. "X7R 16V" could search Mouser for the value and the keywords "X7R" and "16V"). The latter might benefit from a simple "/ for logical OR" rule so that "X5R/X7R" will be interpreted are "X5R or X7R" when searching.

Names: The current scheme of using 7400 or 555 is fine, the same as the part names in the libraries. Such parts are more like generic passives than manufacturer specific ICs. If the user wants a particular variant they can fill that in themselves. BOM generators that also order parts will need to have some intelligence to handle those special cases.

[jkriege2]

@kuro68k : So what you want are mostly default-empty fields (data, ...), or stuff that is not currently doable with the KiCAD features (value, swap). So all of this seems to be more of a KiCAD and less of a library topic. Wouldn't it be better to ask the KiCAD developers to add some default fields to KiCAD that simply appear there also when not explicitly defined in the lib. Should somebody define them in the lib, that case will be covered automatically and we should then decide how to fill these fields, when we know which system is implemented in KiCAD.

Also the question of who checks these info (if supplied) atc. are still open ... Please keep in mind that there is currently only a small team of lib maintainers and progress is sometimes relatively slow ... so adding even more work will not really help to improve the general libs ... at the moment.

Best, JAN

[kuro68k]

@jkriege2: No, I'm not saying that. I'm saying let's come up with a standard and stick to it.

For libraries, the main change would be the way the "value" field works, and the way that some parts have many aliases for what are essentially different values.

The reason it needs to be done here is because the library conventions page on the Wiki is the only documentation and would need to change in any event. You do have a point, there is a need for a global standard on metadata that BOM plugins can rely on, but I'm working with what we have.

I don't think there is any need for Kicad to generate these fields. If they are not there then the BOM generator can easily ignore them. The empty fields was simply to save the user effort - it's relatively easy to add them to the libraries ready to be filled in.

I'll for the project and see if I can bulk update some libraries as an example.

[jkriege2]

Hi!

I'm really confused by now ... Could you please - very simply with a few easy examples, say on typical discrete like R and one more IC like 741/555/7400/... - write down what you proposal would mean! Also e.g. consider a microcontroller family such as ATmega88P and varaints (see http://www.atmel.com/images/Atmel-8271-8-bit-AVR-Microcontroller-ATmega48A-48PA-88A-88PA-168A-168PA-328-328P_datasheet_Complete.pdf).

If you say there is no need for empty fields pre-generated by KiCAD, but also say that mostly the fields should be empty in the lib, I don't see what you really want!

The KLC-page defines Standards that symbols for this lib have to (mostly) follow, in order to be accepted. So if you set up these rules, you require people to stick to them and generate those fields (as otherwise symbols without the fields won't be accepted anymore), unless you say they are strictly optional (then that should be in your proposal). Still, even if they are optional, the questions of checking and formatting (also of the different ManufacturerNames) remains, if somebody supplies the data!

For libraries, the main change would be the way the "value" field works,

OK, I think here some examples would really help (see above)

and the way that some parts have many aliases for what are essentially different values.

What would you propose here? A change in KiCAD, or a change how we use the current structure (no ALIASES anymore, as currently they only have the one provided functionality)?

Best, JAN

[SchrodingersGat]

I am also a bit confused. Isn't this functionality already available to the user? In eeschema, under Schematic Editor Options:

If you want blank default fields for all components...

[kuro68k]

Okay, for R:

Library:
Name = R
Value = R

User:
Data = 1% 0.125W

For the MEGA:

Library:
Name = ATMEGA48A-A
Value = 8K
ManufacturerPart = ATMEGA48A-A
ManufacturerName = Atmel

Aliases can be used with the Atmel MCU to generate the other parts in the range, but all they need is metadata changes and the user can easily switch. Ideally they would all be normalized into a generic ATMEGA48-88-168-328A-A part but it's not critical.

For the 555, currently we have an LM555N. Well, the original was the NE555 so IMHO it would make sense to list it under that name, but whatever...

Library:
Name = NE555
Value = [none]
ManufacturerPart = NE555

I omitted the manufacturer name here because so many people make it these days. If you go to Farnell or Mouser and search for NE555 it will bring up compatible variants like the LM555 from Texas.

It's not complicated. All we need is enough metadata in an agreed format to allow BOM generators to do their work.

[kuro68k]

I just made a new part for work:

Value = MCP1640
Datasheet = www.microchip.com/datasheet/MCP1640
ManufacturerName = Microchip
ManufacturerPart = MCP1640T-I/CHY

If I want to use a variant all I need to do is edit the ManufacturerPart field, and maybe the value just to make it clear on the schematic. This way the value isn't overly specific; the part name contains the temperature range and package, things which are irrelevant to the schematic (which explains the function of the circuit) but important for the BOM.

[jkriege2]

OK, slowly I see where you want to go with this ... atill I'm not so sure about some things (from the practical POV of a lib maintainer):

1. If we do that, the fields should be there by default in KiCAD ... otherwise most contributors won't add them (IMHO), or we get a mess of differently named fields (case, typos, ...)
2. Today we don't even use the datasheet field, as it is also in the DCM-files
3. This means we will mostly have to abandon the current ALIAS mechanism, as it does not allow to alter the fields (except the special fields description, keyworkds, datasheet, i.e. F1-3), which I really dislike as it makes a mess of many libs (e.g, transistors ...) ... although I would love to have an alias mechanism that covers fields properly, so we can have ALIASes e.g. with different packages.
4. Still you didn't answer to the question of how to maintain consistency (e.g. ManufacturerName) and who should enforce/check these rules ... I think this will really create a lot of work (already now there is really a lot of work included in reviewing parts)! SO library maintenance and improvement will slow down even more ...
5. Should these rules be optional or required?

best, JAN

[kuro68k]

In reply to your questions:

1, 2, 4, 5: These issues can be solved by creating a really table based editor for libraries. I'll look into doing it if I can find the time. Basically if we can bulk edit parts in a table format, perhaps in a CSV file via a spreadsheet app, all these problems become easy to fix. Just agree on some trivial standards for things like manufacturer names etc. and batch changes become easy and quick.

Thinking more generally, such a tool would make library maintenance a lot easier. There are still many inconsistencies with libraries, even down to things like the placement and size of text. It seems like creating a tool to tidy all that up might be quite useful.

3: The current ALIAS system is not ideal, I agree... I don't know if it would be easier to fix it or just to ignore it. This certainly needs some thought.

Parts libraries are one of the hardest things for EDA packages to get right, but also provide huge gains if they can be done well.

[jkriege2]

So we are back in KiCAD-country ... as I said above: It would make much more sense to first create the required tools/features in the software and then see how we go on here ...

BTW: Having such a table-based tool would be really helpful ... I suggested some such changes here (https://forum.kicad.info/t/improvement-to-kicad-schematic-library-editor/) ... but didn't come anywhere ...

Best, JAN

[kevinb456]

I'm also very much in favour of this! Coming from other more mature tools this is a striking omission. However, KiCAD needs to support two main userbases;

1. Hobbyist / casual user - only really cares that the tool has the correct symbol to draw the schematic, correct footprint for the PCB, and an identifying name/value. The rest of the nuances e.g. Part number / Supplier etc will probably be in an external excel file, on a scrap paper in their project box, or just in their head! They will only look at occasionally and work out what each part should be when they order.
2. Larger OSHW projects / companies - want a system more akin to industry standard practices - the schematic contains the full design information. So a full and accurate BOM can be exported including part numbers, tolerances, voltages etc.

The tool already does everything for 1), but not for 2). There needs to be a way of allowing 2) but without removing the ability to work 'quick n dirty' for 1).

To me it seems an agreed convention for part fields is the way to go. Blank by default (or '_' if needed), so they would not interrupt any stage of the workflow for 1) and do not require extra effort to maintain the official libraries (other than running a script over them once to add the empty fields).

People would see these fields when they edit a part but if they dont want to use them that would be the sum total of their impact, it wouldn't change their workflow. They may even decide that writing the info in the part fields is better than in their external excel sheel / head / scrap of paper!

Personally, I would like to see added by default; Description ManufacturerName ManufacturerPart SupplierName SupplierPart Fit

ManufacturerName and ManufacturerPart (same for SupplierPart) could be combined into one field with an agreed separator e.g. ManufacturerPart as"Linear:LTC4359" (Idea by https://github.com/kasbah from the 1-click-BOM tool and https://kitnic.it/). If no separator is used it would default to understanding as just a part number. There could then be any number of extra ManufacturerPart's added by the user (the problem with separate fields is you have to start naming them ManufacturerName1, 2, 3... and ManufacturerPart1 , 2 , 3... etc to make sure one lines up with the other. Keeping as a single field with agreed delimiter would solve this.

Fit is true or false and specifies if the component is to be fitted. Description - To balance features with simplicity, I would not create separate fields for "Voltage" "Tolerance" "Dielectric" etc as this is probably overkill. The user can put these in the Description field.

This is akin to the system I implemented at my companies library. A part can be dropped on the schematic and left with whatever basic fields are in the library, i.e. simply Value "1k" and footprint "1206" if the user doesn't care. Or if the user does care they can go nuts and set the description as "1% 0.5W Pulse-proof Thick-Film Chip Resistor" and the Manufacturer/Part/supplier fields for an exact part (if crucial to the design).

The point with the system is it doesn't have any extra overhead for people that don't want to use it, but a big benefit for users/tools by having one set of practices and conventions to work to for those that do want to use it.

[kasbah]

I made a request for comment on the KiCad dev mailing list to and a standard MPN field: https://lists.launchpad.net/kicad-developers/msg27021.html

[SchrodingersGat]

Thanks @kasbah - it will be interesting to see what the other devs think. The new format offers a lot of improvements, hopefully it is implemented soon.

[kuro68k]

I think this really needs to be broken down into two separate issues.

1. Standardized names for things like MPN and other metadata, for BOM generators. The goal should be to have exactly one place where an attribute needs to be changed, which will be the schematic, rather than having to edit multiple files just to change a part. That will ensure consistency (BOM always matches schematic) and efficiency.

2. Consider adding some of those fields to the standard part libraries. This is much lower priority.

[kasbah]

After not much success in convincing people it's a good idea to set some defaults in KiCAD source code I have prepared a draft of a community standard on the forums.

[bobc]

No one seems to have mentioned i18n. I am sure many users will want to use their own language names. rather than english names or abbreviations.

[kasbah]

It's a fair point @bobc, and I will need to consider it when I actually get around to writing up the document for the community standard.

My initial thoughts are that internationalization is an interface issue that would be aided by using standard names. I.e. the KiCAD user interface can better translate the field names if it knows what names to expect.

[poeschlr]

If this discussion is still relevant please reopen it over at the new repo https://github.com/KiCad/kicad-symbols. This repo is now declared legacy and will no longer receive updates. (v5 will use the new repo, there is no new v4 release planned)