Package file name requirement

[endofexclusive]

The information that a file contains a package description is currently required to be encoded in three places to be recognized when updating the pool:

1. The file is located somewhere at /packages relative to the pool.
2. The file contains the JSON "type": "package".
3. The file name is package.json.

Number 3 can be a useful convention. However, it is not fully clear why it is a program requirement. Relaxing it would give the user more freedom to manage the file hierarchy to his taste, without limiting the program functionality or complexity (I assume).

[carrotIndustries]

Packages are directories with a package.json and a padstacks folder in it since in the early days of horizon there weren't global padstacks so each package had to bring it's own padstacks. Technically there's no reason why packages without custom padstacks couldn't just be a file.

OTOH, the pool manger always creates packages as a directory so there's the possibility to add custom padstacks when necessary. Asking the user whether a new package might include custom padstacks in the future so that the new package could be created as a file is something I'd rather not do since this will very likely confuse them ("The computer is asking me things I don't understand").

[endofexclusive]

Thank you for the clarification.

I ran into this when working on a Horizon output module of a footprint generator library in the making (to be available "soon"). After finding out the file name key, the files loaded well, and greeted with numerous JSON parsing error messages :-). The parser messages were very handy when tracking down my syntax issues.

On the file naming topic, I believe that programs should be very careful about setting rules on how the user shall organize data files. Instead, the user shall be able to place files in a way suitable for his project, workflow, taste and organization policies.

For example, one user may want to keep everything belonging to a MC68000 under a certain directory. Another user may want to temporarily soft-link (ln -s) a certain self-contained sub-directory of .json files into his current project pool to inspect and possibly borrow something. Many users may also see the current layout as the best option.

Here are two different suggestions which would make the naming more flexible.

1. Remove the directory name requirement. When updating the pool, the program determines the file type by inspecting the "type" field of the .json file.

or

2. Remove the directory name requirement. When updating the pool, the program determines the file type by inspecting the file name suffix. It could be .packson for packages, .partson for parts, or similar.

For both 1. and 2. above, custom pad-stacks for a package could go in a directory named as the package file with suffix removed.

Either of 1. or 2. above would leave the user free to implement a file hierarchy of choice, including arbitrary grouping of e.g units with symbols, etc, or self-contained directory for those who prefer that style.

The Horizon library design and implementation is excellent. I believe it would benefit further by relaxing the directory naming rules.

[atoav]

I often forget that packages need their own directory and are named package.json, so unless there is a really good reason to do things differently for packages I'd second @endofexclusive proposal to relax things there. Better now than never.

I am mostly fine with the directory name requirement, although I see the benifits of @endofexclusive second proposal (special file-name-sufficies like .horizonpackage, .horizonpart or any other idea).

The cool thing that this would allow you to do is to keep a part and all related things in the same directory. Also you would instantly know what a file is, without looking at the path of the current directory. Under the hood it could still be the same json.

[carrotIndustries]

Technically, modifying the pool update process to also consider other files isn't a big deal.

For example, one user may want to keep everything belonging to a MC68000 under a certain directory. Another user may want to temporarily soft-link (ln -s) a certain self-contained sub-directory of .json files into his current project pool to inspect and possibly borrow something

How do you envision this to work well with the global pool?

The cool thing that this would allow you to do is to keep a part and all related things in the same directory. Also you would instantly know what a file is, without looking at the path of the current directory.

As above, this is more an organisational thing on how the (global) pool should be organized than a technical thing.

[atoav]

The issue

I had some thought about this recently. When I create parts, I usually tend to go the extra mile and just add the whole series. I might not need them, but I hate the feeling of adding one part of some series only to discover a week later that I need the same part with a differen voltage rating or value or whatnot.

That means in the worst (or best, depends who is asking) case I end up adding 700 capacitors in one go by utilizing scraping and automatic package, model and parts generation. In the long run this might turn out to be unfeasable for the global pool, because updating the pool won't magically get faster and adding parts will slow things down.

The question is: Should everything be included in the global pool or is it better to leave the users with a more granular choice to make?

How others tackle this

You might be aware of librepcb. I tried it alongside horizon and ended up liking horizon more, because many concepts you implemented are so much more well thought out in terms of actual usability. But one thing I kinda liked about librepcb was their library manager:

That thing pops up on first startup and makes it quite easy to get going. A library to them is simply a repository (similar to what we have), but the way it is presented makes it look easier to understand to non-git-people.

They seem to split their libraries in multiple ways, some are "recommended" and some seem to be third party. The feeling I get as a user is that it is very uncomplicated to include new parts and libraries, should anybody end up making them.

A possible solution

We also can have this at the moment. E.g. if you wanna have my wima capacitors just git clone the right branch of my pool and include it into your default pool, click Update Pool and get going.

This is entirely possible but there are multiple issues with this.

1. You need to know this is actually possible and you need to know how to do it. It is not front and center, but some Tab in "the depths" of the pool editor. Same goes for upgrading the pool. This should be available on the start page. A regular user wouldn't go and edit the Default pool unless they already know what they are looking for. Even I tend to forget this upgrade button exists, and I am managing my own pool with git from the CLI
2. As useful the feature of including pools in other pools is, maybe the right way to solve the problem of multiple sources is to allow multiple pools to coexist side by side, each with their own repository and namespace (to avoid UUID-collisions and questions of who is allowed to override whom). I could imagine this to replace the Recent pools section in the startup window. Maybe elements that have the same UUID and contents in all pools could still be displayed as one to avoid unecessary duplicates. This is also visible in #165
3. We need to be aware that the way this is implemented also shapes the way people will participate. This has to do with visibility (if you see others are participating, you are more willing to do so yourself) and with perception (if you feel it is not too complicated to participate, you might do so earlier)
4. What if you want to try something which user XYZ included in a PR to the official pool? Is there a way for a non-git-person to try that PR in a way that is easily reversible? Maybe this could even be good for verification or finding issues? The more people look at these parts, the better.

I got the feeling that a clear concept would greatly improve this, but as a designer I am quite sure, that this is one of those issues where a face to face talk between multiple contributers would be more effective than weeks of writing.

[carrotIndustries]

As useful the feature of including pools in other pools is, maybe the right way to solve the problem of multiple sources is to allow multiple pools to coexist side by side, each with their own repository and namespace

Unfortunately, having pools with no linkage isn't preferable either since that'd make it impossible to easily exchange parts across pools. If resistor parts in two pools both reference the same resistor entity in the base pool, these can easily be exchanged for each other in the schematic editor. If each pool would have its own resistor entity that wouldn't work and the user would be confused since there are two resistor entities.

Some thoughts: With the current implementation, a librepcb-style pool would be entirely possible, but moving items to different pools creates the issue of fragmentation and where to put things and managing multiple repos can get troublesome as well.

UUID collisions aren't something to really worry about, colliding UUIDs are much less likely than hitting a bug in horizon.

Is there a way for a non-git-person to try that PR in a way that is easily reversible?

I've got some WIP implementation of that gathering dust in a branch of mine. In my current design, it's only useful for trying out new items since won't overwrite existing items.