A pool side scenario

[atoav]

Disclaimer: this is not exactly an Issue in the typical form, but more like a thought. I decided against writing a blog post and to post it here, where it reaches the right people. It is likely that somebody (@carrotindustries) already grappled with similar thoughts and ideas. This is going to touch multiple conceptual topics and is more meant as a base of discussion than a real proposal for anything.

While working on some projects using horizon I had a question: how would I go about sharing my schematic/board in a way that other people can open it without hassle. As for now I found two ways:

1. Use only parts that are in the official pool
2. Share my own pool right besides the project

A pool side scenario

Say you need a 32 Ohm resistor quickly and there is none to be found in the public pool – so you build it yourself. Quick and dirty, you don't want to waste an evening to pedantic accuracy. It is probably incompatible to all the other parts in the pool and there is no real MPN because it was a part you had lying around and finding the exact manufacturer number is work. If only you invested a little extra time, it could be pushed into the pool and if you didn't forget a thing the part might become part of the pool after a while. But you don't invest that extra time because you are a typical lazy user and you are interested to get moving with that project of yours and not deal with other things. Building that resistor already slowed you down enough, you love open source, but right now you have better things to do.

So you keep that quick'n'dirty 32 Ohm resistor – it is fair enough – as long as you keep the project to yourself it is. But now you have to ship that pool of yours with your project, so people can open it without getting strange messages about UUIDs which make little to no sense to non-programmers. Because your local pool is a mess nobody should see you decide to push it to the repository later. And because it is located in a completely different place than your project you obviously just forget about it till somebody complains a month later.

Too bad the person who complained did so only after your hard drive crashed and the pool is lost (see Murphy's Law). The only way to get it back now is to create new parts and to manually change their UUIDs to match the lost parts and finally upload it again. So you fire up Sublime Text ...

This story might sound a tiny bit dystopian, but this will definitly happen once people try to share the stuff they made.

A possible solution

All of this wouldn't happen if our user would limit him/herself to using the public pool. I think the pool is a great concept for sharing parts with each other and within a community, but maybe it lacks a sort of project-centric storage that allows for quick modifications/adaptions/creation of parts that are not meant to clutter the pool. A bit like git stash: temporary in nature. This would make sharing much easier as files that are not part of the public pool would be able to travel with the project files. Every user could still go back to the public pool version of a part, if they like to. And the other way round: if they find a cool part in a project they downloaded they can easily integrate it into to their local pool, clean it up and make a pull request for the public pool. So essentially this solution is to automatically bundle the used units/symbols/entities/3d-models/parts/packages/padstacks with each project, which would ensure that without user action every project looks just the same it looked like when somebody created it, similar to library version lock files e.g. in a language like Rust:

The purpose of a Cargo.lock is to describe the state of the world at the time of a successful build. It is then used to provide deterministic builds across whatever machine is building the project by ensuring that the exact same dependencies are being compiled.

This state of the world is saved locally with the pool. So you update only when you have the will and time to deal with possible changes, even if they should just be cosmetic in nature. If I am right, the idea beehind the existing pool cache is something similar, only without storing a project pool itself. If there are better and newer versions of parts in the library, the lock can be lifted. And if you don't care and always want to rely on the public pool, just leave the lock away. So basically this would be a the pool cache we have, but working also without local pool.

With this train of thought we would have multiple levels of pools:

public pool and forks  ->  local pools  ->  project pools

In this concept the project pool is only there to act as bridge between different people's local pools to aid the exchange of projects and "not for production" pool objects that are not ready for the public pool yet.

New Users

The steps a new user has to take when they want to open a horizon project they found online. They At the moment they would have to:

1. Download/Clone the Project
2. Download/Clone the fitting Pool
3. Download and run Horizon
4. Select that pool Pool in the project manager
5. Open the project

The problem here is – these not few steps for somebody who has no clue how the software works and who bascially just liked to check it out. Also if they install horizon and then right-click on my .hprj and open it with horizon-prj-mgr.exe things will fall appart as they get yelled at for not having a pool installed.

Much better would be:

1. Download/Clone the Project
2. Download and run Horizon
3. Open the Project

What do you think? There are many variation.Maybe I am overcomplicating things here or my assumptions are wrong altogether and this is not a problem at all. Maybe there are already plans on how to solve this issue. I am just trying to brainstorm a little and find out where the journey is going..

Misc pool related stuff

Maybe the pool should be downloaded automatically on the first start of the software (user just needs to select the location). I imagine a window that explains what the pool is:

horizon has a community maintained public pool on github. The pool contains all parts, symbols, packages and so on. It is a good start to create your own local pool as a clone from the public pool. Please select a location for it

Also nice would be a way to quickly try out pull requests listed in the Remote Tab of the Pool Manager. This would make sense to quickly check out new pull requests for maintainers, but also would make sense for people who contributed as they can already use their newly created objects in production without manually copying them.

[carrotIndustries]

While working on some projects using horizon I had a question: how would I go about sharing my schematic/board in a way that other people can open it without hassle.

A project already caches all pool items it uses in the cache folder. When sharing a project, just include this folder and everything should be fine. That also locks a pool item's version.

I've had some thoughts about pools being able to inherit from each other, i.e. pool A inheriting from pool B sees and is able to use pool A's items as well, but only in a read-only fashion. Items in pool A take precedence, so that if the user want's to make a modification they can just copy the item over to pool B and it'll shadow the item in pool A.

Maybe the pool should be downloaded automatically on the first start of the software (user just needs to select the location). I imagine a window that explains what the pool is Definitely something to consider for the first relaease.

Also nice would be a way to quickly try out pull requests listed in the Remote Tab of the Pool Manager.

The git interaction is already more complex than I envisioned it to be, let's see...

[atoav]

Ahh so the function of the project cache is exactly what I had in mind there. Have to test that one again, maybe I somehow forgot the cache folder.

Pool inheritance is a great idea, because typically you want to keep up to date with the official public pool, while still beein able to do your own tomfoolery in your own pool.

This pool is quite a technical concept (and not a bad one!). That's why focusing on the human side of things is important here. People will find ways to deal with things they want to do and good software should make it easy to keep best practice in a environment where things often happen in chaotic and unplaned ways. Often this is also about the feeling a software environment gives you. Git for example is great because it helps you stay organized. You can fool around however you like, but only when you're done you have to commit to it.

Yesterday I used the horizon board editor for example to aid the creation of a stripeboard-layout for a audio mixer we will build in a university course – and I missed vertically placed th resistors (useful for stripeboard stuff). So I quickly made a new package and a part etc. But because it is something quick and dirty, that was just made to fix a problem on the spot I would like to somehow mark it as "dirty" or "work in progress" or something like that. Using a tag for this, is what I do right now, but ideally a part should be marked as dirty/unfinished/work in progress as long as a user doesn't explicitly say "this is done now".

The whole git part has the potential for doom or eternal glory alike. The library implementations we grew to hate in Eagle or KiCAD also have their usable and unusable sides and they probably all started with the pretence to make it better than EDA applications before. In order to avoid falling into the same traps of organically growing the pool into a system nobody can or wants to manage I think it is necessary to take a step back and reassess which parts of the concept "pool" work well and which do, can or will cause trouble. At least before the first release.

For now I decided to ignore the built in remote function (except for upgrading my pool) and to use command line git to manage my branches, pull requests etc. And at some points I manually change json files. This has multiple reasons, which I could explain if there is interest, but for now the important take away is that I feel it is either easier, faster or helps me stay organized better.

Before the first release there should be a decision what the goal is for the whole pool and the pool managment alike. This also means it should be very clear what are non-goals. E.g. a goal could be to allow to do all necessary things via the pool manager (things like changing a symbols unit or a parts partent should be possible without manually editing the file etc.). This also means the whole remote section should have a well defined scope – currently it is upgrading, commiting pull requests and seeing a list of said pull requests. Once users want to try out one of these listed pull requests, help with them or add to these, they have to do manual git magic.

Note: I don't say this has to be part of horizon, it just should be clear how you can take part if you want to. If we decide that checking out pull requests is not part of the scope, I'd gladly write a guide for new users.

Is there anything where the goals for the first release are listed/discussed?

[carrotIndustries]

7f157d8 should address this issue in some ways

[atoav]

@carrotIndustries I am not totally clear what is actually possible with the pools right now and what isn't:

• I can Add pool in Preferences > Pools and radio select one of them
• In the Settings Tab of the Pool manager I see two panes called Pools available and Pools included, but I am unsure how to add pools here, how I am meant to use this feature and what it means for the structure I should aim for
• There is a per-project-pool-cache which is actually kinda amazing and solves a lot of trouble

Are these two panes in the Settings Tab of the Pool manager implemented yet? If yes, would you mind giving some hints what I can do with it? Maybe writing it in the wiki would make sense too.

If it is not implemented yet: I am just a bit curious what's gonna come : )

[endofexclusive]

I use the Pools included as a convenient way to browse other pools and to copy parts into project specific ones which are version controlled with the project. This may or may not be the intended way to use the functionality. However, the possibility for different work flows is a positive thing. (After adding a pool to Pools included, the Save button and Update pool must be clicked).

[atoav]

Okay interesting, how do I add a pool to Pools included tho?

In my case it looks like this:

[carrotIndustries]

Let me try to explain how horzion EDA currently handles pools:

There's a global list of pools available that can be manipulated using the Pools page in the Preferences dialog. These pools are identified by their UUID. Since it might occur that the user has two copies of the same pool for whatever reason, pools can be enabled and disabled such that for each given UUID there's no more than one enabled pool.

When opening a project, the global pool list is queried for the pool UUID that's stored in the project to find out it's location on the disk.

Now on to the settings tab.

The metadata for the items in the pool is maintained in a SQLite database. Normally, updating the pool just collects all json files in the pool and populates the database accordingly. Including a pool just makes the pool update also search for files in the included pools so that they show up in the current pool. For an item that's present in both pools, the item in the included pool will take precedence and a red box (Overridden) will appear in the item list. For the pools to show up in the available pools list they'll need to be added in the pools dialog and enabled.

This enables several usage scenarios:

• Per-project pools: Create a pool for each project and include the default pool. If your project happens to require some oddball part that's not suitable for the global pool, create it in that pool. For items from the global pool that need to be modified only for that project should be copied into the project pool. These will then show up as overriding.

• Librepcb-style libraries: Instead of one global pool, have several bundles of related items that you can include in your project-specific pool.

• Maintaining a set of custom items: If your workflow requires some special items, the easiest way to handle these is to create a pool for these and include the global pool in that one.

Since project-specific pools is something that might turn out to be pretty useful, there might come some support to facilitate creating a pool as a project is created.

[endofexclusive]

@atoav This is the method I have used

• On Horizon EDA main screen, Create a pool with New... > Pool. Give a name and base path, then create.

• The newly created pool opens in the Pool manager.

• Other pools known to Horizon should show up in the Settings.

• Select pools to include, press Save and update.

• Color coded item, which I believe denotes it is from an included pool

@carrotIndustries Thank you for the explanation!

[atoav]

My workflow

Ok now it seems to make more sense. I have my own fork of horizon-pool. For each new part (or a series of parts that seem to belong together) I create a branch. These branches can be used to create PRs on the parent repository.

In order to use them for myself I have a branch called "work" where I merge all part branches. This is essentially a global pool approach, that helps me to achieve two goals:

• I want to share these parts (forces you to be more accurate)
• I want to use these parts without having to wait for my parts to be merged into master

An easier solution

I could take all the files that differ between work and master by creating a new custom pool and running cp -pv --parents `git diff --name-only work master` CUSTOM-POOL-DIRECTORY. This custom pool with just my parts can then be included in the official main pool.

If I want to share a part, I just click merge. The only downside I see there, is that the branch names are not as beautiful as if I did it myself. But for a beginner and somebody who has no idea about git this could work quite well.

[atoav]

One issue I found is following:

If I want to make my own pool of parts, but it should depend on some common things from the official pool (e.g. Padstacks) then I have three options: A) Don't copy it, because the pool I mean to include it in has it anyways (this means the editor will throw an error etc) B) Copy it, so my sub-pool is an independent thing that could potentially run on its own. This has one problem: I encountered crashes whenever I override parts/units/etc already present in the parent pool C) Don't copy it, but include the offical pool (which makes authoring the pool harder because all the other parts show up)

Nothing is really optimal here. A big issue seems to be keeping parts and their dependencies coupled strongly enough so they don't fall appart too easily and weakly enough so you get that cool flexibility.

Edit: I tried to reproduce the crash for option B once again, but couldn't. So maybe it really depends on what is updated and what isn't. Additionally it seems that 3D models of included pools don't show up for some reason?

[carrotIndustries]

C) Don't copy it, but include the offical pool (which makes authoring the pool harder because all the other parts show up)

Since that's the preferred method, any idea how to improve on that situation? Maybe an option to only show items from the current pool (hide included items).

[atoav]

Since that's the preferred method, any idea how to improve on that situation? Maybe an option to only show items from the current pool (hide included items).

Any additional filters for the search are always welcome.

But a even better way might be if we – similar to the include pool feature – could include pools as a "invisible parent": items in this pool won't show up, unless any of the items in the child pool inherit from it. Maybe that could also be a option for the included pools (e.g. like an option called "only include dependencies").

That way a child pool could inherit in every way they like without dragging in parts that are not needed.

[atoav]

I think most of the topics I wanted to adress in the original issue have been adressed since, so I will close this