Lattice Structure for Composed Atoms

[nrdxp]

TL;DR

Propose a semantic structure in the manifest for composing atoms, both within mono-repos and remotely, while maintaining isolation and providing an intuitive user experience.

Background

While the low-level wiring for composing atoms is in place, we need to define a clear and intuitive semantic structure in the manifest. This structure should allow for seamless composition of atoms while preserving isolation between them.

Proposed Solution

Introduce the concept of a "lattice" - a higher-order structure that represents a collection of atoms. This lattice structure would provide a framework for composing atoms and managing their interactions.

Key features of the lattice structure:

1. Ability to compose atoms within a mono-repo
2. Support for remote atom composition
3. Preservation of isolation between atoms
4. Intuitive user experience for managing composed atoms

Implementation Approach

Leverage the existing test directory as a starting point for developing the lattice structure. This approach offers practical benefits:

1. We already need access to other atoms in the repo for testing purposes
2. It provides a real-world use case to guide the design

Benefits

1. Standardized approach to atom composition
2. Improved modularity and reusability of atoms
3. Easier management of complex projects involving multiple atoms
4. Maintained isolation, reducing unexpected interactions between atoms

Potential Challenges

1. Ensuring that the lattice structure doesn't introduce unnecessary complexity
2. Balancing flexibility with maintainability
3. Handling potential conflicts between composed atoms

Next Steps

1. Define the basic structure of a lattice in the manifest
2. Implement a prototype using the test directory as a framework
3. Develop guidelines for atom composition within a lattice
4. Create examples demonstrating both local and remote atom composition
5. Gather feedback on the proposed lattice structure

Questions for Consideration

1. How should dependencies between atoms in a lattice be managed?
2. What mechanisms should be in place to resolve conflicts between composed atoms?
3. How can we ensure that the lattice structure remains intuitive for users of varying experience levels?
4. What limitations, if any, should be placed on remote atom composition to maintain security and stability?

Request for Feedback

We welcome input on:

1. The concept of using a lattice structure for atom composition
2. Potential use cases we may have overlooked
3. Suggestions for improving the intuitiveness and flexibility of the lattice structure

[blaggacao]

When Atoms Interact: Orchestration vs. Choreography

When two atoms interact or "wechselwirken" (phys. German for "interact"), we encounter two primary paradigms:

Orchestration vs. Choreography

Do multiple atoms simply behave harmoniously as in a choreography, or are they (more passively) orchestrated by a single atom?

Orchestration: Which Atom Becomes the Conductor?

A typical pattern I've observed is the inference of "glue-atoms" in the presence of a set of atoms. For example:

• If a dev atom and package atoms are present, then a dev-pkg atom is always pulled in as well, and so forth.

These glue-atoms often take on the role of the conductor and exhibit the following characteristics:

1. They can be attached as optional dependencies to either atom, a decision that needs to be negotiated between two maintainer teams beforehand.
2. They encapsulate the entirety of the "foreign" API.

This approach allows for more flexible and modular system design, enabling better separation of concerns and easier maintenance of complex atom interactions.

[nrdxp]

Sounds interesting, but I'm not sure I fully follow, could you maybe give an concrete example or something.

[tgunnoe]

Could you give an example of the choreography-based atoms? from how you were explaining it, that seems the "more modular" design, but I believe you were referring to the orchestration by other atoms as the "more flexible and more modular" paradigm.

[blaggacao]

I'm having in mind the following scenario:

• Atom 1: pkg (or X)
• Atom 2: dev (or Y)
• Glue-Atom: suppose the teams negotiated that it would be dev.pkg and maintained by the dev team.

Now when Atom 1 & Atom 2 are present, the glue-atom could be brought in scope automatically.

Maybe one could say "feature-flagged" on dev by the presence of pkg, or in general on Y by the presence of X.

[nrdxp]

That's interesting, and should be independently possible to do with the feature flags, since they allow depending on optional dependencies, you should already be able to accomplish that in our model (just mark an atom as optional, and refer to it in a feature flag).

More generally though, I was thinking atoms can naturally compose within a mono-repo when "contained" within one another, but not downwards, to avoid implicit dependencies. Here's how it might look:

1. Directory structure example:
   .
   ├── atom
   │  ├── dev
   │  │  └── mod.nix
   │  ├── dev.toml
   │  ├── pkg
   │  │  ├── atoms
   │  │  │  ├── pkg-lib
   │  │  │  │  └── mod.nix
   │  │  │  ├── pkg-lib.toml
   │  │  │  ├── wrapper
   │  │  │  │  └── mod.nix
   │  │  │  └── wrapper.toml
   │  │  ├── mod.nix
   │  │  ├── other
   │  │  │  └── mod.nix
   │  │  └── some
   │  │     └── mod.nix
   │  └── pkg.toml
   └── atom.toml

2. Composition rules:

   • Parent atoms must explicitly declare child atoms as dependencies in their manifest.
     • if they want to re-export them in whole or part via it's public API, it must do so explicitly as well in the mod.nix
   • Child atoms can automatically see the public interface of their immediate parent.
   • Parents are not automatically aware of their children, allowing for selective vendoring.

3. Scope examples:

   # Top-level atom
   {
    # Explicitly declared atom dependencies
    dev;
    pkg;
   
    # Modules
    pkg.some;
    pkg.other;
   
    # Note: pkg's subatoms (pkg-lib, wrapper) are not visible here
    # i.e. they were not declared as explicit dependencies in atom.toml
   }
   
   # Child atoms (pkg-lib & wrapper)
   {
    # atom.pre refers to parent's (pkg) public interface
    foo = atom.pre.some;
   }
   
   # pkg atom
   {
    # Can refer to public children of its parent
    value = atom.pre.dev;
   }

4. Key points:

   • Child atoms use `atom.pre` to access their parent's public interface.
   • If a child doesn't need its parent's interface, it should be a sibling, not a child.
   • For private parent interface access, use a module instead of an atom.
   • Vendoring a child requires vendoring its parent, but not all siblings.

This structure provides clear guidelines for code organization based on scoping relationships and dependencies.

Some open questions:

• Do we need child atoms to have a recursive view of parents all the way to the root (i.e. atom.pre.pre.pre, etc), or just the immediate parent? The latter would probably be easier for vendoring, but perhaps the former is more semantically appropriate?

[blaggacao]

Could you give an example of the choreography-based atoms?

I don't think that's a desirable scenario. But it would look like a bit of scattered conditional logic all over the place that is then triggered by the presence of X or Y, on either side.

It's probably hard to maintain.

[nrdxp]

After some brainstorming I have this high-level draft:

[atom]
name = "my-awesome-atom"
version = "0.1.0"

[atom.source]
url = "github:this/repo:path/to/this/atom" # canonical url for the source repo, either a full url with schema, or a shorthand, e.g. `github:`

[dependencies]
other-atom = { source = "github:user/repo", tag = "v1.0.0" }
mono-repo-atom = { source = "github:org/mono-repo:path/to/atom" }
local-atom = { path = "../sibling-atom" }

[dev-dependencies]
test-atom = { source = "github:user/test-repo", branch = "main" }

lock format (in toml for conciseness, but would probably be json, or maybe toml is good for human readability?):

version = 1

[[atoms]]
name = "other-atom"
version = "1.0.0"
source = "github:user/repo"
rev = "abcdef123456789"
hash = "sha256-..."

[[atoms]]
name = "mono-repo-atom"
version = "0.2.0"
source = "github:org/mono-repo"
path = "path/to/atom"
rev = "fedcba987654321"
hash = "sha256-..."

[[atoms]]
name = "local-atom"
version = "0.1.0"
source = "github:this/repo"
path = "path/to/this/sibling-atom"
hash = "sha256-..."

[[atoms]]
name = "test-atom"
version = "0.3.0"
source = "github:user/test-repo"
rev = "123abc456def789"
hash = "sha256-..."

Which achieves these high-level goals:

1. Declarative: The manifest clearly specifies dependencies with their sources.
2. Vendorable: The lock file provides exact versions and hashes, allowing for vendoring.
3. From source: Dependencies are specified by their source repositories by default.
4. Efficient: For mono-repos, you can specify a path within the repo.
5. Not verbose: Local dependencies can be specified with a simple path.
6. Single source of truth: The lock file serves as the single source of truth for exact versions and hashes.
7. Self-contained: All dependencies are explicitly declared.

Perhaps we can also have something like:

[atom.registry]
url = "https://my-company-registry.com"

And atom will first search the registry for atom's matching the spec before pulling from source. Also there would be an eka vendor or similar command to bring all the dependencies into the current repo in a vendor directory.

[nrdxp]

this issue would be moving to the cli boundary via ekala-project/eka#4, which defines a powerful type system for atom's that is flexible enough to define arbitrary collection types. For clarity, we might call these special higher-level types something besides atoms to make clear that we have left the boundary of singular atomic units, and have moved on to collections of them. For that, maybe we keep the lattice venacular, or something similar