Introduce component templates

[dglazkov]

To unleash the full potential of composability in Breadboard, let's introduce a new concept: component templates.

Invoking boards using Core Kit invoke component is how Breadboard enables composition of reusable behaviors. One way to view component templates is as a form of syntactic sugar on top of invoking boards to make invoking feel like just using components. This is something we've been doing with composite Kits (like Agent Kit, Gemini Kit, etc.), and component templates make this mechanism more accessible and general.

A component template is an instance of an existing component that has some (or all) of the configuration filled in. Every component dragged onto the Visual Editor surface can become a component template.

Component templates are stored in a separate templates array inGraphDescriptor as peers to nodes and edges. Every template must have an id that is unique within this GraphDescriptor.

{
  "nodes": [ "..." ],
  "edges": [ "..." ],
  "templates": [
    {
      "id": "local:sonnet",
      "type": "invoke",
      "metadata": {
        "title": "Claude 3.5 Sonnet",
        "description": "Calls Claude 3.5 Sonnet API"
      },
      "configuration": {
        "$board": "https://myboardserver.com/boards/@dimitri/call-sonnet-api.bgl.json"
      }
    }
  ]
}

This template id is then added to the pool of available component types. In the example above, in addition to all the component types provided by various kits, the local:sonnet type becomes one of the valid component types.

In Visual Editor, this would mean that "Claude 3.5 Sonnet" shows up under a new "Templates" section of the component picker.

Templates introduce a level of abstraction: instead of using an invoke component from the Core Kit, I now work with a "Claude 3.5 Sonnet" component, and I don't have to think about what's underneath.

When a new component of this type is instantiated, it is an instance of this template:

• the template's underlying component type is used for invoke and describe
• the instance configuration is pre-populated with the template configuration. In Visual Editor, we could hide that information behind a twistie or a menu option, so that it's out of the way for the most common use cases.

The introduction of component templates changes how we view kits, particularly composite kits (kits made of boards). Looking from the component template perspective, a type in the composite kit is just a template: an underlying invoke component type (from Core Kit) with some pre-configured URLs. I can build one for myself for my board. I can also publish my board and let others copy/paste that template into their boards if they wish to use it.

If I want to publish a composite kit, I just publish a board with a bunch of templates. We may want to introduce some machinery to load boards "as kits" so that I could easily access all the templates I find interesting.

Templates can refer to other templates as their underlying type.

In the example below, the "Writer" component template uses "Claude 3.5 Sonnet" template and populates the "persona" port.

{
  "nodes": ["..."],
  "edges": ["..."],
  "templates": [
    {
      "id": "local:writer",
      "type": "local:sonnet",
      "metadata": {
        "title": "Writer",
        "description": "A Writer"
      },
      "configuration": {
        "persona": {
          "role": "user",
          "parts": [
            {
              "text": "You are a writer who knows how to write. You are super-good."
            }
          ]
        }
      }
    },
    {
      "id": "local:sonnet",
      "type": "invoke",
      "metadata": {
        "title": "Claude 3.5 Sonnet",
        "description": "Calls Claude 3.5 Sonnet"
      },
      "configuration": {
        "$board": "https://myboardserver.com/boards/@dimitri/call-anthropic-api.bgl.json"
      }
    }
  ]
}

The local:writer depends on local:sonnet, which depends on invoke.

This unlocks another useful scenario. For instance, if I use a Specialist a lot, but most of the time, I have a fairly narrow set of uses for it (a writer, a critic, an editor, a comms strategist), I can start collecting my own templates of Specialists and use them as I need them -- eventually perhaps working up to a full-blown composite kit.

Template may not be referred to from outside the board. The ids are local to the board. If an author of a board wants to refer to a template, they must first copy/paste it (and any dependent templates) into the GraphDescriptor.templates.

Interesting question: Are template instances dynamic or static?

• when instances are dynamic, the instance retains the reference to the template id and so when the template changes, these changes are picked up by the instances. Here, the "template" becomes a bit of a misnomer, since there's more of a prototype/inheritance relationship between the instance and the template.
• when instances are static, the instance take on the underlying type, and the template's configuration becomes its own. Any changes to the template are not propagated to instances.

My inclination is to go with static (actual templates) first and see how far we get. In this case, there will still be remnants of the template-ness in the instances. In the example above with local:sonnet, we will need to inform the Visual Editor to hide the $board port somehow. Perhaps an advanced behavior that only exists on this instance? This behavior is applied during the stamping of the template.

[aomarks]

Could you expand a bit on the problem that this new concept solves? Maybe a motivating example (the given example is good, but it's not really obvious to me why you wouldn't just have a normal invoke node).

[dglazkov]

Thank you for reading and helping me figure it out. I added these three sections in the top post.

To answer your specific question:

Templates introduce a level of abstraction: instead of using an invoke component from the Core Kit, I now work with a "Claude 3.5 Sonnet" component, and I don't have to think about it what's underneath.

Component templates and kits:

The introduction of component templates changes how we view kits, particularly composite kits (kits made of boards). Looking from the component template perspective, a type in the composite kit is just a template: an underlying invoke component type (from Core Kit) with some pre-configured URLs. I can build one for myself for my board. I can also publish my board and let others copy/paste that template into their boards if they wish to use it.

Emergent kits:

This unlocks another useful scenario. For instance, if I use a Specialist a lot, but most of the time, I have a fairly narrow set of uses for it (a writer, a critic, an editor, a comms strategist), I can start collecting my own templates of Specialists and use them as I need them -- eventually perhaps working up to a full-blown composite kit.

[dglazkov]

And also:

If I want to publish a composite kit, I just publish a board with a bunch of templates. We may want to introduce some machinery to load boards "as kits" so that I could easily access all the templates I find interesting.

[dglazkov]

Maybe it's not a separate array? Maybe it's just a "template": true in component's metadata?

[dglazkov]

?

To answer my own question: I can't refer to the template this way, which means that I can't build a "Writer" that is a "Sonnet" that is an "invoke".

Need a way to bridge component instance -> component type.

[aomarks]

Templates introduce a level of abstraction: instead of using an invoke component from the Core Kit, I now work with a "Claude 3.5 Sonnet" component, and I don't have to think about it what's underneath.

Interesting! Playing devil's advocate: Why wouldn't you create a "Claude 3.5 Sonnet" board and then use that like a component? Aren't boards already a good way to encapsulate some re-usable behavior?

[dglazkov]

Templates introduce a level of abstraction: instead of using an invoke component from the Core Kit, I now work with a "Claude 3.5 Sonnet" component, and I don't have to think about it what's underneath.

Interesting! Playing devil's advocate: Why wouldn't you create a "Claude 3.5 Sonnet" board and then use that like a component? Aren't boards already a good way to encapsulate some re-usable behavior?

Yep, that's what the example does. What it encapsulates is invoking that board.

[dglazkov]

One way to think of it is that boards are our way to encapsulate reusable behaviors.

Template components are our way to encapsulate invoking these reusable behaviors and make these invocations feel like a component.

[dglazkov]

Added this at the top:

Invoking boards using Core Kit invoke component is how Breadboard enables composition of reusable behaviors. One way to view component templates is as a form of syntactic sugar on top of invoking boards to make composition feel like just using components. This is something we've been doing with composite Kits (like Agent Kit, Gemini Kit, etc.), and component templates make this mechanism more accessible and general.

[dglazkov]

Something doesn't add up: if the instantiation is static, then I will never see type = local:sonnet, I will only see invoke. So are these types just labels for inside the templates?

[aomarks]

Walking through the example from the user's perspective would help me I think:

---

Let's say I'm trying to write a board that writes two poems (independently). Here's how I would do it with our current feature set with the visual editor:

1. I drag a Claude Sonnet onto the canvas. (Sonnet is a board, so under the hood it's actually an invoke node, but I just see it as a component visually.)
2. I configure the node and set "You are a super-good poet" as the first part of the prompt.
3. I also set "Write a poem about cats" as the second part of the prompt.

So far so good. Now I decide I want to write another poem. So, I repeat steps 1-3, creating another Sonnet node, but changing step 3 from "Write a poem about cats" to "Write a poem about coffee".

That works, but it's a bit laborious. I might want to to create a ton more poems and not have to repeat all the steps. Or, I might want to change the board URL we're using, or the common part of the prompt, without having to update all the instances individually.

So, I need some kind of abstraction. What do I do?

My first thought is that I should create a "poet" board. This poet board would have a Sonnet node, an input node with one port called "poem topic", and some kind of template or code node that combines the common part of the prompt with the variable part of the prompt.

(We could also discuss how we could make this kind of "refactor-into-board" action really nice from the Visual Editor, since it's not super easy right now, but that's probably a separate discussion).

---

So, given this example, why would I as a user reach for a template instead of making a new board?

[dglazkov]

Thank you! In this:

I drag a Claude Sonnet onto the canvas. (Sonnet is a board, so under the hood it's actually an invoke node, but I just see it as a component visually.)

When I examine the component, does it look like an Invoke component? Or does it look like a Claude Sonnet component? What's the UI like?

Put differently, where does it get its identity from? Is it an invoke component that is configured with a particular URL or is it a Claude Sonnet component?

[dglazkov]

One thing we could do is make invoke a shape-shifter: once it has a valid board to describe, it changes itself to pretend to be that board, from metadata (title, description, help) to inputs/outputs (this already happens), and hides the board param behind the "Advanced" twistie.

[dglazkov]

@aomarks your example got my thoughts going. It does look like the "component templates" is primarily about identify of a component, rather than actual templating. What I was trying to do is come up with a way to present an "invoke" component in VE as a "Claude Sonnet" component, where the actual invoke machinery is hidden away so that it doesn't get in the way -- but is still available if needed.

[dglazkov]

The question I want to answer is: how do I emulate the currently-entirely-magical behavior of "Specialist" or "Export File" where an invoke turns into a nicely encapsulated component? And maybe this is as simple as adding a few new "behavior" tags, like "advanced" and adding the shape-shifting capability to the invoke describer?

[dglazkov]

As an experiment, I am going to try the "shape-shifting" idea today.

[aomarks]

The question I want to answer is: how do I emulate the currently-entirely-magical behavior of "Specialist" or "Export File" where an invoke turns into a nicely encapsulated component? And maybe this is as simple as adding a few new "behavior" tags, like "advanced" and adding the shape-shifting capability to the invoke describer?

Ah, very interesting! I think I see the gap this proposal is trying to address much better now.

It seems like from the user's perspective, you really never want to see an invoke node. It's always the board you are referencing that you actually care about; invoke is just an implementation detail.

This actually makes me wonder if we just should hoist invoke up as a core BGL feature and stop thinking of it as a node type:

{
  "nodes": [
    {"type": "#my-local-poet-board", /* ... */},
    {"type": "https://my-favorite-board-server/claude.bgl", /* ... */} 
  ]
}

You could combine this with something analogous to an import map in JS:

{
  "typedefs": {
    "poet": "#my-local-poet-board",
    "claude": "https://my-favorite-board-server/claude.bgl",
  },
  "nodes": [
    {"type": "poet", /* ... */},
    {"type": "claude", /* ... */} 
  ]
}

This might also solve the global type-name collision problem we have, since you'd always have a fully qualified handle to find for a node type.

[dglazkov]

{"type": "#my-local-poet-board", /* ... */},

🤯 that might very very very cool

[dglazkov]

Ah, we can't get rid of invoke node, since we want to retain the ability to choose which board to invoke at runtime. For example, that's what Specialist's tool calling relies on.

[dglazkov]

The analogy of import and import() in JS is striking.

[aomarks]

Ah, we can't get rid of invoke node, since we want to retain the ability to choose which board to invoke at runtime. For example, that's what Specialist's tool calling relies on.

Ah yeah, very good point!

This now makes me wonder if we need something like interfaces. It seems like a lot of the time when we have a dynamimc invoke node, it's because we want to be able to switch between multiple implementations of the same abstract functionality. E.g. an "LLM" interface which can be implemented by Gemini, Claude, GPT, etc.

{
  "nodes": [
    {
      "type": "invoke",
      "configuration": {
        "interface": "https://my-favorite-board-server/llm.bgl.interface",
        "implementations": [
          "https://my-favorite-board-server/claude.bgl",
          "https://my-favorite-board-server/gemini.bgl"
        ]
      }
    }
  ]
}

The interface would be just an IO JSON schema (like the result of describe, without a invoke), and the implementations would be expected to conform to the same schema as the interface (or a superset of it, using the logic from https://github.com/breadboard-ai/breadboard/blob/main/packages/schema/src/subschema.ts).

Now, when the visual editor encounters an invoke node that has an interface in its configuration, it can eagerly fetch the interface schema , and display the invoke node as though it were a regular node with that schema (with some kind of visual affordance to show that it's dynamic).

[dglazkov]

This now makes me wonder if we need something like interfaces

YESSS

[dglazkov]

Something like:

{
  "configuration": {
    "$interface": "https://my-favorite-board-server/llm.bse.json"
  }
}

Where llm.bse.json file must return a Breadboard Service Endpoint describe response.

And add a $ to interface to reduce input port collisions.

[dglazkov]

It sounds like there are couple of distinct capabilities emerging in this conversation:

• hoisting invoke URL up to NodeTypeIdentifier and teaching Breadboard about something similar to statically resolved imports in JS.
• the invoke interface capability, which allows specifying the interface of the expected board.

Sounds about right?

[aomarks]

It sounds like there are couple of distinct capabilities emerging in this conversation:

• hoisting invoke URL up to NodeTypeIdentifier and teaching Breadboard about something similar to statically resolved imports in JS.
• the invoke interface capability, which allows specifying the interface of the expected board.

Sounds about right?

Yep! I now think the hoisting idea is orthogonal and probably lower priority, since as you point out we're still going to need invoke nodes to support the dynamic use case.

A third related thing we could add is: When the visual editor encounters an invoke node that has a static $board in its configuration, fetch that board and display the node as though it had that signature. So the full logic for how the visual editor displays invoke nodes could be, in order of precedence:

1. If there is a static $board, fetch that board and display the invoke node as though it were that board as a component.

2. If there is a static $interface, fetch that interface and display the invoke node as though it were a component with that interface, along with some indication that it's a dynamic node.

3. Fall back to the behavior we have now, display an opaque invoke node.

(I'm not sure if this somehow just happens magically in the inspector API, or is a feature directly implemented by the visual editor, TBD I guess)

[dglazkov]

Yep! I now think the hoisting idea is orthogonal and probably lower priority, since as you point out we're still going to need invoke nodes to support the dynamic use case.

I got a bit obsessed with this idea and implemented it: https://youtu.be/NmBJDo_CnuI

[aomarks]

I got a bit obsessed with this idea and implemented it:

Cool! So does BGL itself now allow URLs in the type position, or was it just a visual editor feature?

[dglazkov]

Cool! So does BGL itself now allow URLs in the type position, or was it just a visual editor feature?

Yup, all the way down to BGL. I wonder what this would look like for Build API?

[aomarks]

Cool! So does BGL itself now allow URLs in the type position, or was it just a visual editor feature?

Yup, all the way down to BGL. I wonder what this would look like for Build API?

Yeah I was pondering that. I don't think it really affects the core syntax, because invoke nodes are handled behind-the-scenes. So whether we generate an invoke node or an inline board url doesn't directly affect how you write a board.

There is some related logic to improve in this space, though, which is that right now when we embed a board we just serialize the BGL with a #graph reference in the invoke node. We should probably instead embed a URL when we can, computed from metadata in the Kit. We need to be a bit careful with versions and such, since there's no guarantee that if you are importing an npm board at some particular version, that the URL in the metadata will point to the same exact code. More broadly, as we develop the ecosystem around board servers, it's a bit less obvious how (or whether at all?) we support npm interop.

[dglazkov]

Declaring victory 🚢