search

breadboard-ai / breadboard

A library for prototyping generative AI applications.
Apache License 2.0
209 stars 31 forks source link

Make `breadboard debug` support `--kits` option #661

Open dglazkov opened 9 months ago

dglazkov commented 9 months ago

Let's figure out how to enable adding kits at runtime for breadboard debug command. We can work around it for now with just adding kits to breadboard-web, but having this work becomes important when we have more light kits, since it enables us to supply still-WIP kits into the breadboard debug to actually work with them.

PaulKinlan commented 9 months ago

I did look into this a while ago, I wanted to see if I could intercept the Harness and dynamically inject the kits.... But I couldn't reliably work out what to replace because I would only have obfuscated code.

@paullewis any thoughts on how to provide hooks into breadboard-web that I could add runtime injected kits?

One thought is to have a standard url/import that I could intercept in the web-server to provide all the kits.

Another could be to have a query string parameter as a list of kits that a board needs...

paullewis commented 9 months ago

I thought the Breadboard Web harness automatically included all kits by default. Is that not carrying through? FWIW I think there's something we need to ponder there, because the kits aren't expressed in the JSON anywhere, so it's assumed any harness would know what - say - invoke means. It is possible someone could have different kits which offer nodes of the same name but with different behaviours.

PaulKinlan commented 9 months ago

Yes it's possible to have two kits with the same names. There's been a long-standing namespacing question iirc.

The Web UI via the harness includes the kits in this repo, but it also ties the debugger and kits to the same release. That's not to mention, I've a heap of kits that I just can't use in the debugger.

PaulKinlan commented 9 months ago

Just bumping this back up. I'd love to be able to get kits loaded into the breadboard-web.

Some thoughts about how to do this:

  1. Create a special file that the UI will fetch to get a list of kits it needs to import.

preview.ts and config.ts have a list of kits, I think I can have the CLI rollup any additional kits that are specified on the CLI and then make that file available to be served from the http-server the CLI has, and then add to that kits list.

I can do the work @paullewis for this, I just wanted to check if you have other ideas.

  1. ??? - I'm open to suggestions.
paullewis commented 9 months ago

That seems sensible tbh. I think the BGL specifies the kits, though, so I wonder if we can load them dynamically in some way? Not spent that much time with the kits side of things.

paullewis commented 9 months ago

Hmm, thinking about it some more, is the concern that we have to manually specify the kits on the CLI and then we might not have the right ones to power the board that's loaded?

PaulKinlan commented 9 months ago

The concern is that any kit that is made (light-weight, or heavy) needs to be hardcoded into the UI right now. At least as a first step - enabling you to add new kits (like my ClaudeKit) would be beneficial because we don't have to include every new little board. We can keep the current hard-coded list so it's not like you have to specify everything all the time.

re: BGL has a kits property, but I thought it was unused now - that being said, I think there was some discussion yonks ago that it's a risk to load kits dynamically without intent and there is a strong chance you won't know that a board is included in the BGL.

dglazkov commented 9 months ago

What if we rearrange the deps to do this:

PaulKinlan commented 9 months ago

Ok - that's a bit different from what I have here: https://github.com/breadboard-ai/breadboard/pull/848

I will look at this suggestion though, although we have to teach bb-main and RunConfig on how to dynmically include the kit (that's why I use rollup when I start up the CLI)

The linked PR the UI has two new HTTP endpoints:

  1. kits.json - a list of kits that will be dynamically loaded
  2. /kits/[kit] - the compiled code of the kit.
dglazkov commented 9 months ago

I think #848 is fine as a workaround (and a first step), but I'd like to strive toward a more permanent, better layered solution.

PaulKinlan commented 9 months ago

totally.

PaulKinlan commented 9 months ago

@dglazkov Chatting with Metro Paul today and we had a question that we're trying to work out if we should resolve now or not.

We hard code the kits in to the Web UI, the PR mentioned here can solve that, but we're trying to work out if we should make the switch now and ensure that the UI doesn't hard-code any kits and rather we work out a way to pass them in on the CLI (or hosting) without having to do --kit for everything.

It does raise challenges though:

1) you have to ensure for any starter project you have the kits in a local/accessible node_modules. 2) We will also hit an issue that I hit the other day - because each Kit references breadboard directly and if a Kit is not in-sync with the current version of breadboard we can't add those nodes to the graph (it errors with #scope error). I know @aomarks was looking at a cleaner separation of Breadboard and Kit etc, I really do think we should look at this soon rather than later.

dglazkov commented 9 months ago

t the UI doesn't hard-code any kits and rather we work out a way to pass them in on the CLI (or hosting) without having to do --kit for everything.

Yeah, that seems directionally right!

  1. you have to ensure for any starter project you have the kits in a local/accessible node_modules.

Yup, this seems right, too

  1. We will also hit an issue that I hit the other day - because each Kit references breadboard directly and if a Kit is not in-sync with the current version of breadboard we can't add those nodes to the graph (it errors with #scope error).

This one is important. It comes down to coming up with a stable API for a kit, so that it can be consumed regardless of what version of Breadboard it was built with. Agreed on SRTL. Need to figure this out.

dglazkov commented 9 months ago

I just realized what I think we're looking for:

Shift breadboard debug from being a "just calls into the goop that is breadboard-web" to "uses Breadboard run API directly and relies on breadboard-ui to render results".

PaulKinlan commented 9 months ago

I kind of want to have breadboard run to use the harness work you did in main breadboard, too.

dglazkov commented 9 months ago

okay, crazy idea time. What if breadboard debug brings up a node proxy server that proxies all the kits from the localhost? This way, we wouldn't need to have them be all rolled up with vite.

PaulKinlan commented 9 months ago

Oh - interesting. Very interesting. Given we have breadboard proxy working (and it's running live on one of my apps) it shouldn't be a great stretch to enable this... We know tunnelling works for keys too, so they can still be defined in the UI.

We'd have to teach the UI to also proxy the function names defined by those nodes (I'm still worried about function name clashes in that proxy).

One thing that I would have to work out is more around expectation. Would people expect it to all just be working in the browser, if it's loaded through the web ui?

dglazkov commented 9 months ago

I thought about it some more and we'd still need Kits bundled in with the frontend, because they have the describers for the nodes. If so, then maybe proxying is not really giving us that much.