search

breadboard-ai / breadboard

A library for prototyping generative AI applications.
Apache License 2.0
208 stars 31 forks source link

Show possible mistakes when graphs don't end on an `output` node. #154

Open seefeldb opened 1 year ago

seefeldb commented 1 year ago

When hacking on the new runner, I added a done event that gives a bit of data about the last nodes that left some of their downstream nodes with missing inputs, and my cli tool outputs that if the last node wasn't an output node. This helped me pin down bugs a few times already. Example output:

◇  done ──────────────────────────────────────────────────────╮
│                                                             │
│  Last invoked node runJavascript-2 not an output.           │
│  These last invoked nodes led to incomplete opportunities:  │
│                                                             │
│  runJavascript-2:                                           │
│   receiving nodes:                                          │
│   incomplete nodes: output-1 (text)                         │
│   unused keys: text                                         │
│                                                             │
│                                                             │
├─────────────────────────────────────────────────────────────╯

(the bug here was that the wire was result->text instead of text->text)

I'll add unused wires as well.

But mostly, this would be great to have as well in breadboard-web. Happy to walk anyone through the new bits.

paullewis commented 1 year ago

Oh I really like this idea! Is there a particular board I should look at to get the extra data out of done?

seefeldb commented 12 months ago

it's working with the new runner code, basically:

the event is just a way to get it out. if we go with this new runner eventually, you might anyway build a debugger via the callbacks directly, e.g. the beforehandler one returns a promise that the runner waits for, so makes it easy to implement a pause button (just return the promise that gets resolved when unpausing, etc.).

you can build the same in the current runner of course, it might just require a bit of plumbing to get all the information out.

paullewis commented 10 months ago

Just revisiting this and wondering if the EndResult should carry some of this information? Wdyt, @dglazkov?

seefeldb commented 10 months ago

Yes! I'll try to get wrap up the new runner code this or early next week and have it feed the new harness API.

dglazkov commented 10 months ago

@seefeldb sgtm. Do you want me to sketch out the EndResult protocol based on https://github.com/breadboard-ai/breadboard/issues/154#issue-2038456518 or just wait for you to switch over?