AI Function Node Iteration

[joepavitt]

Description

• [x] Build a REST API service that wraps the same intelligence running in the function-gpt node.
• [ ] Build a new function-ai?? node that will run on all NR instances on FlowFuse Cloud (and maybe self-hosted if we can work out smooth way of deploying)
• [ ] Configure FF Cloud to auto-include the new function node

Which customers would this be available to

All

Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

[Steve-Mcl]

Design

Having been working on the 1st part (API) and I have stuck with Open AI.

Using functions (example: https://www.datacamp.com/tutorial/open-ai-function-calling-tutorial) I have got it to return parsable data including any additional libs

example response:

This lead me to the thought - what if we didnt have a 3rd full duplicate of the core function node but instead, made this available to FF based Node-RED instances via a plug in?

The plugin, when called could return all of the elements required to simply generate a new function node for the canvas.

Additionally, when the function node is already created in the workspace, we can still call the API via the plugin by adding a prompt in the monaco editor.

Thoughts?

[joepavitt]

So, from a UX perspective, two entry points:

• Plugin sidebar? Ask any request, get a new function node
• Core function node has extra method/input in it as per our node and they can interface with that?

It feels more intuitive and obvious to a user if it's an explicit text input available to use, rather than them having to write comment-based prompts for example

[joepavitt]

Agree with the plugin direction, rather than another node. Coupling tightly with existing function node would be preferable

[ZJvandeWeg]

@joepavitt Which tier of customer is this available to?

[joepavitt]

Would be all - not sure why that was missed

[Steve-Mcl]

Just sharing some progress and a request for feedback

Current operational state (rough n ready)

Generating a function node

Generating a flow

Generating JavaScript while inside a function

Notes

You might wonder why there is a separate "Flow Builder" and "Function Node Builder"... In short, asking the AI to make a function node is super reliable. I provide system prompts to generate plain JavaScript (suitable for a node-red function) and use the OpenAI function + schema to respond with a JS object containing number of outputs, function code, any npm modules it may have used etc. Then, in FF I hydrate that into a well formed Node-RED JSON array containing a single function node. This works very well.

The flow builder is more more suseptible to "halucinations" and gets thigs wrong very often

Lastly the editor "Ask FlowFuse 🪄" uses something called a Code Lens (monaco feature) and calls to the same endpoint as the Function Node builder. This means we can do similar things like adjust the output count and add libs to the setup tab.

Accessing these menu items (please vote which option)

I could not decide (discussed with @joepavitt) how to surface these. we agreed i would post screenshots for comparison. Please take a look and chose what feels best.

Option 1 - front and center (not easy to miss)

Option 2 - top level side menu

Option 3 - added to an existing side sub menu (I chose "view", but it could be any)

Other feedback would be great too

The menu items, dialogs titles / descriptions of the feature are very much thown together. Some feedback and suggestions are welcome.

• Global operations (flow builder / function Node builder)
  • Menu item titles
  • Dialog titles
  • Prompt descriptions on dialog body
  • AI Magic Icon in top header (linked issue https://github.com/FlowFuse/website/issues/2258)
• Inline code action
  • "Ask flowfuse 🪄`" Good? Bad? Other?
  • Dialog title
  • Prompt description on dialog body

[Steve-Mcl]

Future iteration though dump

Via Monaco Editor...

1. Code lens could be used on JSON editor - "Generate an array of 10 objects containing random names and dates"
2. Code lens could be used on the template editor for the various code styles (JSON, CSV, JavaScript, SQL)
3. Code Lens could be used in the Dashboard 2.0 ui-template for assistance building templates (untested at this time)

Additional menu items:

1. Select a bunch of nodes and "Ask FlowFuse what this flow does"
2. More specific prompts that can have fine tuned system hints like "Add an endpoint", "Make a HTTP Request"
   3. Under the hood, this would work similar to the function node builder where specific system hints asked for specific structured response then we hydrate the response into a well formed flow

[Steve-Mcl]

Naming things...

Taking suggestions for module/repository name for this plugin?: @flowfulse/nr-ai-plugin (similar to nr-tools-plugin) @flowfulse/nr-ai-assistant @flowfulse/nr-ai-assist @flowfulse/node-red-ai-plugin other?

[knolleary]

Given the hallucinations, lets keep the scope on the Function node for this iteration (so we can get it released this week) and do the flow builder for v2.

Of the three options, I think Option 1 is probably the best to begin with to get this feature noticed. Burying it under the menu will make it hard to discover.

The Code Lens piece looks really nice 👍🏻

For naming, I'd suggest @flowfuse/nr-assistant - I could see this expanding out to other features which don't necessarily have to be anything to do with AI, but are there to help the user.

Where can I see details of the rest api this uses? We need to make sure the security around it is appropriate.

I presume this will work on the Device Agent as well?

[Steve-Mcl]

@knolleary I had written most of the below when your comment landed (so it may seem a bit mixed up.

Where can I see details of the rest api this uses? We need to make sure the security around it is appropriate.

Regarding integration with FF platform.

currently, this plugin is under local development and uses an endpoint hosted in a Node-RED instance on FFC. This make iteration and improvement super quick.

However, to implement this securely, currently, the endpoint is protected using a single token setup on the instance security settings.

I cannot immediately think of a way to avoid having to bring this endpoint and API into the FF platform.

Can I grab you for 5 mins to discuss a scalable and suitable direction for integration with FF core please?

I presume this will work on the Device Agent as well?

It was not mentioned in scope and is a question I think you have just answered ;) I also think that ties up with the previous point above.

For naming, I'd suggest @flowfuse/nr-assistant

Yes, makes sense & I love it too (bonus).

Given the hallucinations, lets keep the scope on the Function node for this iteration (so we can get it released this week) and do the flow builder for v2.

I agree completely. will remove that menu item for 1st iteration.

Of the three options, I think Option 1 is probably the best to begin with to get this feature noticed. Burying it under the menu will make it hard to discover.

Perfect.