Steve-Mcl
commented
2 weeks ago As discussed, as a first step, I have setup demos on a dashboard for OpenIA, Gemini 1.0 Pro and Gemini 1.5 Pro - here: https://inquisitive-pacific-swift-9466.flowfuse.dev/dashboard (FF Auth)
Unfortunately, after trying 2 different accounts, trying different network connections I am unable to "top up" for testing the OpenAI GPT API, so the GPT trials were not performed via API but rather via its web app.
Issues:
- Flows often missing starter inject node
- Sometimes, the nodes settings are so misconfigured the node edit panels TypedInputs are unable to initialise
- Mistaking HTTP Request with HTTP In node
- Forgetting to add the HTTP Response node when building an endpoint
- Config nodes settings are the same but have different ID and name (would cause duplication in users flows)
- Non deterministic results - posing the same Q results in different answers. Gemini does however have a means of enabling deterministic responses.
- Many flows simply do not work or are not what was requested unless the prompt is super explicit and uses terms more advanced or knowledgeable NR users might state (which sort of defeats the purpose)
- Slow: +20 secs was not uncommon for anything more complex than "Send a value over MQTT"
Recommendations:
Fine Tuning
While testing the LLMs I added Q+As to the data each time I got a nonsense result and it did improve the quality of a response. This however uses tokens and slows down the overall response. Both Gemini and GPT have the facility to fine tune. They require at least 500 Q+As to be effective (and more is better). To make a model that is more accurate and reliable, I suspect this will be a necessary task. Gemini also offers the ability to "slim down" a model which focuses it and improves response times.
Curated
To limit scope and improve accuracy we could add an initial primer e.g. a first field on the AI Flow Builder might be a dropdown that asks the user to chose an option that best describes the flow they are building (e.g. "A CRUD API", "a Dashboard for collecting user data") This list would be tightly aligned with and Fine Tuning Q+As we add for steering the LLM
Post processing
I believe a level of post processing will be needed. For example, during tests involving nodes with config nodes, the LLM returned
the same config setup but with different name and ID. This leads to the flows adding multiple same configs to a users flows. We may wish to
consider a level of post-processing to ensure things like MQTT nodes are always pointed at localhost:1883 and have the same ID.
Another place where post processing might be necessary is with inject/change/switch nodes where the value was misconfigured such that the nodes edit panel could not initialise the TypedInput correctly. This may of course be fixed or reduced with lots of fine tuning.
joepavitt
Create an API endpoint (whether hosted in FlowFuse Cloud or elsewhere) that given a text-based prompt, returns a
flow.jsonthat can be deployed to Node-RED.Service or Self-Hosted:
Do we build/train/host our own LLM, or utilise an existing LLM as a Service? My gut would say, especially for first iteration, the latter is sensible.
Staging Prompts
Need to experiment with sending config/setup prompts to whichever AI service we use in order to ensure our prompts always return a valid Node-RED flow, etc.
You can see the setup prompts we use with
function-gptnode here: https://github.com/FlowFuse/node-red-function-gpt/blob/42d3eeb7a28bef1a5d1b4bffb76a2906e0f8389f/config/index.js#L46-L49API Hosting
Do we provide a single endpoint that all FlowFuse instances can call, or do we make this a configuration on a FF instance-by-instance basis, and consequently each FF-instance has it's own endpoint to call?
Quality of Results
We should be flagging this as "Alpha" or "Beta" for some time to caveat quality of results given the hallucinations of LLMs