christad92
commented
2 weeks ago @josephjclark please update this issue with the AI Genserver context.
Closed christad92 closed 2 days ago
christad92
commented
2 weeks ago @josephjclark please update this issue with the AI Genserver context.
josephjclark
commented
2 weeks ago Here is the PR in apollo with this work: https://github.com/OpenFn/apollo/pull/69
Here is the full readme to the chat service, which should be all the technical docs you need? Happy to jump in on a call on this.
The service isn't live yet but should be soon.
And here's a detailed spec for what I think we need:
The UI for this is a very simple HTML panel in the Inspector, probably sitting in the right hand side next to input/output (I actually think we should have input, docs and help on the left, and output and logs on the right).
EDIT: We've decided now to put the AI chat on the left
The UI shows the chat history, probably in simple chat bubbles, but so long as it's clear what the user question is and what the AI response is, it doesn't matter.
Users can type a free input into a text box in the panel, which will be sent up to the server
At the time of writing the AI will return markdown in its results. I think right now it's fine to show markdown in the HTML, although we may want to later either parse the markdown into HTML (we'll get better formatting like this anyway I guess?), or ask the AI to return results as HTML.
I think the UI should be be implemented in Elixier. We need a lot of context from the server and the UI should be very simple. I don't see any benefit in using React.
The GPT backend supports returning the results as stream, resulting in a much better UX. I don't want to support this on day zero, but it is an option available to us.
I suggest for now that when a question is sent, the UI is locked and a loading spinner is shown until the response comes back.
I'll write up detailed usage on the backend API shortly, but here's how it works.
The apollo server is currently sitting at http://34.95.82.109
Lightning will need to POST a JSON payload with all the inspector's context about the job to http://34.95.82.109/services/job_chat. [at the time of writing this API is offline]
The payload can include as much user context as we want: the adaptor, job code, input data, output data, logs, etc. The service right now will take whatever info you give it and add it to the prompt. I suggest we at least send the adaptor and expression.
The payload must also include the chat history, ensuring the AI has context. There is no session maintained by the AI server or the GPT backend.
The payload also includes an api key, which is a little tricky (see the note below)
The server returns a response as JSON object with the content and updated history.
For this to work Lightning will need to save the chat history and associate it with the step id it relates to.
I suggest that there's a 1:1 mapping between a step and its chat. You can't create a new session, you can't clear history, you can't have multiple chats. I don't think we should support a search at this time.
I think eventually we should include a certain number of tokens as part of a user's plan, and show the user how many tokens they have left etc.
But I don't think we should do any of that right now.
I would make this version only available to admin users, and generate a new openai key for lightning (I can do this if it'll help)
The apollo server is not battle tested - hell it's barely tested - and we should anticipate it being offline for a long period of time.
We should think about error handling and what could go wrong.
Perhaps we need to poll the server and check that the job_chat endpoint is responding, and if not, report an error to the user?
We have implemented an AI service that can help users write better job codes or find wrong patterns in their job codes by providing simple prompts and we want to integrate this with Lightning.
As a user, on the editor page,
Design: https://www.figma.com/design/4hGWdO4sajnjcOwE81j8BK/App-(new)?node-id=4492-680&t=sEmPLXFk5O3YAVwy-0
This is related to https://github.com/OpenFn/apollo/issues/52