Altinn / digdir-slack-bot

MIT License
2 stars 0 forks source link

[DEMO] Studio Assistant - ask for examples of related validation code #16

Open bdb-dd opened 1 year ago

bdb-dd commented 1 year ago

Description

Demo objective:

Demonstrate one possible assistant flow targeted at new Studio developers with some C# experience.

Concept:

Create new training data set by converting existing C# code in public Altinn 3 app repositories into a repository of validation logic defined in structured English. This allows for a standard RAG approach to querying for relevance while limiting to a very specific domain (ValidateData, ValidateTask, ProcessNext, etc)

Shortcuts:

For demo purposes, we propose limiting the scope to a single domain, ValidateData. Note, many simple validation rules are handled by field-level configuration and not in C#.

User interface:

Interaction can be via Slack chat bot, VS Code plugin, directly in Studio, etc. For this demo, we will extend the existing Studio Assistant demo with a new keyword [validatedata]

Potential:

If the concept works well, it should be possible to generalize to multiple parallel domains, with either an agent-based categorizer.

### Training
- [x] Ingest InstanceValidator.cs files, parsing to include only ValidateData or ValidateTask methods (and dependencies)
- [x] Generate English-language version of validation rule from C# implementation, in structured format
- [x] Generate embeddings from rule summaries
### Inference
- [x] Test inference quality with manual prompts
- [x] Use vector database to find similar documents
- [x] Use document metadata to locate original source file
- [x] Format retrieved code
- [ ] Replace manual prompts with real user input (field name, rule description, error message)
- [ ] Deploy Studio Assistant slack bot to suitable hosting environment for broader testing
- [ ] Test results with a broader audience, identifying quality issues
- [ ] Use Azure OpenAI endpoints instead of OpenAI
### Tasks
bdb-dd commented 8 months ago

Need contractual clarification before proceeding.