search

WEKIT-ECS / MIRAGE-XR

MirageXR is a reference implementation of an XR training system. MirageXR enables experts and learners to share experience via XR and wearables using ghost tracks, realtime feedback, and anchored instruction.
Other
28 stars 4 forks source link

Event Trigger for the Virtual Instructor #2080

Open AKissMail opened 1 month ago

AKissMail commented 1 month ago

Refactoring of the VirtualInstructorManager into a VirtualInstructorOrchestrator. Therefore, the business logic of the VirtualInstructorManager has to be adapted. Mainly, the AskClosestInstructor function will be split into multiple functions:

  1. DetermineVirtualInstructor Should determine the relevant Instructor in space. A relevant Instructor is considered to be within the user's field of view and closer than other instructors.

  2. AskClosestInstructor(AudioClip) Takes an Audio Clip and formulates an answer based on the question. The return value contains the answer from the LLM/RAG model as a String and Audio and is based on the result of the LLM. This function uses DetermineVirtualInstructor to choose a VI.

  3. AskClosestInstructor(String) Takes a String and formulates an answer based on that String and the content of the Queue. The return value contains the answer from the LLM/RAG model as a String and Audio and is based on the result of the LLM.

  4. AddToQueueOfClosestInstructor(String) Adds the relevant information as a String to the Queue of the VI. This can later be used as contextual information for a prompt.

Functions 3 and 4 will be listening to events that follow the pattern of the gaze trigger, but it has to be discussed what the final implementation of that looks like.

AKissMail commented 1 month ago

@fwild did I include everything from our discussion?