[BC Idea]: [Number Series Copilot] Implement Semantic Text Comparison for Enhanced Number Series Matching

[DmitryKatson]

BC Idea Link

https://experience.dynamics.com/ideas/idea/?ideaid=ae4b0c3a-30d6-ee11-92bd-000d3adb9365

Description

New Feature: Semantic Comparison for Number Series Copilot

Feature Request: Implement Semantic Text Comparison for Enhanced Intent Understanding

• Current Behavior: The Number Series Copilot relies on a keyword-based comparison to match user requests with the corresponding entities in Business Central. For example, when a user asks for an employee Number Series, the system works as expected because it can find a match with the "Employee Nos." keyword in the "Human & Resources Setup" table.

• Problem: If a user uses a synonym or a different phrase, such as "payroll worker," the current keyword comparison does not find the corresponding entity (e.g., "employee"), leading to failure in generating the Number Series.

• Proposed Solution:

  • Implement a semantic comparison mechanism using a cosine similarity algorithm to compare user requests with existing entities, even if synonyms or different phrases are used.
  • Semantics Vocabulary Table: Build a vocabulary of requested entities (e.g., "employee", "payroll worker") and existing setup tables and no. series fields (e.g., "human resources employee") with text values mapped to vectors.
  • Vectors: Generate and store vectors in the blob field of this semantic vocabulary when the user installs the app or upon the first query.
  
  • Azure OpenAI Embeddings: If a match is not found in the predefined vocabulary, use Azure OpenAI Embeddings to generate vectors for the new phrase or term.
  • Embedding Deployment: To integrate this feature, add text-embedding-ada-002 embedding model to the AOAIDeployments.GetEmbeddingAda002(). Ensure that the Azure OpenAI Embeddings are accessible for use when a semantic comparison is required.

---

Steps to Implement:

• Build a semantic vocabulary of text values with corresponding vectors for entity name comparison.
• Store the vectors in Blob field during installation or first use.
• Integrate Azure OpenAI Embeddings for cases where the vocabulary does not provide a match.
• Ensure the Number Series Copilot can dynamically switch between keyword comparison and semantic comparison as needed.

---

Acceptance Criteria:

• Semantic comparison allows the Copilot to correctly interpret synonymous terms (e.g., "payroll worker" as "employee") and generate the correct Number Series.
• Integration with Azure OpenAI Embeddings for real-time semantic comparison when no match is found in the stored vocabulary.

I will provide the implementation for this BC Idea

• [X] I will provide the implementation for this BC Idea Internal work item: AB#503148

[JesperSchulz]

I will consult our platform team to hear about their plans to make semantic search available for AL developers. Stay tuned.

[JesperSchulz]

Platform will collaborate with you on providing support. We will park this issue for now!