The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.
Azure Cognitive Search has skillsets for language detection/processing. Exploration will be required to determine the best implementation. Should experiment with German, Italian, and English as languages to be tested, as these are currently being used in an active customer engagement.
### Tasks
- [x] Determine if the default Standard Lucene language analyzer is sufficient
- [x] Determine support guidance for index creation and querying (e.g. blended or language-specific indexes)
- [x] Add LanguageDetectionSkill support
- [x] Create language analyzer settings `analyzers`, `tokenizers`, `token_filters`, etc to `search_config.json` and update SearchIndexClient settings
- [x] Set data ingestion limits/constraints on chunk size to adhere to maximum record size (i.e. 50k characters) - see `search_config.json`.
Azure Cognitive Search has skillsets for language detection/processing. Exploration will be required to determine the best implementation. Should experiment with German, Italian, and English as languages to be tested, as these are currently being used in an active customer engagement.