Update Testing Framework to Replace Mock Responses with Real API Calls

[mfaizanse]

Description

The testing framework has been developed and integrated into the main pipeline, but the current implementation still relies on mock responses when querying the Kyma Companion. To complete the integration, we need to update the function responsible for querying the Companion to use real API calls. This update should involve two API calls: one to initialize the conversation with the agent, and another to query the agent itself. Additionally, the response must be parsed and passed to the evaluator.

Subtasks

0. Introduce the new field into scenario YAMLs with the question to the companion:

   • Update YAMLs with scenarios and include there new field called user_query which would represent the request user makes to the companion. For example, "My deployment is failing" or "Why my app is not working"

1. Implement API Call for Initialization:

   • Develop the functionality to send an API call to initialize the conversation with the Kyma Companion agent.
   • Ensure the initialization request includes the resource_type, resource_name, and namespace as specified in the scenario description.
   • We only use initialization call to initialize conversation with the companion, but we don't use initial questions which were generated for the following requests to the companion. Use user_query field for questions

2. Implement API Call to Query the Agent:

   • Develop the functionality to send a follow-up API call to query the agent after initialization.
   • Capture the response from the agent and parse it appropriately.

3. Integrate with Evaluator:

   • Pass the parsed response from the agent to the evaluator for assessment.
   • Ensure that the evaluation process works seamlessly with the real responses instead of mock data.

4. Testing and Validation:

   • Test the updated framework to ensure that real API calls are functioning correctly and that the entire pipeline—from initialization to evaluation—operates as expected.

Acceptance Criteria

• [x] The new user_query field is introduced into scenario YAMLs
• [x] The function to initialize the conversation with the agent using an API call is implemented and functional.
• [x] The function to query the agent using an API call is implemented and functional.
• [x] Agent responses are correctly parsed and passed to the evaluator.
• [x] The testing framework is fully operational with real API calls replacing mock responses.
• [x] The entire pipeline is tested and validated to ensure accuracy and functionality.

Notes

• Reference method to update

[Teneroy]

Add new task for supporting switching between different tabs and injecting resources if you are on different tab