Add Stubbed Functional test suite for backend API

[adamdougal]

Motivation

To have ultimate confidence that changes are non-breaking, a stubbed functional test suite should be added. This would reduce the manual testing burden when reviewing PR's, especially dependabot. The benefits of having this in addition to the E2E test suite (https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/419) are:

• Can test error scenarios
• Reduces OpenAI consumptions costs
• Can run locally

How would you feel if this feature request was implemented?

Requirements

• Focussed on the backend API only (suites for other components can come later)
• Include at least one happy path and one unhappy path test
• Functional tests that run against a locally started accelerator that points to mock servers for all downstream dependencies e.g. openai
• Should run for every PR

Tasks

• [x] Create test for /api/config (basic test framework) - https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/438
• [x] Run tests as part of pipeline - https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/438
• [x] Allow overriding of application config in functional tests - https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/441
• [x] Decide on most appropriate mock server for OpenAI - https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/445
• [x] Introduce mock server - https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/447
• [x] Create happy path test for /api/conversation/custom (downstream mock framework required) - https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/504
• [x] Create unhappy path test for /api/conversation/custom - https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/520

[cecheta]

For local storage we can look at using Azurite

[adamdougal]

I've unfortunately run into some issues implementing a test against the /api/conversation/custom endpoint.

• The first issue was that pytest recalls the fixtures for the app subprocess, meaning the tests and the app end up with a different mock server url. This is fixed by using a thread instead.
• The next issue is that the Azure search library, does not let you call the downstream mock server over http, instead it must be over https:

  azure.core.exceptions.ServiceRequestError: Bearer token authentication is not permitted for non-TLS protected (non-https) URLs

• The good news, is that our mock server pytest_httpserver does support running with https enabled https://pytest-httpserver.readthedocs.io/en/latest/howto.html#running-an-https-server
• The bad news, is that because we don't control the http request, we can't turn off SSL verification. Instead we need to configure the certs. We can do this via env vars https://www.python-httpx.org/environment_variables/, however the trustme library only seems to provide us with .pem files, not the .crt files required.

[adamdougal]

21st March 2024 Update: I managed to fix the SSL issue. I was wrong when I stated that we needed a .crt file. I was incorrectly generating the .pem file which is why it was invalid.

I have now raised a PR testing the /conversation/custom endpoint. I will now work on verifying the downstream responses that have been made. It's not currently supported by the mock library but I have raised an issue for it https://github.com/csernazs/pytest-httpserver/issues/293.

[adamdougal]

22nd March 2024 Update

• Two PR's are currently open #519 and #520
• Once merged this ticket is complete
• Unfortunately there will be conflicts after the first is merged, but they should be minimal

[reechamishra]

Based on feedback from team , moving it to DONE status