Add test::mock providers

[ashwinb]

🚀 The feature, motivation and pitch

We have a decently flexible testing system for testing various combinations of providers when composing a Llama Stack.

See https://github.com/meta-llama/llama-stack/blob/main/llama_stack/providers/tests/README.md

We need to add some mocks / fakes so it is easy quickly test through dependent code without provisioning heavy resources (especially for inference / tool calling)

Alternatives

No alternatives

Additional context

No response

[vladimirivic]

Taking this one. I spent some time reviewing the code and figuring out how we can do this in a way that is easy to setup yet generic enough to work seamlessly with the existing tests.

Proposal details:

Point 1

• Have a generic “test::test-fake” provider fixture. This fixture will take config that will set the Api -> TestFake mappings.
• For simplicity, those mappings should provide fully setup fake objects:

# Test fake config
class TestFakeProviderConfig(BaseModel):
  impls: Dict[Api, Any]

# Example:
memory_impl = TestFakeMemory()
memory_banks_impl = TestFakeMemoryBanks()

config = TestFakeProviderConfig(
    impls={
        Api.memory: memory_impl,
        Api.memory_banks: memory_banks_impl,
    }
)

 ProviderFixture(
    providers=[
        Provider(
            provider_id="test::test-fake",
            provider_type="test::test-fake",
            config=config.model_dump(),
        )
    ],
)

test_stack = test_fake_stack_fixture(config)

Point 2

• Each provider test module implements its own test fakes that implement provider Api behaviors just like any provider implementation.
• Those test fakes are used as fixtures and are instantiated during TestFakeProviderConfig creation.
• During test fake instantiation we can stub/mock test fake's methods
  • That way all setup is localized to specific providers test module.

# Fakes and their setup (dependency injection and stubbing)
memory_banks_impl = MemoryBanksTestFakeImpl()
memory_impl = MemoryTestFakeImpl()
memory_impl.set_memory_banks(memory_banks_impl)
memory_impl.set_stubs("query_documents", query_documents_stubs)

config = TestFakeProviderConfig(
    impls={
        Api.memory: memory_impl,
        Api.memory_banks: memory_banks_impl,
    }
)

Point 3

• With those changes, we should be able to have generic yet enough customizable approach where custom changes like stubbing and fake setup are localized inside each provider's test module.
  • Inside provider's test module
  • We implement test fakes
  • We instantiate fakes and setup stubs during fixture setup
  • Inside parent test module (extended by all tests)
  • We implement support for validating and instantiating generic "test::test-fake" providers

[vladimirivic]

Updating proposal. Actually we do not want to mock/fake provider as “test::test-fake” provider. Instead we want to initiate mocks for underlying APIs while provider fixtures can go through the full setup process.

Main points:

• Provider tests will support --mock-overrides option which will specify which provider fixtures should use mock overrides
• Mock naming will follow the same naming convention as provider fixtures so we can use mock dynamic setup
• Provider tests that want to implement mock support should write mocks inside mocks.py

This way when developing support for new provider, writing mocks becomes a part of the implementation workflow but it's intuitive to follow and easy to implement.