[model-controller] Setup e2e tests for model registry and serving reconciliation

[lampajr]

Is your feature request related to a problem? Please describe. In the first implementation of model registry and serving implemention we focused on creating unit testing by mocking the model registry service because the TestContainer approach was not feasible as it would not work in openshift-ci environment [1]. On the other hand having some ITs where we can actually test the connection to an existing model registry/ml-metadata service could be very helpful and furthermore we could setup real use cases in these tests.

[1] https://cloud.redhat.com/blog/running-testcontainers-in-openshift-pipelines-with-docker-in-docker

Describe the solution you'd like Setup e2e test suite in odh-model-controller where we could setup model-registry service by directly applying a deployment and then run all expected tests.

Describe alternatives you've considered Keep just unit tests, unfortunately we these we cannot cover all possible tests (especially those that writes on model registry are not really testable as we are mocking it)

Additional context Some examples we could take inspiration from:

• https://github.com/opendatahub-io/opendatahub-operator/tree/42b2bdd6eccbab5669b467d44ee9a1a25a3a449e/tests/e2e
• https://github.com/openshift/release/blob/37ce76256337378b57c80f696a36b863cb2ec330/ci-operator/config/opendatahub-io/opendatahub-operator/opendatahub-io-opendatahub-operator-incubation.yaml#L57-L77
• https://github.com/operator-framework/operator-sdk/blob/e001847a67f16d7173ae4fee7cff57cd4405edbb/testdata/go/v3/memcached-operator/test/e2e/e2e_test.go

[lampajr]

Investigating how to implement those e2e tests in the mode generic fashion, such that they will work regardless the infrastructure will actually run them (either openshift_ci or gha).

Tests assumption:

• Kubernetes / Openshift cluster up and running.
• The ODH model controller is deployed and running, this is responsability of the infrastructure to deploy it.