spring-cloud / spring-cloud-app-broker

Spring Cloud project for implementing service brokers that conform to the Open Server Broker API specification
http://projects.spring.io/spring-cloud/
Apache License 2.0
26 stars 38 forks source link

Make integration testing a first class citizen #244

Open mheath opened 5 years ago

mheath commented 5 years ago

App Broker users should be able to test their service configuration without invoking the OSBAPI.

For example, if I have configuration like the following:

spring.cloud.appbroker.services:
- service-name: example
  plan-name: standard
  ...

I should be able to write an integration test that looks something like:

@SpringBootTest
public class MyIntegrationTest {

  @Autowired
  AppBrokerInterface appBroker;

  @Test
  public void test() {
    appBroker.provision("example", "standard")
      .serviceInstanceId(...)
      .parameters(...)
      .execute();

    // Code to validate apps, services, etc. were deployed correctly
    appBroker.update ... ;

    // Verify apps, services, etc. were updated correctly
  }

}

This would enable tests that are typically written as acceptance tests to be run on a developer's machine. The tests could be run without having to deploy the service broker and without all the pain that comes from invoking the service broker through CAPI.

royclarkson commented 4 years ago

hmm. we could also potentially eliminate a lot of the mocked OSBAPI requests and responses in our current integration tests. I'm already liking the sound of this. Unfortunately, this feels like an architecture change that would need to happen in a 2.x timeframe.

gberche-orange commented 4 years ago

This relates to discussion with @Albertoimpl at https://gitter.im/spring-cloud-app-broker/community?at=5e67a04585f81e18a8f917ff

+1 for making it easy for authors to test their brokers with a generic life cycle (provision, bind, unbing, update, unprovision), and possibly common fixtures for custom assertions on brokered apps/services, in addition to black box assertions from the returned service bindings.

As an app broker user, I would not be confident with smoke or acceptance tests that diverge much from the target runtime environment. Osb API is quite versatile, and testing a broker outside of a reference OSB client (CAPI or K8S svcat) seems risky. In other words, replacing CAPI calls with OSB API calls (e.g. with an OSB client discussed at https://github.com/spring-cloud/spring-cloud-open-service-broker/issues/77, see sample osbclient impl and its test suite ) would expose authors to risks that such calls get incompatible with CF-originating OSB calls, and that bugs get undetected by the test and slip through to end-users.

I would therefore suggest to make part of the spring-cloud-app-broker-acceptance-tests easily accessible to broker authors, and illustrate this with an example in the spring-cloud-samples/spring-cloud-app-broker-samples repo

spikymonkey commented 4 years ago

Related to #350