search

spring-cloud / spring-cloud-app-broker

Spring Cloud project for implementing service brokers that conform to the Open Server Broker API specification
http://projects.spring.io/spring-cloud/
Apache License 2.0
26 stars 38 forks source link

Can App Broker provide more assistance around concurrency control? #308

Open spikymonkey opened 4 years ago

spikymonkey commented 4 years ago

We've hit a problem in our App Broker based broker when an unbind operation races with a service instance deletion. Our unbind involves an update of the SI backing app, so of course this goes wrong when the deletion operation starts concurrently tearing down the app and space that contains it.

We'll need to handle this, I guess, in our ServiceInstanceService and ServiceInstanceBindingService implementations or their collaborators, by detecting the concurrent operations and responding accordingly. That boils down to tracking some extra state and then throwing instances of Spring Cloud Open Service Broker's ServiceBrokerConcurrencyException when needed.

I started to wonder if this is something App Broker could abstract away for us. What if, before proceeding with each operation, App Broker asked the implementing app 'hey, operations A, B and C are already in flight for service instance X (okay, three at once is unlikely, but let's not limit ourselves...) - can we also start operation D? Yes or no?' The default answer should always be 'no' to avoid implementing brokers hitting the same kind of concurrency issues we have.

poblin-orange commented 4 years ago

Experiencing concurrency issues with an Openshift OSB client. Not sure the broker sending concurrent access exception is correct wrt Osb spec. Could a "per service instance" lock be set (eg: based on Spring Cloud lock https://github.com/spring-cloud-samples/locks/blob/master/src/main/java/locksdemo/RedisLockService.java ) ?

gberche-orange commented 4 years ago

See related PR #343 which uses the service instance state repository to handle concurrent provision/update/delete

Similar mechanism could be used to handle race condition between service unbinding and service instance unprovisionning