Unable to update/delete ansible playbook bundles (APB)

[fretboarder]

I have a OpenShift Origin 3.7 cluster running and I'm using the Ansible Service Broker to provision Microsoft Azure services such as CosmosDB and MongoDB. The provisioning/deprovisioning process basically works well but it is not possible to update an existing APB.

To be more specific, I have created a CosmosDB APB with a certain set of parameters. Finally it appeared on the Service Catalog page in the OpenShift console. Later on I updated the APB with new parameters and pushed it to the registry as well. Now, when I try to create the service again still the old set of parameters appear in the input mask.

In the end, I even deleted the Image Stream with the APB from the registry but the CosmosDB service is still being displayed.

I don't understand what the problem is and which components of OpenShift are involved in the entire service catalog workflow. How do I update an APB and how can I even delete an APB?

Any hint/help would be highly appreciated.

[rthallisey]

Later on I updated the APB with new parameters and pushed it to the registry as well. Now, when I try to create the service again still the old set of parameters appear in the input mask.

I've seen this reported one other time. It's something the community will look into. Thanks for the report @fretboarder.

In the end, I even deleted the Image Stream with the APB from the registry but the CosmosDB service is still being displayed.

The UI displays all the apbs the service-catalog is aware of. This is a resource called clusterserviceclass. You can list them this way:

kubectl get clusterserviceclass -o custom-columns=Name:spec.externalName,ID:spec.externalID,BrokerName:spec.clusterServiceBrokerName

A specific instance of a clusterserviceclass is the resource serviceinstance. You should be able to interact with these resources like any other through the cli. kubectl create/delete/edit.

[eriknelson]

@fretboarder Here's an overview of the design, should give you a general idea about the components at play: https://github.com/openshift/ansible-service-broker/blob/master/docs/introduction.md#service-catalog

When an APB is pushed, the broker needs to be "bootstrapped" to run its own discovery again, and then the catalog needs to be "relisted" to force the catalog to sync its clusterserviceclasses with the broker's new inventory. The tooling should handle all three of these things with an apb push. Ultimately, clusterserviceclasses are the objects that are going to drive what's available in the UI.

WRT delete, if you try to delete a clusterserviceclass (APB) and there are outstanding serviceinstances that exist, I believe it will get marked for deletion, but will not actually be removed until the outstanding serviceinstances have been deprovisioned and there are no longer any dangling resources.

The apb tooling has a remove function that I think will remove it from the broker, but I don't believe it will go out to openshift and delete the Image. That's probably an issue I need to double check on.

I'll try and reproduce this today.

[fretboarder]

The UI displays all the apbs the service-catalog is aware of. This is a resource called clusterserviceclass.

@rthallisey Ok, but what happens if there exist multiple docker images within the same ImageStream? We use GitLab-CI to build the APBs and push them into OpenShift's internal docker registry. Whenever we update one of the bundle files (either the apb.aml or the roles/playbooks), a new docker images gets added to the corresponding ImageStream. No we have multiple images with the same name:tag (but different SHA). But, upon bootstrapping the ASB finds all of them, because they all are matching the white_list configuration (observed that by looking into the logs with debug log level). The question is, which image will be mapped to the clusterserviceclass object that is finally displayed in the console?

In that case, just deleting the clusterserviceclass object seems not to be sufficient. You will also have to delete all the corresponding images from the container registry.

So here's what I did to get rid of the services being displayed in the Service Catalog:

1. Delete the corresponding docker images (oc delete image ... => it took me a while to find out that deleting only the ImageStream leaves the images intact...)
2. Delete the clusterserviceclass object.
3. Refresh the Console

At the end, the safest way seems to be to delete all the images with the same tag from the ImageStream before pushing an updated version of the APB. Is this the way it is meant to be done?

@eriknelson Thanks for the info.

BTW: It seems that all the resources related to the service catalog are still not available in bash_completion for the oc client (I have version oc v3.7.1+ab0f056)

$ oc get cluster <TAB pressed> cluster clusterpolicy clusterresourcequota clusterrolebinding clusternetwork clusterpolicybinding clusterrole

[rthallisey]

@rthallisey Ok, but what happens if there exist multiple docker images within the same ImageStream? We use GitLab-CI to build the APBs and push them into OpenShift's internal docker registry. Whenever we update one of the bundle files (either the apb.aml or the roles/playbooks), a new docker images gets added to the corresponding ImageStream. No we have multiple images with the same name:tag (but different SHA). But, upon bootstrapping the ASB finds all of them, because they all are matching the white_list configuration (observed that by looking into the logs with debug log level). The question is, which image will > be mapped to the clusterserviceclass object that is finally displayed in the console?

The newest build image with matching name:tag will be mapped to the displayed clusterserviceclass.

From what I can tell, the image stream functions exactly like docker does locally, where you can have lots of containers with the same name and different IDs, but only one of them has the tag. So as you publish images to the image stream, only the most recently pushed image will be referenced by name:tag while the others will need to be referenced by sha.

In that case, just deleting the clusterserviceclass object seems not to be sufficient. You will also have to delete all the corresponding images from the container registry.

So here's what I did to get rid of the services being displayed in the Service Catalog:

Delete the corresponding docker images (oc delete image ... => it took me a while to find out that deleting only the ImageStream leaves the images intact...) Delete the clusterserviceclass object. Refresh the Console

At the end, the safest way seems to be to delete all the images with the same tag from the ImageStream before pushing an updated version of the APB. Is this the way it is meant to be done?

The broker looks for images in the configured registry adaptors and publishes the clusterserviceclasses to the service catalog periodically. So it makes sense that to ensure everything is cleaned up you also need to delete the image from the registry. You can change this setting with refresh_interval in the broker config.

[rthallisey]

@fretboarder if you are running into any more problems please create another issue or re open this one. Also, if you have any further questions feel free to join us on Freenode in #asbroker.