kadel commented 6 years ago

It is not possible to create two services of the same type in one project :-(

▶ odo service create postgresql-persistent
Service 'postgresql-persistent' was created.

▶ odo service list
NAME                      TYPE                      STATUS
postgresql-persistent     postgresql-persistent     ProvisionedSuccessfully

▶ odo service create postgresql-persistent mypost
Service 'mypost' was created.

▶ odo service list
NAME                      TYPE                      STATUS
mypost                    postgresql-persistent     ProvisionRequestInFlight
postgresql-persistent     postgresql-persistent     ProvisionedSuccessfully

▶ odo service list
NAME                      TYPE                      STATUS
mypost                    postgresql-persistent     Provisioning
postgresql-persistent     postgresql-persistent     ProvisionedSuccessfully

▶ odo service list
NAME                      TYPE                      STATUS
mypost                    postgresql-persistent     DeprovisionBlockedByExistingCredentials
postgresql-persistent     postgresql-persistent     ProvisionedSuccessfully

It looks like this is caused by Secret. Every new postgress service creates a secret that is called postgresql (similarly for mysql there is a mysql secret). This is probably causing conflicts during the second instance provisioning.

▶ oc get secrets
NAME                                   TYPE                                  DATA      AGE
96fb37ba-d214-11e8-a200-0242ac110007   Opaque                                0         23m
builder-dockercfg-7k5g5                kubernetes.io/dockercfg               1         2h
builder-token-b4lqx                    kubernetes.io/service-account-token   4         2h
builder-token-cq8mq                    kubernetes.io/service-account-token   4         2h
d2078e6e-d20e-11e8-a200-0242ac110007   Opaque                                0         1h
default-dockercfg-tbc5p                kubernetes.io/dockercfg               1         2h
default-token-fp4jw                    kubernetes.io/service-account-token   4         2h
default-token-z4tk5                    kubernetes.io/service-account-token   4         2h
deployer-dockercfg-4vfn5               kubernetes.io/dockercfg               1         2h
deployer-token-trqpj                   kubernetes.io/service-account-token   4         2h
deployer-token-xq766                   kubernetes.io/service-account-token   4         2h
postgresql                             Opaque                                3         1h
postgresql-persistent                  Opaque

I'm adding it here just for the record.

This is not something that we have to address in this PR. I think that we can live with this for now ;-) I'll convert this to the Issue and we can look at this later.

Originally posted by @kadel in https://github.com/redhat-developer/odo/pull/723#issuecomment-430649256

geoand commented 6 years ago

@kadel Could you assign this to me please?

jorgemoralespou commented 6 years ago

there's a prefix capability in the web UI that they did just for this

geoand commented 6 years ago

@jorgemoralespou do you know why this is? I still haven't figured out the root cause of this problem

jorgemoralespou commented 6 years ago

@geoand you have created a service with a fixed name, as you haven't provided one from CLI, and this is not using the random name generation. That means that second time you run it already exists.

You should instead give the service a name, for now:

$ odo service create postgresql-persistent postgres1
Service 'postgres1' was created.
$  odo service create postgresql-persistent postgres2
Service 'postgres2' was created.
$ odo service list
NAME                      TYPE                      STATUS
postgres1                 postgresql-persistent     ProvisionRequestInFlight
postgres2                 postgresql-persistent     Provisioning

geoand commented 6 years ago

Very helpful! Thanks @jorgemoralespou !

geoand commented 6 years ago

Note to self: check out this :)

geoand commented 6 years ago

@jorgemoralespou even when use different names like you suggest, the provisioning still doesn't happen.

I see:

NAME         TYPE                  STATUS
jenkins1     jenkins-ephemeral     ProvisionRequestInFlight
jenkins2     jenkins-ephemeral     Provisioning

and then

NAME         TYPE                  STATUS
jenkins1     jenkins-ephemeral     Deprovisioning
jenkins2     jenkins-ephemeral     Deprovisioning

Then I get the initial statuses in a sort of loop :)

Actually, I couldn't even get this scenario to work from the Openshift Console - meaning that I tried to create 2 Jenkins services from Openshift Console one after the other and I was seeing the same behavior :confused: . So I am not really sure what we can do in this case.

jorgemoralespou commented 6 years ago

What OpenShift release are u using? If I recall correctly ocp 3.10.14 had a service catalog race condition where provisioning 2 services of the same type too fast gave an error. Okd 3.10 has same issue. This should be fixed in later releases.

Can you check what error the provisioning provides looking at the service catalog and broker logs?

jorgemoralespou commented 6 years ago

One way to test is provision one, wait until completed and then provision another. That should work.

geoand commented 6 years ago

I am using OKD 3.10 (from Minishift).

And yes, I did see that when I wait for the provisioning to complete for the first service, the second one could then be provisioned :)

geoand commented 6 years ago

So now I am hitting another problem on the Openshift Console, which is exactly the same as the problem reported @kadel that we have with odo.

I provision jenkins-ephemeral and opt-in to create the secret. Jenkins is eventually provisioned and service. Now when I try to provision another Jenkins instance while again selecting to create the secret, I see that the status of the second service is: DeprovisionBlockedByExistingCredentials

jorgemoralespou commented 6 years ago

@geoand this sounds like something related to how you're provisioning the service as the one from the UI is probably thinking it's the same serviceinstance.

Happy to help you further diagnose via BJN or if you give me access to the cluster.

geoand commented 6 years ago

@jorgemoralespou I'll probably take you up on the offer next week (or the week after if you don't have time).

Just FYI, I am using Minishift 1.24 with OKD 3.10 and starting it like so: MINISHIFT_ENABLE_EXPERIMENTAL=y minishift start --extra-clusterup-flags="--enable=*,service-catalog,automation-service-broker,template-service-broker"

jorgemoralespou commented 6 years ago

@geoand can you do a yaml dump of the resources that are created when you do it with odo (serviceintance, servicebinding, ...) and the same to when you create them via UI. That might give you a hint.

geoand commented 6 years ago

I'll get you those on Monday when I get access to the computer again 😉

On Fri, Oct 19, 2018, 17:06 Jorge Morales Pou notifications@github.com wrote:

@geoand https://github.com/geoand can you do a yaml dump of the resources that are created when you do it with odo (serviceintance, servicebinding, ...) and the same to when you create them via UI. That might give you a hint.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/redhat-developer/odo/issues/843#issuecomment-431375028, or mute the thread https://github.com/notifications/unsubscribe-auth/AELBv6WQoKS8ijZFF6k-ej7HRPBwV6gzks5umdx-gaJpZM4Xs-f2 .

On Oct 19, 2018 17:06, "Jorge Morales Pou" notifications@github.com wrote:

@geoand https://github.com/geoand can you do a yaml dump of the resources that are created when you do it with odo (serviceintance, servicebinding, ...) and the same to when you create them via UI. That might give you a hint.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/redhat-developer/odo/issues/843#issuecomment-431375028, or mute the thread https://github.com/notifications/unsubscribe-auth/AELBv6WQoKS8ijZFF6k-ej7HRPBwV6gzks5umdx-gaJpZM4Xs-f2 .

geoand commented 6 years ago

@jorgemoralespou It does look the Openshift UI thinks it's the same serviceinstance being provisioned. When provisioning Jenkins for example, I provision the first instance (without creating the servicebinding), everything works as expected. However when I try to provision the second one, the pod if the first instance is taken down and then started as part of the second serviceinstance.

If I add the servicebinding to the mix, then the second provisioning never works at all :(

jorgemoralespou commented 6 years ago

@geoand I have tried to create 2 services of the same type, and the second one is not provisioned:

NAME                    TYPE                  STATUS
nationalparks-db        mongodb-ephemeral     ProvisionedSuccessfully
nationalparks-py-db     mongodb-ephemeral     DeprovisionBlockedByExistingCredentials

The name of the resources created are dependant on a parameter called DATABASE_SERVICE_NAME, so if I do create the service with this parameter and different value, I can deploy both services:

$ odo service create mongodb-ephemeral nationalparks-db
$ odo service create mongodb-ephemeral nationalparks-py-db -p DATABASE_SERVICE_NAME=nationalparks-py-db

$ odo service list
NAME                    TYPE                  STATUS
nationalparks-db        mongodb-ephemeral     ProvisionedSuccessfully
nationalparks-py-db     mongodb-ephemeral     ProvisionedSuccessfully

@geoand This service name is part of the template and it's not uniform amongst services created via a template :-( so not sure how easy is to provide an easy fix for this, in non-interactive mode.

@kadel FYI

geoand commented 6 years ago

Ah yes, this issue 😎

I remember looking into this a little but not coming up with an easy fix. We will have to investigate futher

jorgemoralespou commented 6 years ago

Easy tests.

Create a new project and application:

odo project create test
odo app create test

Create a service:

$ odo service create mongodb-ephemeral database
Service 'database' was created.

$ odo service list
NAME          TYPE                  STATUS
database      mongodb-ephemeral     ProvisionedSuccessfully

$ oc get secret | grep Opaque
1376e765-e126-11e8-b005-0a580a010008   Opaque                                0         5m
mongodb                                Opaque                                4         5m
database                               Opaque                                5         5m

The first secret is the one with the provided parameters to the servicebroker via the command line, hence empty.
The second one, mongodb is the created secret with the default parameters provided from the template to the service broker by the TSB.
The third one, database is created by the ServiceBinding and are the credentials type of secret.

I delete the service via odo service delete database.

Now, I create with a parameter:

$ odo service create mongodb-ephemeral database -p DATABASE_SERVICE_NAME=mydatabase
Service 'database' was created.

$ odo service list
NAME          TYPE                  STATUS
database      mongodb-ephemeral     ProvisionedSuccessfully

$ oc get secret | grep Opaque
9a7d4621-e127-11e8-b005-0a580a010008   Opaque                                1         0s
mydatabase                             Opaque                                4         0s
database                               Opaque                                5         1m

The first secret is the one with the provided parameters to the servicebroker via the command line, hence 1 parameter DATABASE_SERVICE_NAME=mydatabase.
The second one, mongodb is the created secret with the default parameters provided from the template to the service broker by the TSB.
The third one, database is created by the ServiceBinding and are the credentials type of secret.

I delete the service via odo service delete database.

Now, I create the mongodb-ephemeral from UI, providing only the servicename=mydatabase and create a binding.

$ oc get secret | grep Opaque
653c9bff-e128-11e8-b005-0a580a010008        Opaque                                5         1m
mongodb-ephemeral-parameters12x26           Opaque                                1         1m
mydatabase                                  Opaque                                4         1m
mongodb-ephemeral-x5sf6-credentials-jtn1e   Opaque                                5         51s

As the instance was provisioning, only the first 3 secrets existed. Once the instance was provisioned, the fourth secret mongodb-ephemeral-x5sf6-credentials-jtn1e was created, which is the ServiceBinding created secret.

Secret 653c9bff-e128-11e8-b005-0a580a010008 has the values provided in the template. (These are default values existing there)
Secret mongodb-ephemeral-parameters12x26 are the same in json format, with an owner of ServiceInstance.
Secret mydatabase is created by the template instantiation.

geoand commented 6 years ago

Thanks for the analysis @jorgemoralespou!

geoand commented 5 years ago

I tried this again and I am indeed seeing the same problem with postgresql from the template service broker. The interesting thing however is that the same problem does not exist when using postgresql from the openshift ansible broker (version v3.11).

For example the following scenario works:

odo service create dh-postgresql-apb postgresql2 --plan dev -p postgresql_user=luke -p postgresql_password=secret -p postgresql_database=my_data -p postgresql_version=9.6
odo service create dh-postgresql-apb --plan dev -p postgresql_user=luke -p postgresql_password=secret -p postgresql_database=my_data -p postgresql_version=9.6
odo service list

yields:

NAME                  TYPE                  STATUS
dh-postgresql-apb     dh-postgresql-apb     ProvisionedAndBound
postgresql2           dh-postgresql-apb     ProvisionedAndBound

jorgemoralespou commented 5 years ago

@geoand This is because the APB is using the name that you're providing as service name, whilst the template is using a parameter for that.

geoand commented 5 years ago

Thanks @jorgemoralespou

What do you think is the best way to handle this? Adapt to the broker?

jorgemoralespou commented 5 years ago

IMHO this is a flaw from the template service broker implementation that we can not easily fix.

I would probably start a discussion with the devexp team, who owns the template-service-broker to see if there's a way to fix this on their side. Else, we need to have checks on the code, which is very cumbersome, and not sure if it would get this type of problem.

geoand commented 5 years ago

IMHO this is a flaw from the template service broker implementation that we can not easily fix.

I would probably start a discussion with the devexp team, who owns the template-service-broker to see if there's a way to fix this on their side. Else, we need to have checks on the code, which is very cumbersome, and not sure if it would get this type of problem.

Thanks @jorgemoralespou , your input on this is very much appreciated!

jorgemoralespou commented 5 years ago

maybe involve @bparees

Hey, Ben, quick recap of the issue: When we instantiate services from the service catalog, the name of the service/dc ...can be provided if the service is provisioned by the automationbroker but when it's provisioned by the TSB it needs to be provided via a parameter (usually templates use a parameter for the service names). This causes that if we programmatically instantiate two services via the catalog API, if the service is provided by the ASB, they deploy correctly, as each gets a randomly generated name (if not explicitly provided). When the service comes from the TSB, the second service fails, as the resources will have the same name for both instances (unless the param is explicitly provided).

We would like to have a consistent way of invoking the API so that no matter where the service comes from, two services of the same type can be instantiated.

Can you provide us any guidance or point us to the right people to get this answer?

Thanks

bparees commented 5 years ago

The TSB is owned by the service catalog/broker team lead by @jwmatthews so i'd start with him.

But the short answer would probably be to change the templates to use generated parameter values (the template feature). That would give you likely unique (but not guaranteed unique) names for things without you having to pass a parameter value on instantiation.

If you don't control the templates in question, then I don't have another suggestion.

jorgemoralespou commented 5 years ago

@bparees We don't control the templates, as this would be anything provided by the catalog in any deployment.

As I see it, any possible solution would eventually require to modify the templates.

Thanks.

jwmatthews commented 5 years ago

@jorgemoralespou my initial thoughts are that this would be best addressed via the templates. The Template Service Broker is in maintenance as leveraging Operators is the path forward we are investing in.

If you have some ideas of how we could solve your problem in the TSB, happy to discuss and see if we can include, but I'm not seeing a straight forward solution at this point.

cc @johnkim76 @eriknelson

jorgemoralespou commented 5 years ago

@jwmatthews the only way I can think of is by adding an annotation to the templates that the servicecatalog would know how to manage, like we do for some other things (https://github.com/openshift/library/blob/master/official/sso/templates/sso72-https.json#L16-L18) but that would require to modify every template :-(

And whilst operators will become a thing (in the future), templates will still be around for long, long time, hence, we need to support them.

I guess, we need a nice way to error here.

kadel commented 4 years ago

odo service as we are slowly moving towards Operator backed services the service command is being reworked. We can look into this as part of that work, and fix this issue for Operator backed services.

2161

/priority medium /remove-priority high

openshift-bot commented 4 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

girishramnani commented 4 years ago

We are aware of this limitation /lifecycle frozen

valaparthvi commented 3 years ago

I don't think this is an issue any longer. Closing this issue.

redhat-developer / odo

Can't create two services of the same type #843

2161