dmytro-dorofeiev commented 5 years ago

Bug Report

What happened: every restarting service-catalog-catalog-apiserver pod erases all information about classes and instances

What you expected to happen: after each restart or upgrade pod shouldn't clear information about classes and instances

How to reproduce it (as minimally and precisely as possible):

Deploy service-catalog and osba helm chart
Create osba instance

Check svcat get instances or kubectl get serviceinstances.servicecatalog.k8s.io that it contains the instance


$ svcat get instances

           NAME                    NAMESPACE             CLASS           PLAN       STATUS
+---------------------------------+-----------------+----------------------+-------
test-postgresql-instance   service-catalog   azure-postgresql-9-6   basic   Provisioning

4. Check that classes exist with svcat get classes or  kubectl get serviceclasses.servicecatalog.k8s.io shouldn't be empty
5. Delete pod service-catalog-catalog-apiserver
`
$ kubectl delete pod $(kubectl get pod | grep service-catalog-catalog-apiserver | awk {'print $1'})
`
or upgrade service-catalog helm chart
`
$ helm upgrade service-catalog  svc-cat/catalog --namespace service-catalog
`
6. Check `svcat get instances` or `kubectl get serviceinstances.servicecatalog.k8s.io` that it contains your instance. In my case it's gone
7. Check that classes exist with `svcat get classes` or  `kubectl get serviceclasses.servicecatalog.k8s.io`. In my case they empty

**Anything else we need to know?**:
After upgrade osba helm chart with the same chart, I see that classes now in place. But the instance is not.

**Environment**:
- Kubernetes version (use `kubectl version`):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T18:55:03Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}


- service-catalog version:

$ helm ls NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE osba 25 Fri Aug 9 17:17:00 2019 DEPLOYED open-service-broker-azure-1.8.2 1.8.2 osba service-catalog 33 Fri Aug 9 19:29:17 2019 DEPLOYED catalog-0.2.1 service-catalog

- Cloud provider or hardware configuration: - **Azure AKS**
- Do you have api aggregation enabled? - **don't know**
- Do you see the configmap in kube-system? - **yes**

osba.v3 1 17d osba.v4 1 16d osba.v5 1 15d osba.v6 1 15d osba.v7 1 15d osba.v8 1 15d osba.v9 1 15d service-catalog.v1 1 18d service-catalog.v10 1 15d service-catalog.v11 1 14d service-catalog.v12 1 14d service-catalog.v13 1 14d service-catalog.v14 1 14d service-catalog.v15 1 14d service-catalog.v16 1 14d service-catalog.v17 1 9d service-catalog.v18 1 9d service-catalog.v19 1 7d5h service-catalog.v2 1 17d service-catalog.v20 1 2d4h service-catalog.v21 1 2d3h service-catalog.v22 1 2d3h service-catalog.v23 1 2d1h service-catalog.v24 1 29h service-catalog.v25 1 3h28m service-catalog.v26 1 3h26m service-catalog.v27 1 3h19m service-catalog.v28 1 3h17m service-catalog.v29 1 3h13m service-catalog.v3 1 17d service-catalog.v30 1 3h12m service-catalog.v31 1 3h11m service-catalog.v32 1 155m service-catalog.v33 1 4m16s service-catalog.v4 1 16d service-catalog.v5 1 15d service-catalog.v6 1 15d service-catalog.v7 1 15d service-catalog.v8 1 15d service-catalog.v9 1 15d

- Does it have all the necessary fields?
    - `kubectl get cm -n kube-system extension-apiserver-authentication -o yaml` and look for `requestheader-XXX` fields

$ kubectl get cm -n kube-system extension-apiserver-authentication -o yaml | grep requestheader requestheader-allowed-names: '[]' requestheader-client-ca-file: | requestheader-extra-headers-prefix: '["X-Remote-Extra-"]' requestheader-group-headers: '["X-Remote-Group"]' requestheader-username-headers: '["X-Remote-User"]'

- Install tools:
  - Did you use helm? What were the helm arguments? Did you `--set` any extra values?

$ helm version Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

$ helm upgrade service-catalog svc-cat/catalog --namespace service-catalog


- Are you trying to use ALPHA features? Did you enable them? - **No**

dmytro-dorofeiev commented 5 years ago

I have to add that after upgrade helm chart with service-catalog or restart service-catalog-catalog-apiserver pod I'm also losing secret in a namespace with connection to database

dmytro-dorofeiev commented 5 years ago

apiserver.storage.etcd.persistence.enabled is disabled by default, set it to true solve the issue.

helm upgrade service-catalog svc-cat/catalog --namespace service-catalog --set apiserver.storage.etcd.persistence.enabled=true

mszostok commented 5 years ago

Hi @timon1984

But please remember that in documentation we are saying that apiserver.storage.etcd.useEmbedded IS INADEQUATE FOR PRODUCTION USE

so if u are running your service catalog in production then you should use external etcd clusters for having HA and scalability.

when we will merge the Replaces the Aggregated API Server with the CustomResourceDefinitions (CRDs) solution then this api-server and ectd will be removed and installation will be simpler :)

dmytro-dorofeiev commented 5 years ago

Hi Mateusz,

thank you for your reply.

Do you have any ETA when it will be merged?

What would you recommend to do at the current moment? We are going to production in the nearest future.

Thanks for your help

пн, 12 авг. 2019 г. в 12:42, Mateusz Szostok notifications@github.com:

Hi @timon1984 https://github.com/timon1984

But please remember that in documentation we are saying that apiserver.storage.etcd.useEmbedded IS INADEQUATE FOR PRODUCTION USE!

so if u are running your service catalog in production then you should use external etcd clusters for having HA and scalability.

when we will merge the Replaces the Aggregated API Server with the CustomResourceDefinitions (CRDs) solution https://github.com/kubernetes-sigs/service-catalog/pull/2630 then this api-server and ectd will be removed and installation will be simpler :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/service-catalog/issues/2686?email_source=notifications&email_token=AAFV6MFM7O6DPMWSU2YJN5TQEEWBNA5CNFSM4IKVVXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4CA6GI#issuecomment-520359705, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFV6MFL3M3CKYEUTBD6IETQEEWBNANCNFSM4IKVVXOQ .

-- С уважением, Дмитрий

mszostok commented 5 years ago

about the ETA here is the epic about that story: https://github.com/kyma-project/kyma/issues/4391

we hope to finish that in two weeks and in next month use it on our platform as a production option, so the final release of the new version with CRD should be ready in about 2months.

For now, you have 3 options:

if you own the k8s cluster then you can configure the api-server against your main ETCD from k8s
if you have managed kubernetes e.g. GKE, AKS etc. then u will not have access to main ETCD so you should deploy your own ETCD cluster (with HA and backups). We are doing that for our platform.
still, use the embedded option but monitor that and perform regular backups in case of some failure.

MHBauer commented 5 years ago

useEmbedded is separate from persistence.enabled https://github.com/kubernetes-sigs/service-catalog/blob/master/charts/catalog/values.yaml#L65-L104

It's not 100% clear to me if with the embedded etcd, but a persistent volume is still not a production ready case.

kubernetes-retired / service-catalog

apiserver pod restart delete all osba classes and instances info #2686

Bug Report