cf unbind-service error

[tanjoht]

currently have version 0.5.0 of the broker deployed and seeing this on an instance that was renamed (and renamed back to the original). This instance may have been created prior to 0.5.0 where there were issues with renaming instances.

The error is: Error: An unbind operation for the service binding between app mp-adapter and service instance mongodb-dso failed: Service broker error: cannot find plan for instance “instance id”: instance metadata has the wrong type map[string]interface {}

[jasonmimick]

Waiting on more info from customer

[caspertsui]

Hi @jasonmimick ,

Logs from Mongo Atlas OSB, please have a look.

2020 - 10 - 29 T11: 53: 43.656 + 08: 00[APP / PROC / WEB / 0][OUT] {
  "level": "error",
  "ts": 1603943623.6556885,
  "caller": "broker/instance_operations.go:341",
  "msg": "Cannot find instance in maintenance DB",
  "func": "getInstance",
  "instanceID": "c20bdcea-6bf1-4804-ab06-bed618054e7d",
  "orgID": "5f08682e11d6660eebe9928e",
  "error": "value with name \"c20bdcea-6bf1-4804-ab06-bed618054e7d\" not found",
  "stacktrace": "github.com/mongodb/atlas-osb/pkg/broker.Broker.getInstance\n\t/tmp/app/pkg/broker/instance_operations.go:341\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getInstancePlan\n\t/tmp/app/pkg/broker/broker.go:138\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getPlan\n\t/tmp/app/pkg/broker/broker.go:153\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getClient\n\t/tmp/app/pkg/broker/broker.go:179\ngithub.com/mongodb/atlas-osb/pkg/broker.Broker.Unbind\n\t/tmp/app/pkg/broker/binding_operations.go:136\ngithub.com/pivotal-cf/brokerapi/handlers.APIHandler.Unbind\n\t/home/vcap/go/pkg/mod/github.com/pivotal-cf/brokerapi@v5.1.0+incompatible/handlers/unbind.go:56\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/mongodb/atlas-osb/pkg/broker.authMiddleware.func1.1\n\t/tmp/app/pkg/broker/auth.go:42\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/vcap/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2831\nnet/http.(*conn).serve\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:1919"
}
2020 - 10 - 29 T11: 53: 43.656 + 08: 00[APP / PROC / WEB / 0][OUT] {
  "level": "error",
  "ts": 1603943623.655811,
  "caller": "app/logger.go:75",
  "msg": "unknown-error",
  "error": "cannot find plan for instance \"c20bdcea-6bf1-4804-ab06-bed618054e7d\": cannot fetch instance: cannot find instance in maintenance DB(s): no instances found",
  "errorVerbose": "cannot find instance in maintenance DB(s): no instances found\ngithub.com/mongodb/atlas-osb/pkg/broker.Broker.getInstance\n\t/tmp/app/pkg/broker/instance_operations.go:348\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getInstancePlan\n\t/tmp/app/pkg/broker/broker.go:138\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getPlan\n\t/tmp/app/pkg/broker/broker.go:153\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getClient\n\t/tmp/app/pkg/broker/broker.go:179\ngithub.com/mongodb/atlas-osb/pkg/broker.Broker.Unbind\n\t/tmp/app/pkg/broker/binding_operations.go:136\ngithub.com/pivotal-cf/brokerapi/handlers.APIHandler.Unbind\n\t/home/vcap/go/pkg/mod/github.com/pivotal-cf/brokerapi@v5.1.0+incompatible/handlers/unbind.go:56\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/mongodb/atlas-osb/pkg/broker.authMiddleware.func1.1\n\t/tmp/app/pkg/broker/auth.go:42\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/vcap/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2831\nnet/http.(*conn).serve\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:1919\nruntime.goexit\n\t/tmp/contents998050889/deps/0/go1.13.15/src/runtime/asm_amd64.s:1357\ncannot fetch instance\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getInstancePlan\n\t/tmp/app/pkg/broker/broker.go:140\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getPlan\n\t/tmp/app/pkg/broker/broker.go:153\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getClient\n\t/tmp/app/pkg/broker/broker.go:179\ngithub.com/mongodb/atlas-osb/pkg/broker.Broker.Unbind\n\t/tmp/app/pkg/broker/binding_operations.go:136\ngithub.com/pivotal-cf/brokerapi/handlers.APIHandler.Unbind\n\t/home/vcap/go/pkg/mod/github.com/pivotal-cf/brokerapi@v5.1.0+incompatible/handlers/unbind.go:56\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/mongodb/atlas-osb/pkg/broker.authMiddleware.func1.1\n\t/tmp/app/pkg/broker/auth.go:42\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/vcap/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2831\nnet/http.(*conn).serve\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:1919\nruntime.goexit\n\t/tmp/contents998050889/deps/0/go1.13.15/src/runtime/asm_amd64.s:1357\ncannot find plan for instance \"c20bdcea-6bf1-4804-ab06-bed618054e7d\"\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getPlan\n\t/tmp/app/pkg/broker/broker.go:160\ngithub.com/mongodb/atlas-osb/pkg/broker.(*Broker).getClient\n\t/tmp/app/pkg/broker/broker.go:179\ngithub.com/mongodb/atlas-osb/pkg/broker.Broker.Unbind\n\t/tmp/app/pkg/broker/binding_operations.go:136\ngithub.com/pivotal-cf/brokerapi/handlers.APIHandler.Unbind\n\t/home/vcap/go/pkg/mod/github.com/pivotal-cf/brokerapi@v5.1.0+incompatible/handlers/unbind.go:56\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/mongodb/atlas-osb/pkg/broker.authMiddleware.func1.1\n\t/tmp/app/pkg/broker/auth.go:42\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/vcap/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2831\nnet/http.(*conn).serve\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:1919\nruntime.goexit\n\t/tmp/contents998050889/deps/0/go1.13.15/src/runtime/asm_amd64.s:1357",
  "stacktrace": "main.(*LagerZapLogger).Error\n\t/tmp/app/logger.go:75\ngithub.com/pivotal-cf/brokerapi/handlers.APIHandler.Unbind\n\t/home/vcap/go/pkg/mod/github.com/pivotal-cf/brokerapi@v5.1.0+incompatible/handlers/unbind.go:63\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/mongodb/atlas-osb/pkg/broker.authMiddleware.func1.1\n\t/tmp/app/pkg/broker/auth.go:42\nnet/http.HandlerFunc.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2036\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/vcap/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:2831\nnet/http.(*conn).serve\n\t/tmp/contents998050889/deps/0/go1.13.15/src/net/http/server.go:1919"
}
2020 - 10 - 29 T11: 53: 43.657 + 08: 00[RTR / 0][OUT] mongodb - atlas - osb.apps.sea.preview.pcf.manulife.com - [2020 - 10 - 29 T03: 53: 36.112581905 Z]
"DELETE /v2/service_instances/c20bdcea-6bf1-4804-ab06-bed618054e7d/service_bindings/3ad3a3f3-6ca6-4972-9d2d-ff8cbda6aa93?accepts_incomplete=true&plan_id=aosb-cluster-plan-template-m30-small&service_id=aosb-cluster-service-template HTTP/1.1"
500 0 175 "-"
"HTTPClient/1.0 (2.8.3, ruby 2.5.5 (2019-03-15))"
"10.231.251.246:39862"
"10.231.56.101:61050"
x_forwarded_for: "10.231.56.41, 10.231.251.246"
x_forwarded_proto: "https"
vcap_request_id: "f54458d4-915c-4be9-4e86-0987465f1019"
response_time: 7.544368 gorouter_time: 0.000128 app_id: "8389ce70-29ef-4ca9-9ee9-fa2ae79a2985"
app_index: "0"
x_cf_routererror: "-"
x_b3_traceid: "32b054e6d805ad89"
x_b3_spanid: "32b054e6d805ad89"
x_b3_parentspanid: "-"
b3: "32b054e6d805ad89-32b054e6d805ad89"

[caspertsui]

Hi @jasonmimick , Did I provide all info you need? Thanks a lots.

[vasilevp]

Hi, This happened because the service instance was created with an older (pre-v0.5) version of the Broker. It's not possible to use versions v5.0 and newer with these old instances as there was no backwards compatibility during these early stages of Broker development. If you only have this one instance with such issue and are not hitting it with later versions of the Broker, please remove the broken instance by hand (delete from Atlas + cf purge-service-instance).

That being said, in the more recent releases we're able to provide better backwards compatibility since the whole design is getting a lot more stable.

Closing since this should not happen in latest versions.

[caspertsui]

Hi @jasonmimick , We updated to v0.7.0 but still see this issue, please kindly advice.

[tanjoht]

there's actually 4 of the broker-state realm apps in the SEA preview org.

We are waiting for @vasilevp to implement a fix to the broker and also provide a script to fix the duplicate realm apps.

We can manually fix the instancesif needed.