lagom / online-auction-scala

Other
200 stars 128 forks source link

Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items' #78

Open lejoow opened 6 years ago

lejoow commented 6 years ago

I am not 100% sure if this is the issue specifics to my local settings or not. But every time I try to deploy the item-svc or bidding-svc to my Google KBE, I am getting this exception.

Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'

screen shot 2018-05-09 at 7 23 54 am
TimMoore commented 6 years ago

I think I've found the problem. GCP enables Role-Based Access Control (aka RBAC) by default in new Kubernetes clusters. This requires configuring roles to allow access to resources within the Kubernetes API. Lightbend Orchestration (specifically, the Akka Cluster Bootstrap component) uses the Kubernetes API to automatically form a cluster. Without binding your pods' service account to a role that grants access to the required resources, these API calls fail.

To correct this, you'll need to follow the instructions on this page for configuring role-based access control: https://developer.lightbend.com/docs/akka-management/current/discovery.html#discovery-method-kubernetes-api

For example, you can supply these Kubernetes resources to kubectl apply:

---
#
# Create a role, `pod-reader`, that can list pods and
# bind the default service account in the `default` namespace
# to that role.
#

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
subjects:
# Note the `name` line below. The first default refers to the namespace. The second refers to the service account name.
# For instance, `name: system:serviceaccount:myns:default` would refer to the default service account in namespace `myns`
- kind: User
  name: system:serviceaccount:default:default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Once you've created this role and role binding, your Lagom services should start correctly. You might want to tailor this to your specific security needs.

@lejoow please let us know if that fixes the problem for you. Even if it does, we can leave this issue open until we update the documentation.

lejoow commented 6 years ago

@TimMoore fantastic. Let me try this fix today and come back to you on this post. Thanks a lot!

lejoow commented 6 years ago

hmm... I don't get that error anymore, but I am getting the different errors now:

2018-05-10T17:54:27.363Z [info] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=application-akka.actor.default-dispatcher-24, akkaTimestamp=17:54:27.363UTC, akkaSource=akka.tcp://application@10.16.1.48:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=application] - Exceeded stable margins without locating seed-nodes, however this node is NOT the lowest address out of the discovered IPs in this deployment, thus NOT joining self. Expecting node Some(ResolvedTarget(10.16.0.13,Some(10002))) (out of List(ResolvedTarget(10.16.0.13,Some(10002)), ResolvedTarget(10.16.1.48,Some(10002)))) to perform the self-join and initiate the cluster.

I am still investigating..

lejoow commented 6 years ago

It worked! I just had to re-deploy. Thanks! Btw, when I tried to create the role by applying that file you pasted, I ran into "pod-reader forbidden" error message, which I resolved by following the instructions here:

kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=$primary_account https://github.com/jcbsmpsn/gke-rbac-walkthrough/blob/master/create-roles.md

TimMoore commented 6 years ago

Let's leave this open until the documentation is updated. Thanks!

TimMoore commented 6 years ago

Added an issue for the Akka Management documentation to note that Google Cloud users will need to grant themselves the cluster-admin ClusterRole: https://github.com/akka/akka-management/issues/199

Canislupax commented 6 years ago

That solved my problem! Thanx a lot Tim! You`re the best!

idoshamun commented 6 years ago

@TimMoore I followed your suggestion but now I receive:

14:47:46.025 [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'ports'

For what it's worth, I am trying to run Lagom with Istio

idoshamun commented 6 years ago

Disabling Istio resolves this issue, I will try to look further and share results

normenmueller commented 6 years ago

@TimMoore @lejoow I have the same issue running on Minikube. Here are my deployment steps:

$ sbt docker:publishLocal

$ minikube delete
$ minikube start --memory 8192
$ eval $(minikube docker-env)          <<< Should this command come before docker:publishLocal?
$ minikube addons enable ingress
$ kubectl apply -f RBAC.yml            <<< The file posted by @TimMoore

$ sbt "deploy minikube"

$ kubectl cluster-info
$ kubectl get deployments
$ kubectl get pods
$ echo "http://$(minikube ip)"

All Pods have status Running.

Then I had a look at the minikube dashboard and all my services are marked with

Readiness probe failed: Get http://172.17.0.8:10002/platform-tooling/ready: dial tcp 172.17.0.8:10002: getsockopt: connection refused

Then I had a look in the logs at the minikube dashboard and get:

[error] akka.actor.ActorSystemImpl [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=16:32:22.971UTC, akkaSource=akka.actor.ActorSystemImpl(application), sourceActorSystem=application] - Failed to unmarshal Kubernetes API response status [400 Bad Request]; check RBAC settings
Bunyod commented 6 years ago

@TimMoore @lejoow @normenmueller I'm also facing with the same getsockopt: connection refused issue for Minikube. I follow this link.

Does anybody have an idea how to fix it?

OguzhanSaltik commented 6 years ago

We have the same error, due to Lightbend Orchestration documentation.

TimMoore commented 6 years ago

The getsockopt: connection refused message itself only indicates that the service is not running. Of course, there could be many reasons for that. You'll need to look at the logs for the pod to understand more. I'd like to keep this issue thread focused on the original issue reported, which has a solution described in a comment above: https://github.com/lagom/online-auction-scala/issues/78#issuecomment-387933707

If you're having another problem, the best place to start is by posting an issue on the Discourse forum: https://discuss.lightbend.com/

bahtiyarerden commented 6 years ago

We are trying to deploy our lagom application with kubernetes. But we could not fix this issue. We clone this repository (chirper) and deploy with minikube to be sure the problem does not about our configuration. It is the result:

screen shot 2018-06-21 at 15 29 29

And this is the log file:

2018-06-21T12:36:25.703Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:25.703UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:26.756Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:26.756UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:26.883Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:26.883UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:27.802Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:27.801UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:28.839Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:28.839UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:28.882Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-14, akkaTimestamp=12:36:28.882UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:29.863Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-14, akkaTimestamp=12:36:29.863UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:30.883Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-4, akkaTimestamp=12:36:30.882UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:30.936Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-4, akkaTimestamp=12:36:30.935UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:32.051Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:32.051UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:32.893Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:32.892UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:33.089Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:33.088UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:34.116Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:34.115UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:34.873Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:34.872UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:35.159Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:35.157UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:36.257Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:36.257UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:36.873Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:36.873UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:37.304Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:37.301UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:38.332Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:38.332UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:38.885Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-2, akkaTimestamp=12:36:38.885UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:39.378Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:39.376UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:40.420Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:40.420UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:40.882Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:40.882UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:41.468Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:41.468UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'

I think the problem is about configuration that given below. My microservices that do not have this configuration are working without any error. When i look the logs then i see the akka management port is 10002. The readiness prob also check this port and getting getsockopt: connection refused error.

The application.conf file configuration:

akka.actor.serialization-bindings {
  "akka.Done"                 = akka-misc
  "akka.actor.Address"        = akka-misc
  "akka.remote.UniqueAddress" = akka-misc
}
ale222 commented 6 years ago

Trying to deploy online-auction, getting:

akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=12:27:31.874UTC, akkaSource=akka.tcp://application@10.48.0.13:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=application] - Discovered [1] observation, which is less than the required [2], retrying (interval: 1.000 s)

any ideas?

zhangzhonglai commented 6 years ago

same error because of akka/akka-management#222 when using sbt-reactive-app >= 1.2.0 if someone encounter the error like me, simply back to use sbt-reactive-app 1.1.0 will temporary fix the error now.

ktoso commented 6 years ago

The related issue #222 was resolved, 0.15 should fix it then. Thanks for digging into this!

OguzhanSaltik commented 6 years ago

Great news @ktoso , when will be 0.15 version applied to the Lagom?

ktoso commented 6 years ago

I'll have to rely this to @TimMoore, not sure on release schedule of intermediate deps etc.

TimMoore commented 6 years ago

Lagom has no dependency on akka-management. That comes in by way of Lightbend Orchestration, so I think @mitkus is the right one to ask.

OguzhanSaltik commented 6 years ago

You are right, the related fix is part of the sbt-reactive-app plugin. We look forward to the version updates, thanks to everyone.