Deployment of v0.2.0 on local microk8s : emailservice looking for GCP project ID

[didier-durand]

Hi there,

After successfully deploying on GKE, now trying to deploy v0.2.0 on a laptop microk8s instance (independent of GKE / GCP)

All services / deployments come up properly except emailservice going to CrashLoopBackOff.

Its logs say:

microk8s kubectl logs emailservice-8848674-kcrm7 

{"timestamp": 1607061532.1000514, "severity": "INFO", "name": "emailservice-server", "message": "starting the email service in dummy mode."}
{"timestamp": 1607061532.100424, "severity": "INFO", "name": "emailservice-server", "message": "Profiler disabled."}
{"timestamp": 1607061532.1006563, "severity": "INFO", "name": "emailservice-server", "message": "Tracing enabled."}
WARNING:google.auth._default:No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable
Traceback (most recent call last):
  File "email_server.py", line 191, in <module>
    transport=AsyncTransport)
  File "/usr/local/lib/python3.7/site-packages/opencensus/trace/exporters/stackdriver_exporter.py", line 199, in __init__
    client = Client(project=project_id)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/trace/client.py", line 43, in __init__
    super(Client, self).__init__(project=project, credentials=credentials)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/client.py", line 223, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/client.py", line 178, in __init__
    "Project was not passed and could not be "
OSError: Project was not passed and could not be determined from the environment.

What should I do? Create a dummy GOOGLE_CLOUD_PROJECT env var with no value in the yaml for emailservice ? Other workaround?

Thanks!

Didier

[didier-durand]

Indeed, I have similar issue with recommendationservice (I didn't notice it before...)

microk8s kubectl logs recommendationservice-764dc66688-2bf4v

{"timestamp": 1607062157.806904, "severity": "INFO", "name": "recommendationservice-server", "message": "initializing recommendationservice"}
{"timestamp": 1607062157.807212, "severity": "INFO", "name": "recommendationservice-server", "message": "Profiler enabled."}
WARNING:google.auth._default:No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable
{"timestamp": 1607062157.907493, "severity": "INFO", "name": "recommendationservice-server", "message": "Unable to start Stackdriver Profiler Python agent. Unable to determine the project ID from the environment. project ID mush be provided if running outside of GCP."}
{"timestamp": 1607062157.907878, "severity": "INFO", "name": "recommendationservice-server", "message": "Sleeping 10 seconds to retry Stackdriver Profiler agent initialization"}
WARNING:google.auth._default:No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable
{"timestamp": 1607062158.919691, "severity": "INFO", "name": "recommendationservice-server", "message": "Unable to start Stackdriver Profiler Python agent. Unable to determine the project ID from the environment. project ID mush be provided if running outside of GCP."}
{"timestamp": 1607062158.919864, "severity": "INFO", "name": "recommendationservice-server", "message": "Sleeping 20 seconds to retry Stackdriver Profiler agent initialization"}
WARNING:google.auth._default:No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable
{"timestamp": 1607062159.930853, "severity": "INFO", "name": "recommendationservice-server", "message": "Unable to start Stackdriver Profiler Python agent. Unable to determine the project ID from the environment. project ID mush be provided if running outside of GCP."}
{"timestamp": 1607062159.93103, "severity": "INFO", "name": "recommendationservice-server", "message": "Sleeping 30 seconds to retry Stackdriver Profiler agent initialization"}
{"timestamp": 1607062160.932162, "severity": "INFO", "name": "recommendationservice-server", "message": "Tracing enabled."}
WARNING:google.auth._default:No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable
Traceback (most recent call last):
  File "/recommendationservice/recommendation_server.py", line 110, in <module>
    transport=AsyncTransport)
  File "/usr/local/lib/python2.7/site-packages/opencensus/trace/exporters/stackdriver_exporter.py", line 199, in __init__
    client = Client(project=project_id)
  File "/usr/local/lib/python2.7/site-packages/google/cloud/trace/client.py", line 43, in __init__
    super(Client, self).__init__(project=project, credentials=credentials)
  File "/usr/local/lib/python2.7/site-packages/google/cloud/client.py", line 223, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/usr/local/lib/python2.7/site-packages/google/cloud/client.py", line 178, in __init__
    "Project was not passed and could not be "
EnvironmentError: Project was not passed and could not be determined from the environment.

[ahmetb]

Agreed, ideally services should print warnings but should not crash if not on GCP. There are env vars to disable tracing and monitoring, that can help, but should not be required.

[didier-durand]

@ahmetb : Thanks for the quick feedback! Is it possible for me now to disable those env vars via manifest ? How ? Or should I wait for a fix? Didier

[ahmetb]

@didier-durand there are already commented out env vars in the manifests, but this issue probably still stay open to address the problem.

[mathieu-benoit]

@didier-durand, have you tried to set DISABLE_TRACING to 1 here: https://github.com/GoogleCloudPlatform/microservices-demo/blob/master/kubernetes-manifests/emailservice.yaml#L38 and here https://github.com/GoogleCloudPlatform/microservices-demo/blob/master/kubernetes-manifests/recommendationservice.yaml#L48?

[didier-durand]

@mathieu-benoit : thanks for those details! That's exactly what I was looking for.

I tried. Now, Online Boutique works fine with Microk8s on my laptop:

microk8s kubectl get pods

NAME                                     READY   STATUS    RESTARTS   AGE
emailservice-6dc6c5c78f-twdf8            1/1     Running   0          87s
recommendationservice-69b4b47f59-2hs75   1/1     Running   0          86s
checkoutservice-8bddb89db-sffjx          1/1     Running   0          86s
frontend-58594bf684-w9qc5                1/1     Running   0          86s
productcatalogservice-7d5f94d858-btkxc   1/1     Running   0          84s
paymentservice-77979f5c66-h4zwm          1/1     Running   0          85s
shippingservice-589dc45c5d-pmc99         1/1     Running   0          81s
currencyservice-7978fb77d8-d72kc         1/1     Running   0          82s
cartservice-65c987d449-ctxkt             0/1     Running   1          83s
loadgenerator-78dc8587f8-srls2           1/1     Running   1          82s
adservice-7659d48d84-cdw56               0/1     Running   0          80s
redis-cart-74594bd569-bvwzl              0/1     Running   0          81s

I am happy to prepare a small PR for the doc to detail it if you want.

[mathieu-benoit]

Hi @didier-durand, closing that one for now, and FYI I'm starting this PR https://github.com/GoogleCloudPlatform/microservices-demo/pull/460 to see if we could silently catch that exception in recommendationservice and emailservice.

[didier-durand]

@mathieu-benoit : Looks good to me! Thanks for the feedback. Didier