search

census-instrumentation / opencensus-go

A stats collection and distributed tracing framework
http://opencensus.io
Apache License 2.0
2.06k stars 327 forks source link

Trace for AppEngine Standard with Go 1.9 : can't deploy #985

Open Deleplace opened 5 years ago

Deleplace commented 5 years ago

Describe the bug gcloud app deploy fails when I try to use the StackDriver exporter and Trace.

To Reproduce Steps to reproduce the behavior:

  1. Go to the Quickstart for Go App Engine Standard Environment, Go 1.9. Download the hello app.
  2. Modify hello.go to use Stackdriver Exporter and Trace, as in this gist
  3. Make sure you have up-to-date dependencies locally 
    go get -u contrib.go.opencensus.io/exporter/stackdriver
go get -u go.opencensus.io/trace
go get -u github.com/jmespath/go-jmespath
  4. gcloud app deploy Result:

descriptor: [/usr/local/google/home/deleplace/Documents/2018/11/spans-OpenCensus/go1.9/app.yaml] source: [/usr/local/google/home/deleplace/Documents/2018/11/spans-OpenCensus/go1.9] target project: [spans-go19] target service: [default] target version: [20181126t112810] target url: [https://spans-go19.appspot.com]

Do you want to continue (Y/n)?

Beginning deployment of service [default]... ╔════════════════════════════════════════════════════════════╗ ╠═ Uploading 761 files to Google Cloud Storage ═╣ ╚════════════════════════════════════════════════════════════╝ File upload done. Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Deployment contains files that cannot be compiled: Compile failed: /work_dir/github.com/aws/aws-sdk-go/aws/awsutil/path_value.go:9:2: can't find import: "github.com/jmespath/go-jmespath" 2018/11/26 02:28:58 go-app-builder: build timing: 38×compile (32.805s total), 0×link (0s total) 2018/11/26 02:28:58 go-app-builder: failed running compile: exit status 2



**Expected behavior**
Deployment succeeds.

**Additional context**
I do have the dependency `go-jmespath` in my local gopath, the compilation failure seems to happen server-side.

IIUC Go 1.9 should be fine as [the doc says](https://github.com/census-instrumentation/opencensus-go#prerequisites) "OpenCensus Go libraries require Go 1.8 or later.".

I understand that many services in GAE Standard are already automatically traced, but currently not Cloud SQL, and I'd like to be able to create custom spans.

[This doc says](https://opencensus.io/exporters/supported-exporters/go/stackdriver/#1) "Stackdriver Trace ... can capture traces from all of your VMs, containers, or Google App Engine projects. [...] OpenCensus Go has support for this exporter" so I suppose I should be able to use the exporter in my code.
rakyll commented 5 years ago

Are you vendoring the dependencies? Not up to date on how AppEngine handles external dependencies but it used to require the developer to vendor them.

/cc @rghetia @tbpalsulich

Deleplace commented 5 years ago

Hi Jaana, the above output was produced without any vendoring.

Thanks for the advice. I tried vendoring (github.com/aws/aws-sdk-go, etc.) and 2 deployment commands. gcloud app deploy didn't work (see below). Somehow goapp deploy did succeed, but I then the deployed app fails.

$ gcloud app deploy
Services to deploy:

descriptor:      [/........../app.yaml]
source:          [/...............]
target project:  [spans-go19]
target service:  [default]
target version:  [20181204t135615]
target url:      [https://spans-go19.appspot.com]

Do you want to continue (Y/n)?  

Beginning deployment of service [default]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 0 files to Google Cloud Storage                ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [default]...failed.                                                                                                                                         
ERROR: (gcloud.app.deploy) Error Response: [9] Deployment contains files that cannot be compiled: Compile failed:
go-app-builder: Failed parsing input: package "vendor/github.com/aws/aws-sdk-go/service/s3control" cannot import internal package "github.com/aws/aws-sdk-go/internal/s3err"
$ goapp deploy
Application: spans-go19; version: 1-0
Host: appengine.google.com
...
...
Completed update of app: spans-go19, version: 1-0

Then when a request comes, the server logs say:

A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. (Error code 204)

If I move the call to stackdriver.NewExporter from main()/init() to the handler itself, then I get a more explicit error:

panic: not an App Engine context

goroutine 18 [running]:
panic(0x11d8de0, 0xc008433a40)
    go/src/runtime/panic.go:491 +0x283
google.golang.org/appengine/internal.fullyQualifiedAppID(0x17f99e0, 0xc00874c840, 0x1843a80, 0x0)
    google.golang.org/appengine/internal/identity_classic.go:58 +0x95
google.golang.org/appengine/internal.FullyQualifiedAppID(0x17f99e0, 0xc00874c840, 0xc0085de180, 0x17e88a0)
    google.golang.org/appengine/internal/api_common.go:77 +0x98
google.golang.org/appengine/internal.AppID(0x17f99e0, 0xc00874c840, 0xc0085de180, 0x1)
    google.golang.org/appengine/internal/identity.go:27 +0x35
google.golang.org/appengine.AppID(0x17f99e0, 0xc00874c840, 0xc0085b9a01, 0x33)
    google.golang.org/appengine/identity.go:20 +0x35
golang.org/x/oauth2/google.FindDefaultCredentials(0x17f99e0, 0xc00874c840, 0xc008750900, 0x4, 0x4, 0x8, 0x12201a0, 0x1)
    golang.org/x/oauth2/google/default.go:101 +0x2a6
google.golang.org/api/internal.Creds(0x17f99e0, 0xc00874c840, 0xc00851a480, 0xab06a7, 0xc0087635a0, 0x10)
    google.golang.org/api/internal/creds.go:44 +0x11d
google.golang.org/api/transport/grpc.dial(0x17f99e0, 0xc00874c840, 0xab1c00, 0xc0087508c0, 0x3, 0x4, 0xc008431360, 0xadfd54, 0xc0087508c0)
    google.golang.org/api/transport/grpc/dial.go:72 +0x5b4
google.golang.org/api/transport/grpc.Dial(0x17f99e0, 0xc00874c840, 0xc0087508c0, 0x3, 0x4, 0xc008750880, 0xc0087508c0, 0x1d)
    google.golang.org/api/transport/grpc/dial.go:39 +0x58
google.golang.org/api/transport.DialGRPC(0x17f99e0, 0xc00874c840, 0xc0087508c0, 0x3, 0x4, 0x1, 0x1, 0x1)
    google.golang.org/api/transport/dial.go:41 +0x53
cloud.google.com/go/monitoring/apiv3.NewMetricClient(0x17f99e0, 0xc00874c840, 0xc008763580, 0x1, 0x1, 0x0, 0x0, 0x0)
    cloud.google.com/go/monitoring/apiv3/metric_client.go:105 +0xfe
contrib.go.opencensus.io/exporter/stackdriver.newStatsExporter(0x133e64d, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    contrib.go.opencensus.io/exporter/stackdriver/stats.go:83 +0x1dd
contrib.go.opencensus.io/exporter/stackdriver.NewExporter(0x133e64d, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    contrib.go.opencensus.io/exporter/stackdriver/stackdriver.go:257 +0x1f0
main.initSD(0x17f9a20, 0xc00872ff80)
    hello.go:42 +0x71
main.handleB(0x17ec920, 0xc00879c300, 0xc008748b00)
    hello.go:74 +0x47
net/http.HandlerFunc.ServeHTTP(0x136d168, 0x17ec920, 0xc00879c300, 0xc008748b00)
    go/src/net/http/server.go:1918 +0x44
net/http.(*ServeMux).ServeHTTP(0x185d600, 0x17ec920, 0xc00879c300, 0xc008748b00)
    go/src/net/http/server.go:2254 +0x130
appengine_internal.executeRequestSafely(0xc00879c300, 0xc008748b00)
    go/src/appengine_internal/api_prod.go:296 +0x77
appengine_internal.(*server).HandleRequest(0x187d2e0, 0xc00871cd80, 0xc008794180, 0xc00854cf00, 0x0, 0x0)
    go/src/appengine_internal/api_prod.go:232 +0x9e9
reflect.Value.call(0x11d6da0, 0x187d2e0, 0x213, 0x13389ad, 0x4, 0xc008428f78, 0x3, 0x3, 0x132a5a0, 0x128df20, ...)
    go/src/reflect/value.go:434 +0x905
reflect.Value.Call(0x11d6da0, 0x187d2e0, 0x213, 0xc008428f78, 0x3, 0x3, 0x0, 0x213, 0x0)

The "not an App Engine context" is not always obvious to work around in main()/init(), as context.Background() doesn't seem to fill in, in the go1 runtimes. When I try to use the request context:

c := appengine.NewContext(r)
sd, err = stackdriver.NewExporter(stackdriver.Options{
        ProjectID:    "spans-go19",
        MetricPrefix: "demo-prefix",
        Context:      c,
    })

Then the requests fail, but without giving much details:

A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. (Error code 204)

I'm out of luck for now...

mblasi commented 5 years ago

Hello, I have same issue, with go 1.12. Any suggestion?

Regards, Matías.

Deleplace commented 5 years ago

Hi Matias, in GAE using Go 1.9, 1.11 or 1.12 are quite different, not because of the language but rather because of the runtime infrastructure:

Afaiu with Go 1.12 you should not be able to call appengine.NewContext at all, or use any lib that requires an appengine context.

Maybe this issue + what unstuck me might help: https://github.com/census-instrumentation/opencensus-go/issues/986

mblasi commented 5 years ago

Thank you @Deleplace !

I solved the issue by changing runtime from go to go112 in project app.yaml. I wasn't aware of go version migration guide, though in my case the imports were not an issue.

Thank you for your support!