pubsub: invalid memory address when NewClient initialization

[acrosson]

I'm getting an invalid memory address when I launch my go app on App Engine. In the main function I attempt to initialize a pubsub client, but it throws an invalid memory address error.

I'm able to get pubsub working locally, I just seem to be running into issues when I deploy it. The error log from GAE is below.

Any suggestions? Thx.

b, _ := json.Marshal("gopher")
ctx := context.Background()
client, err := pubsub.NewClient(ctx, "app-id")
if err != nil {
    log.Fatalf("pubsub Client initialize failed %v", err)
}
topic := client.Topic("events")
ids, err := topic.Publish(ctx, &pubsub.Message{Data: b})
if err != nil {
    log.Println(err)
}

Error log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x40 pc=0x172128f]

goroutine 1 [running]:
panic(0x1cf8500, 0xc01040e070)
    go/src/runtime/panic.go:481 +0x3e6
google.golang.org/appengine/internal.fullyQualifiedAppID(0x2ab784af3db8, 0xc01040fcd8, 0x0, 0x0)
    google.golang.org/appengine/internal/identity_classic.go:27 +0x4f
google.golang.org/appengine/internal.FullyQualifiedAppID(0x2ab784af3db8, 0xc01040fcd8, 0x0, 0x0)
    google.golang.org/appengine/internal/api_common.go:74 +0xe2
google.golang.org/appengine/internal.AppID(0x2ab784af3db8, 0xc01040fcd8, 0x0, 0x0)
    google.golang.org/appengine/internal/identity.go:13 +0x37
google.golang.org/appengine.AppID(0x2ab784af3db8, 0xc01040fcd8, 0x0, 0x0)
    google.golang.org/appengine/identity.go:20 +0x37
golang.org/x/oauth2/google.FindDefaultCredentials(0x2ab784af3db8, 0xc01040fcd8, 0xc01071de00, 0x2, 0x2, 0xc01071de00, 0x0, 0x0)
    golang.org/x/oauth2/google/default.go:86 +0x5a8
golang.org/x/oauth2/google.DefaultTokenSource(0x2ab784af3db8, 0xc01040fcd8, 0xc01071de00, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0)
    golang.org/x/oauth2/google/default.go:43 +0x6b
google.golang.org/api/transport.DialGRPC(0x2ab784af3db8, 0xc01040fcd8, 0xc010734940, 0x3, 0x4, 0x1, 0x0, 0x0)
    google.golang.org/api/transport/dial.go:149 +0x379
cloud.google.com/go/pubsub/apiv1.NewPublisherClient(0x2ab784af3db8, 0xc01040fcd8, 0xc010919bb8, 0x1, 0x1, 0x1b3a620, 0x0, 0x0)
    cloud.google.com/go/pubsub/apiv1/publisher_client.go:118 +0x1cc
cloud.google.com/go/pubsub.newPubSubService(0x2ab784af3db8, 0xc01040fcd8, 0xc010919bb8, 0x1, 0x1, 0x0, 0x0, 0x0)
    cloud.google.com/go/pubsub/service.go:76 +0x76
cloud.google.com/go/pubsub.NewClient(0x2ab784af3db8, 0xc01040fcd8, 0x1e80f10, 0x7, 0x0, 0x0, 0x0, 0xc010919cd8, 0x0, 0x0)
    cloud.google.com/go/pubsub/pubsub.go:65 +0x48f
github.com/username/app-api.SendMsg(0x0)
    github.com/username/app-api/main.go:119 +0x142
github.com/username/app-api.Run()
    github.com/username/app-api/main.go:112 +0xa94
main30024.init.1()
    deploy.go:8 +0x14
main30024.init()
    deploy.go:9 +0x45
_import_tree1.init()
    _import_tree1/_go_main_tree1.go:45 +0x8a

[rakyll]

I can reproduce it also locally.

[rakyll]

You are not passing a valid AppEngine context but a context.Background() to NewClient. Try:

    ctx := appengine.NewContext(r)

    client, err := pubsub.NewClient(ctx, "app-id")
    if err != nil {
        log.Fatalf("pubsub Client initialize failed %v", err)
    }

We should return an error rather than panic'ing without any context about the issue.

[acrosson]

@rakyll

I've tried using the appengine.NewContext(r), and it works (sent message on app engine, received locally on worker) so long as I don't send the request through a goroutine. I'm using gin for the app router, and I've tried copying the gin context but it's still throwing an error.

Example call:

r.GET("/_ah/health", func(c *gin.Context) {
    cCp := c.Copy()
    SendMsg(cCp)
    c.JSON(200, gin.H{
        "message": "ok",
    })
})

func SendMsg(c *gin.Context) {
    logging.Log(c, "Sending Message ...")
    b, _ := json.Marshal("gopher")
    ctx := appengine.NewContext(c.Request)
    client, err := pubsub.NewClient(ctx, "app-id")
    if err != nil {
        logging.Error(c, fmt.Sprintf("%v", err))
    }
    topic := client.Topic("events")
    ids, err := topic.Publish(ctx, &pubsub.Message{Data: b})
    if err != nil {
        logging.Error(c, fmt.Sprintf("%v", err))
    }
    logging.Log(c, fmt.Sprintf("%v", ids))
}

So I guess I have two questions:

1. Am I only able to use appengine context when creating a client on App Engine? If so what context do I use for a background worker, that listens for new publications.
2. How can I send the context.Request through a goroutine so that I don't block the request response?

[acrosson]

RE question 1:

The context I want to use is appengine.BackgroundContext() but this seems to be only exposed on "flex" environments. worker.go:30: undefined: "google.golang.org/appengine".BackgroundContext

BackgroundContext Function: https://github.com/golang/appengine/blob/b2b053be966fce2c050a81415eb3c36238908778/appengine_vm.go

I'm going to try to update my worker app.yaml file for the flex environment and see what happens.

[rakyll]

/cc @jba

[atishpatel]

I had been using context.Background() for gitkit, and it was working fine for the last 6 months. But, I deployed today, and this issue broke my app even though I didn't update any of my deps. :\

[zombiezen]

This really more sounds like AE is not giving clear error messages for using context.Background. I'll try to fix that up. However, there isn't anything PubSub specific here, other than the observation that App Engine Standard doesn't let you have long-standing TCP connections open, and it is more likely one would try to have a long-standing connection with PubSub.

[drewwells]

The basic book example uses context.Background(). How are we supposed to subscribe to topics outside of a web request?

https://cloud.google.com/go/getting-started/using-pub-sub

[jba]

@drewwells Has your question been answered?

[drewwells]

Looks like better errors are being presented. It seems like the getting started should address how to use Pub/Sub (especially Sub) in the context of AppEngine. The context.Context is tied to a web request, so it makes Sub impractical for that application.

[broady]

I agree, docs should be improved.

Right now, for Pub/Sub, I suggest using the google.golang.org/api/pubsub/v1 package for publishing, and configure the subscription to push to the GAE app (instead of pulling messages).

Some sample code for pub/sub on GAE standard: https://gist.github.com/broady/c79a65cb49d7b6a56448b3345a23b3d6

[SameeraLakshithaPerera]

can i use spring mvc with pubsub ?