search

googleapis / google-cloud-go

Google Cloud Client Libraries for Go.
https://cloud.google.com/go/docs/reference
Apache License 2.0
3.72k stars 1.27k forks source link

datastore: query iterating over 36m objects panics with stack overflow #9528

Open gannett-ggreer opened 6 months ago

gannett-ggreer commented 6 months ago

Client

Google Firestore in Datastore mode

Environment

Compile: macOS Ventura Run: Linux (GKE pod)

Go Environment

$ go version go version go1.22.0 darwin/amd64

$ go env GO111MODULE='on' GOARCH='amd64' GOBIN='' GOCACHE='/Users/ggreer/Library/Caches/go-build' GOENV='/Users/ggreer/Library/Application Support/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='darwin' GOINSECURE='' GOMODCACHE='/Users/ggreer/p/go/pkg/mod' GONOPROXY='github.com/GannettDigital' GONOSUMDB='github.com/GannettDigital' GOOS='darwin' GOPATH='/Users/ggreer/p/go' GOPRIVATE='github.com/GannettDigital' GOPROXY='https://proxy.golang.org,direct' GOROOT='/usr/local/Cellar/go/1.22.0/libexec' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/Cellar/go/1.22.0/libexec/pkg/tool/darwin_amd64' GOVCS='' GOVERSION='go1.22.0' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='cc' CXX='c++' CGO_ENABLED='1' GOMOD='/Users/ggreer/p/go/src/github.com/GannettDigital/sports2-aggregator/go.mod' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/Users/ggreer/.tmp/go-build1573180754=/tmp/go-build -gno-record-gcc-switches -fno-common'

Code

There's a lot to the program but the "meat" of the panic is basically this code:

                for {
                        var dsItem datastoreMetadata
                        // Note though retrieved individually this is pulled from Datastore in batches by the datastore client
                        key, err := f.iterator.Next(&dsItem)
                        if err != nil && !strings.Contains(err.Error(), "datastore: cannot load field") {
                                if err == iterator.Done {
                                        close(f.idChan)
                                        level.Info(f.logger).Log("message", "Stopping (done).")
                                        return
                                }
                                level.Error(f.logger).Log("error", err)
                        }

                        // Record the cursor in case resuming is required
                        cursorString, err := f.iterator.Cursor()
                        if err != nil {
                                level.Error(f.logger).Log("cursorError", err)
                        } else {
                                f.stats.setCursor(cursorString.String())
                        }

                        if f.keysOnly && key.Name != "" {
                                // when using a keysOnly query, the key.Name is equivalent to the id in string format
                                dsItem = datastoreMetadata{
                                        Id: key.Name,
                                }
                        }
                        // TODO: Fields will need to be updated for types beyond Tags.
                        level.Debug(f.logger).Log("Key", key, "ID", dsItem.Id)
                        select {
                        case <-stopChan:
                                close(f.idChan)
                                level.Info(f.logger).Log("message", "Stopping (channel).")
                                return
                        case f.idChan <- dsItem.Id:
                        }
                }

Expected behavior

Datastore client enumerates all 36 million objects.

Actual behavior

Datastore client slowly uses more and more memory until it panics with runtime: goroutine stack exceeds 1000000000-byte limit.

Additional context

go.mod Google versions:

    cloud.google.com/go/datastore v1.15.0
    cloud.google.com/go/pubsub v1.36.2
    github.com/google/uuid v1.6.0
    github.com/googleapis/google-cloud-go-testing v0.0.0-20210719221736-1c9a4c676720
    google.golang.org/api v0.168.0
    google.golang.org/grpc v1.62.1
    cloud.google.com/go v0.112.0 // indirect
    cloud.google.com/go/compute v1.24.0 // indirect
    cloud.google.com/go/compute/metadata v0.2.3 // indirect
    cloud.google.com/go/iam v1.1.6 // indirect
    cloud.google.com/go/storage v1.37.0 // indirect
    github.com/google/go-cmp v0.6.0 // indirect
    github.com/google/martian/v3 v3.3.2 // indirect
    github.com/google/s2a-go v0.1.7 // indirect
    github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
    github.com/googleapis/gax-go/v2 v2.12.2 // indirect
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
    google.golang.org/appengine v1.6.8 // indirect
    google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
    google.golang.org/genproto/googleapis/api v0.0.0-20240221002015-b0ce06bbee7c // indirect
    google.golang.org/genproto/googleapis/rpc v0.0.0-20240304161311-37d4d3c04a78 // indirect
    google.golang.org/protobuf v1.32.0 // indirect

Stack trace for the crashing goroutine is:

runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc0a501a388 stack=[0xc0a501a000, 0xc0c501a000]
fatal error: stack overflow

runtime stack:
runtime.throw({0x192f01b?, 0x1?})
        /usr/local/Cellar/go/1.22.0/libexec/src/runtime/panic.go:1023 +0x5c fp=0xc000b2be18 sp=0xc000b2bde8 pc=0x43c43c
runtime.newstack()
        /usr/local/Cellar/go/1.22.0/libexec/src/runtime/stack.go:1103 +0x5bd fp=0xc000b2bfc8 sp=0xc000b2be18 pc=0x4579fd
runtime.morestack()
        /usr/local/Cellar/go/1.22.0/libexec/src/runtime/asm_amd64.s:616 +0x7a fp=0xc000b2bfd0 sp=0xc000b2bfc8 pc=0x47063a

goroutine 124 gp=0xc000703500 m=16 mp=0xc000800808 [running]:
context.(*valueCtx).Done(0xc000845a40?)
        <autogenerated>:1 +0x34 fp=0xc0a501a398 sp=0xc0a501a390 pc=0x50d0f4
context.(*valueCtx).Done(0xc0a3b946c0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a3b0 sp=0xc0a501a398 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a3c8 sp=0xc0a501a3b0 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a3e0 sp=0xc0a501a3c8 pc=0x50d0e4
context.(*valueCtx).Done(0x12?)
        <autogenerated>:1 +0x24 fp=0xc0a501a3f8 sp=0xc0a501a3e0 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a410 sp=0xc0a501a3f8 pc=0x50d0e4
context.(*valueCtx).Done(0xc08c7aeb40?)
        <autogenerated>:1 +0x24 fp=0xc0a501a428 sp=0xc0a501a410 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a440 sp=0xc0a501a428 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a458 sp=0xc0a501a440 pc=0x50d0e4
context.(*valueCtx).Done(0xc0a501c340?)
        <autogenerated>:1 +0x24 fp=0xc0a501a470 sp=0xc0a501a458 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a488 sp=0xc0a501a470 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a4a0 sp=0xc0a501a488 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a4b8 sp=0xc0a501a4a0 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a4d0 sp=0xc0a501a4b8 pc=0x50d0e4
context.(*valueCtx).Done(0x1dbb160?)
        <autogenerated>:1 +0x24 fp=0xc0a501a4e8 sp=0xc0a501a4d0 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a500 sp=0xc0a501a4e8 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a518 sp=0xc0a501a500 pc=0x50d0e4
context.(*valueCtx).Done(0xc0a3b94750?)
        <autogenerated>:1 +0x24 fp=0xc0a501a530 sp=0xc0a501a518 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a548 sp=0xc0a501a530 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a560 sp=0xc0a501a548 pc=0x50d0e4
context.(*valueCtx).Done(0x12?)
        <autogenerated>:1 +0x24 fp=0xc0a501a578 sp=0xc0a501a560 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a590 sp=0xc0a501a578 pc=0x50d0e4
context.(*valueCtx).Done(0xc08c7aee10?)
        <autogenerated>:1 +0x24 fp=0xc0a501a5a8 sp=0xc0a501a590 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a5c0 sp=0xc0a501a5a8 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a5d8 sp=0xc0a501a5c0 pc=0x50d0e4
context.(*valueCtx).Done(0xc0a501c400?)
        <autogenerated>:1 +0x24 fp=0xc0a501a5f0 sp=0xc0a501a5d8 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a608 sp=0xc0a501a5f0 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a620 sp=0xc0a501a608 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a638 sp=0xc0a501a620 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a650 sp=0xc0a501a638 pc=0x50d0e4
context.(*valueCtx).Done(0x1dbb160?)
        <autogenerated>:1 +0x24 fp=0xc0a501a668 sp=0xc0a501a650 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a680 sp=0xc0a501a668 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a698 sp=0xc0a501a680 pc=0x50d0e4
context.(*valueCtx).Done(0xc0a3b947e0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a6b0 sp=0xc0a501a698 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a6c8 sp=0xc0a501a6b0 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a6e0 sp=0xc0a501a6c8 pc=0x50d0e4
context.(*valueCtx).Done(0x12?)
        <autogenerated>:1 +0x24 fp=0xc0a501a6f8 sp=0xc0a501a6e0 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a710 sp=0xc0a501a6f8 pc=0x50d0e4
context.(*valueCtx).Done(0xc08c7af0e0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a728 sp=0xc0a501a710 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a740 sp=0xc0a501a728 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a758 sp=0xc0a501a740 pc=0x50d0e4
context.(*valueCtx).Done(0xc0a501c4c0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a770 sp=0xc0a501a758 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a788 sp=0xc0a501a770 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a7a0 sp=0xc0a501a788 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a7b8 sp=0xc0a501a7a0 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a7d0 sp=0xc0a501a7b8 pc=0x50d0e4
context.(*valueCtx).Done(0x1dbb160?)
        <autogenerated>:1 +0x24 fp=0xc0a501a7e8 sp=0xc0a501a7d0 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a800 sp=0xc0a501a7e8 pc=0x50d0e4
context.(*valueCtx).Done(0x0?)
        <autogenerated>:1 +0x24 fp=0xc0a501a818 sp=0xc0a501a800 pc=0x50d0e4
context.(*valueCtx).Done(0xc0a3b94870?)
        <autogenerated>:1 +0x24 fp=0xc0a501a830 sp=0xc0a501a818 pc=0x50d0e4
...22369403 frames elided...
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c50193d0 sp=0xc0c50193b8 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c50193e8 sp=0xc0c50193d0 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019400 sp=0xc0c50193e8 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019418 sp=0xc0c5019400 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019430 sp=0xc0c5019418 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019448 sp=0xc0c5019430 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019460 sp=0xc0c5019448 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019478 sp=0xc0c5019460 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019490 sp=0xc0c5019478 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c50194a8 sp=0xc0c5019490 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c50194c0 sp=0xc0c50194a8 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c50194d8 sp=0xc0c50194c0 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c50194f0 sp=0xc0c50194d8 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019508 sp=0xc0c50194f0 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019520 sp=0xc0c5019508 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019538 sp=0xc0c5019520 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019550 sp=0xc0c5019538 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019568 sp=0xc0c5019550 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019580 sp=0xc0c5019568 pc=0x50d0e4
context.(*valueCtx).Done(0x10?)
        <autogenerated>:1 +0x24 fp=0xc0c5019598 sp=0xc0c5019580 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c50195b0 sp=0xc0c5019598 pc=0x50d0e4
context.(*valueCtx).Done(0x50?)
        <autogenerated>:1 +0x24 fp=0xc0c50195c8 sp=0xc0c50195b0 pc=0x50d0e4
context.(*valueCtx).Done(0x41a1d8?)
        <autogenerated>:1 +0x24 fp=0xc0c50195e0 sp=0xc0c50195c8 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c50195f8 sp=0xc0c50195e0 pc=0x50d0e4
context.(*valueCtx).Done(0x58?)
        <autogenerated>:1 +0x24 fp=0xc0c5019610 sp=0xc0c50195f8 pc=0x50d0e4
context.(*valueCtx).Done(0x7fe124363508?)
        <autogenerated>:1 +0x24 fp=0xc0c5019628 sp=0xc0c5019610 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019640 sp=0xc0c5019628 pc=0x50d0e4
context.(*valueCtx).Done(0x50d0e4?)
        <autogenerated>:1 +0x24 fp=0xc0c5019658 sp=0xc0c5019640 pc=0x50d0e4
context.(*valueCtx).Done(0x7fe11fa5fa78?)
        <autogenerated>:1 +0x24 fp=0xc0c5019670 sp=0xc0c5019658 pc=0x50d0e4
context.(*cancelCtx).propagateCancel(0xc131b4fef0, {0x1dcc170, 0xc136af6c90}, {0x1dc2ff0, 0xc131b4fef0})
        /usr/local/Cellar/go/1.22.0/libexec/src/context/context.go:465 +0x75 fp=0xc0c50196f0 sp=0xc0c5019670 pc=0x50a795
context.withCancel(...)
        /usr/local/Cellar/go/1.22.0/libexec/src/context/context.go:273
context.WithCancel({0x1dcc170, 0xc136af6c90})
        /usr/local/Cellar/go/1.22.0/libexec/src/context/context.go:236 +0x59 fp=0xc0c5019730 sp=0xc0c50196f0 pc=0x509e79
google.golang.org/grpc.newClientStreamWithParams({0x1dcc170, 0xc136af6c90}, 0x2b34320, 0xc0005bd508, {0x195e7b4, 0x27}, {0x0, 0x0, 0x0, 0x0, ...}, ...)
        /Users/ggreer/p/go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:238 +0x154 fp=0xc0c5019898 sp=0xc0c5019730 pc=0x9004b4
google.golang.org/grpc.newClientStream.func2({0x1dcc170?, 0xc136af6c90?}, 0xc136af6c90?)
        /Users/ggreer/p/go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:192 +0x87 fp=0xc0c5019928 sp=0xc0c5019898 pc=0x9002e7
google.golang.org/grpc.newClientStream({0x1dcc170, 0xc136af6c90}, 0x2b34320, 0xc0005bd508, {0x195e7b4, 0x27}, {0x0, 0x0, 0x0})
        /Users/ggreer/p/go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:220 +0x46f fp=0xc0c50199f8 sp=0xc0c5019928 pc=0x8fff0f
google.golang.org/grpc.invoke({0x1dcc170?, 0xc136af6c90?}, {0x195e7b4?, 0x411adb?}, {0x18218e0, 0xc00042f4a0}, {0x17a9680, 0xc131ef83c0}, 0xc0c5019b10?, {0x0, ...})
        /Users/ggreer/p/go/pkg/mod/google.golang.org/grpc@v1.46.0/call.go:66 +0x77 fp=0xc0c5019a60 sp=0xc0c50199f8 pc=0x8e5cf7
google.golang.org/grpc.(*ClientConn).Invoke(0xc0005bd508, {0x1dcc170?, 0xc136af6c90?}, {0x195e7b4?, 0xc0c5019b58?}, {0x18218e0?, 0xc00042f4a0?}, {0x17a9680?, 0xc131ef83c0?}, {0x0, ...})
        /Users/ggreer/p/go/pkg/mod/google.golang.org/grpc@v1.46.0/call.go:37 +0x23f fp=0xc0c5019af8 sp=0xc0c5019a60 pc=0x8e5bff
google.golang.org/api/transport/grpc.(*roundRobinConnPool).Invoke(0x30?, {0x1dcc170?, 0xc136af6c90?}, {0x195e7b4?, 0xc136af6c90?}, {0x18218e0?, 0xc00042f4a0?}, {0x17a9680?, 0xc131ef83c0?}, {0x0, ...})
        /Users/ggreer/p/go/pkg/mod/google.golang.org/api@v0.79.0/transport/grpc/pool.go:59 +0x9d fp=0xc0c5019b68 sp=0xc0c5019af8 pc=0x98241d
google.golang.org/genproto/googleapis/datastore/v1.(*datastoreClient).RunQuery(0xc00004a940, {0x1dcc170, 0xc136af6c90}, 0xc00042f4a0, {0x0, 0x0, 0x0})
        /Users/ggreer/p/go/pkg/mod/google.golang.org/genproto@v0.0.0-20220505152158-f39f71e6c8f3/googleapis/datastore/v1/datastore.pb.go:2403 +0xc8 fp=0xc0c5019be0 sp=0xc0c5019b68 pc=0x925c28
cloud.google.com/go/datastore.(*datastoreClient).RunQuery.func2({0x1dcc170?, 0xc136af6c90?})
        /Users/ggreer/p/go/pkg/mod/cloud.google.com/go/datastore@v1.6.0/client.go:69 +0x5a fp=0xc0c5019c38 sp=0xc0c5019be0 pc=0x9b753a
cloud.google.com/go/datastore.(*datastoreClient).invoke.func1()
        /Users/ggreer/p/go/pkg/mod/cloud.google.com/go/datastore@v1.6.0/client.go:122 +0x1f fp=0xc0c5019c68 sp=0xc0c5019c38 pc=0x9b843f
cloud.google.com/go/internal.retry({0x1dcc170, 0xc136af6c90}, {0x5f5e100, 0x0, 0x0, 0x0}, 0xc0c5019d10, 0x19d6540)
        /Users/ggreer/p/go/pkg/mod/cloud.google.com/go@v0.101.1/internal/retry.go:40 +0x74 fp=0xc0c5019cd0 sp=0xc0c5019c68 pc=0x911694
cloud.google.com/go/internal.Retry(...)
        /Users/ggreer/p/go/pkg/mod/cloud.google.com/go@v0.101.1/internal/retry.go:33
cloud.google.com/go/datastore.(*datastoreClient).invoke(0x1dcc170?, {0x1dcc170, 0xc136af6c30}, 0xc0c5019dc0)
        /Users/ggreer/p/go/pkg/mod/cloud.google.com/go/datastore@v1.6.0/client.go:121 +0xd2 fp=0xc0c5019d60 sp=0xc0c5019cd0 pc=0x9b83d2
cloud.google.com/go/datastore.(*datastoreClient).RunQuery(0xc000579e30, {0x1dcc170?, 0xc1382e29f0?}, 0xc00042f4a0, {0x0, 0x0, 0x0})
        /Users/ggreer/p/go/pkg/mod/cloud.google.com/go/datastore@v1.6.0/client.go:68 +0x165 fp=0xc0c5019e18 sp=0xc0c5019d60 pc=0x9b7425
cloud.google.com/go/datastore.(*Iterator).nextBatch(0xc00039d290)
        /Users/ggreer/p/go/pkg/mod/cloud.google.com/go/datastore@v1.6.0/query.go:692 +0x14d fp=0xc0c5019e68 sp=0xc0c5019e18 pc=0x9c860d
cloud.google.com/go/datastore.(*Iterator).next(0xc00039d290)
        /Users/ggreer/p/go/pkg/mod/cloud.google.com/go/datastore@v1.6.0/query.go:647 +0x35 fp=0xc0c5019e90 sp=0xc0c5019e68 pc=0x9c82d5
cloud.google.com/go/datastore.(*Iterator).Next(0xc00039d290, {0x157b680, 0xc131290180})
        /Users/ggreer/p/go/pkg/mod/cloud.google.com/go/datastore@v1.6.0/query.go:634 +0x25 fp=0xc0c5019ec0 sp=0xc0c5019e90 pc=0x9c8205
github.com/GannettDigital/content-pipeline-reprocessor/reprocessor.(*allDatastoreFeed).stream.func1()
        /Users/ggreer/p/go/src/github.com/GannettDigital/content-pipeline-reprocessor/reprocessor/feed.go:142 +0x75 fp=0xc0c5019fe0 sp=0xc0c5019ec0 pc=0x14b88b5
runtime.goexit({})
        /usr/local/Cellar/go/1.22.0/libexec/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0c5019fe8 sp=0xc0c5019fe0 pc=0x472361
created by github.com/GannettDigital/content-pipeline-reprocessor/reprocessor.(*allDatastoreFeed).stream in goroutine 1
        /Users/ggreer/p/go/src/github.com/GannettDigital/content-pipeline-reprocessor/reprocessor/feed.go:138 +0x9c
gannett-ggreer commented 6 months ago

The query should be the equivalent of datastore.NewQuery("exclusion").Project("ID").Order("ID") but I haven't verified that. The base context for the .Run of the query is just context.Background().

Edit: It also gets progressively slower and slower to retrieve entries from the query. I'm guessing because it has to walk that massive linked list of contexts to find some value.