search

googleapis / nodejs-pubsub

Node.js client for Google Cloud Pub/Sub: Ingest event streams from anywhere, at any scale, for simple, reliable, real-time stream analytics.
https://cloud.google.com/pubsub/
Apache License 2.0
521 stars 227 forks source link

Retry INTERNAL errors on more request types ? #1693

Closed n-e closed 6 months ago

n-e commented 1 year ago

Hi,

We're getting transient INTERNAL errors when calling createTopic or createSubscription:

Error: 13 INTERNAL: A service error has occurred. Please retry your request. If the error persists, please report it. [code=e8c0]
    at callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:192:76)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
    at /usr/src/app/node_modules/@grpc/grpc-js/build/src/resolving-call.js:94:78
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:160:34)
    at ServiceClientImpl.<anonymous> (/usr/src/app/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
    at /usr/src/app/node_modules/@google-cloud/pubsub/build/src/v1/subscriber_client.js:227:29
    at /usr/src/app/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
    at repeat (/usr/src/app/node_modules/google-gax/build/src/normalCalls/retries.js:80:25)
    at /usr/src/app/node_modules/google-gax/build/src/normalCalls/retries.js:118:13
    at OngoingCall.call (/usr/src/app/node_modules/google-gax/build/src/call.js:67:27)
    at NormalApiCaller.call (/usr/src/app/node_modules/google-gax/build/src/normalCalls/normalApiCaller.js:34:19)
    at /usr/src/app/node_modules/google-gax/build/src/createApiCall.js:84:30

According to the docs, these errors can be retried.

However, the nodejs client doesn't retry them, see https://github.com/googleapis/nodejs-pubsub/blob/main/src/v1/subscriber_client_config.json#L46 for example.

Should this be changed, at least for idempotent requests?

feywind commented 1 year ago

We were just talking about this last week. @kamalaboulhosn This might fit into that discussion.

feywind commented 7 months ago

Ping @kamalaboulhosn, not sure what the state of the INTERNAL retries is.

feywind commented 6 months ago

We're reassessing the errors for auto-retry now, will consider this as part of that.