RST_STREAM with code 0 on bulkExportRelationships

[edorivai]

Hi Team!

We use the experimental bulk export API in a cron to ensure our main DB and authzed are in sync.

As our relations dataset grows (currently over 400K relations), we're starting to see RST_STREAM errors pop up when our cron runs.

Error: 13 INTERNAL: Received RST_STREAM with code 0
    at Object.callErrorFromStatus (/home/edorivai/work/spot/api/node_modules/@grpc/grpc-js/src/call.ts:81:17)
    at Object.onReceiveStatus (/home/edorivai/work/spot/api/node_modules/@grpc/grpc-js/src/client.ts:596:32)
    at Object.onReceiveStatus (/home/edorivai/work/spot/api/node_modules/@grpc/grpc-js/src/client-interceptors.ts:424:48)
    at /home/edorivai/work/spot/api/node_modules/@grpc/grpc-js/src/call-stream.ts:330:24
    at processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at Proxy.makeServerStreamRequest (/home/edorivai/work/spot/api/node_modules/@grpc/grpc-js/src/client.ts:580:26)
    at Proxy.bulkExportRelationships (/home/edorivai/work/spot/api/node_modules/@authzed/authzed-node/src/authzedapi/authzed/api/v1/experimental_service.grpc-client.ts:93:21)
    at /home/edorivai/work/spot/api/node_modules/@authzed/authzed-node/src/util.ts:141:35
    at new Promise (<anonymous>)
    at Proxy.bulkExportRelationships (/home/edorivai/work/spot/api/node_modules/@authzed/authzed-node/src/util.ts:139:12)
    at AuthzedClient.bulkExportRelationships (/home/edorivai/work/spot/api/src/authzed/AuthzedClient.ts:106:39)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async getAuthzedExport (/home/edorivai/work/spot/api/src/authzed/sync.ts:172:24)
    at async /home/edorivai/work/spot/api/src/authzed/sync.ts:60:35
    at async Promise.all (index 0)
    at async sync (/home/edorivai/work/spot/api/src/authzed/sync.ts:54:7) {
  code: 13,
  details: 'Received RST_STREAM with code 0',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }

Couple things we noticed:

1. We'd already get this failure months back (when we had a considerably smaller relationship dataset) if we would run our script from our local dev machines. These dev machines are located in a different continent (EU, AS) than the authzed cluster (US). At that time, the same script calling bulkExportRelationships would run fine when run in GCP within the same region as our authzed cluster.
2. Backups through the zed CLI never run into this error, not even when running from our local dev machines. I think this points to an issue in this node SDK specifically.

Ran this again today to confirm - node.js script with bulkExportRelationships hits RST_STREAM, zed backup works fine

$ zed backup authzed-prod
⠏ backing up (3.8 MB, 225 kB/s) [19s]
3:04PM INF finished backup duration=20.028045293s perSecond=20236.373249148513 relationships=405295

Versions:

$ zed version
client: zed v0.15.2
service: v1.30.1-hotfix+enterprise.v1.hotfix.v1

$ yarn list --pattern '@authzed/authzed-node'
yarn list v1.22.19
warning Resolution field "@grpc/grpc-js@1.6.9" is incompatible with requested version "@grpc/grpc-js@^1.8.3"
└─ @authzed/authzed-node@0.14.0

[pbestoso-nourish]

My team has also been seeing these errors intermittently when calling lookupResources and other methods. We tried implementing a retry when we see this error, but we still see a large proportion of these requests hit the same error on the second try

[josephschorr]

https://github.com/grpc/grpc-node/issues/1532 mentions some approaches to debugging, but nothing concrete