search

googleapis / nodejs-firestore

Node.js client for Google Cloud Firestore: a NoSQL document database built for automatic scaling, high performance, and ease of application development.
https://cloud.google.com/firestore/
Apache License 2.0
643 stars 149 forks source link

query.stream() reportedly fails on very large result sets #1839

Open ehsannas opened 1 year ago

ehsannas commented 1 year ago

It was reported here https://github.com/googleapis/nodejs-firestore/issues/964#issuecomment-1500332130 that when query.stream() is used on a very large collection (or query with a large result set), the following error occurs:

Error: 9 FAILED_PRECONDITION: The requested snapshot version is too old.
    at Object.callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:360:49)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
    at /app/node_modules/@grpc/grpc-js/build/src/call-stream.js:188:78
    at processTicksAndRejections (node:internal/process/task_queues:78:11)
for call at
    at ServiceClientImpl.makeServerStreamRequest (/app/node_modules/@grpc/grpc-js/build/src/client.js:343:34)
    at ServiceClientImpl.<anonymous> (/app/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
    at /app/node_modules/@google-cloud/firestore/build/src/v1/firestore_client.js:225:29
    at /app/node_modules/google-gax/build/src/streamingCalls/streamingApiCaller.js:38:28
    at /app/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
    at Object.request (/app/node_modules/google-gax/build/src/streamingCalls/streaming.js:130:40)
    at makeRequest (/app/node_modules/google-gax/node_modules/retry-request/index.js:141:28)
    at retryRequest (/app/node_modules/google-gax/node_modules/retry-request/index.js:109:5)
    at StreamProxy.setStream (/app/node_modules/google-gax/build/src/streamingCalls/streaming.js:121:37)
    at StreamingApiCaller.call (/app/node_modules/google-gax/build/src/streamingCalls/streamingApiCaller.js:54:16) {
  code: 9,
  details: 'The requested snapshot version is too old.',

Thanks @klon for reporting this.

ehsannas commented 1 year ago

This only happens if the query result set is very large; so large that it would get the UNAVAILABLE after 60 sec error (as described in https://github.com/googleapis/nodejs-firestore/issues/964. To address this, the client under the hood attempts to "continue" getting the results by issuing a new request with a cursor (using startAfter). However, it is important that the new request (with startAfter) shows a consistent view of the documents at the time that the original query was made. If that time is "too old", the above error will be seen.

We plan to address this, but the timeline is currently not clear.

ehsannas commented 1 year ago

For the time being, if such consistency guarantee is not needed by the developer (or if they know the data is not fast changing), one could manually paginate these requests as was suggested in https://github.com/googleapis/nodejs-firestore/issues/964#issuecomment-1500332130.