googleapis / nodejs-storage

Node.js client for Google Cloud Storage: unified object storage for developers and enterprises, from live data serving to data analytics/ML to data archiving.
https://cloud.google.com/storage/
Apache License 2.0
896 stars 370 forks source link

what is the page limit of getFiles with autoPaginate=false #2429

Closed convers39 closed 6 months ago

convers39 commented 6 months ago

In the official example below, it shows that using autoPaginate = false meanwhile tracking a cursor of pageToken.

https://github.com/googleapis/nodejs-storage/blob/36d27212848a1002cc0fa699364a446c4f8d25e4/samples/listFilesPaginate.js#L32-L48

It implies there is a limit to how many files it could get in one request, but I could not find any documentation about it.

Somehow I could not find the implementation of getFiles by searching getFiles in this repo, does anyone have any idea about the getFiles page limit?

And if there is a limit anyway, why not suggest using stream instead of the example above? The example seems not very practical as in most cases you don't know how many pages there are.

ddelgrosso1 commented 6 months ago

Hi @convers39 the recommended upper limit is 1000. However, the API may choose to return less than the requested number. See the documentation for maxResults here. You are correct that you would not know the number of pages in advance which is why the documentation recommends checking for the presence of nextPageToken on the response.

The implementation for getFiles can be found here. We also provide the function getFilesStream which will stream the results.

convers39 commented 6 months ago

Hi @ddelgrosso1 , thank you for your explanation.

the recommended upper limit is 1000. However, the API may choose to return less than the requested number

I see, so it would be better to use autoPaginate:false with maxResults together. And if the files are about to be over 1000 (or the API cannot return that much), it is better to retrieve them with nextPageToken, or getFilesStream.

Indeed I am using getFilesStream now. Thank you for your help!