search

tweedegolf / storage-abstraction

Provides an abstraction layer for interacting with a storage; the storage can be local or in the cloud.
MIT License
106 stars 18 forks source link

GCP: Call to `getMetaData()` within `listFiles()` slows down the list operation a lot #55

Closed steprescott closed 7 months ago

steprescott commented 7 months ago

If you wish to list files from an adapter the metadata for the file is requested with no option to skip the request for metadata.

Doing this slows down the list operation by a great deal (less than a second without fetching metadata Vs 6+ seconds when requesting the metadata).

For my use case I only want to list the files and not the metadata.

Any thoughts why fetching metadata is coupled to listing files? Any suggestions on backwards compatible fix or perhaps decouple these in the API 2.0 design?

https://github.com/tweedegolf/storage-abstraction/blob/91dcd8315fc304e28cb2f00f446f9be3120ec7cf/src/AdapterGoogleCloudStorage.ts#L305

abudaan commented 7 months ago

Yes, you are right, I will fix this in API 2.1.

steprescott commented 7 months ago

Thanks for the quick reply!

abudaan commented 7 months ago

I have just published version 1.5.6, in this version the call to getMetaData has been removed in the Google adapter.

It might take a couple of days before I can publish version 2.1 so in the meantime you can update to 1.5.6 if you like.

steprescott commented 7 months ago

Wow! Fantastic support. I'll try it now.

steprescott commented 7 months ago

Forgot to say that it works well. Thank you.