Clarify the type and number of Cloud Storage Operations performed by gcsfuse

GoogleCloudPlatform / gcsfuse

A user-space file system for interacting with Google Cloud Storage

https://cloud.google.com/storage/docs/gcs-fuse

Apache License 2.0

2.04k stars 423 forks source link

Clarify the type and number of Cloud Storage Operations performed by gcsfuse #853

Open steve-marmalade opened 1 year ago

steve-marmalade commented 1 year ago

Hi there,

I am trying debug a larger than expected number of Cloud Storage Class A operations, and I'm wondering if my usage of gcsfuse may explain it.

Concretely, if I mount a bucket via gcsfuse --implicit-dirs my-bucket /my-bucket and then attempt to read a file in a somewhat deeply nested directory /my-bucket/directory1/directory2/directory3/file, how many Class A Operations would you expect this to make?

I expected the answer to be 0 (just a single storage.objects.get, which is a Class B operation), but from re-reading the --implicit-dirs semantics, it looks like there will be at least one storage.objects.list.

Thanks for your help and for this useful utility.

Tulsishah commented 1 year ago

It is four in this case.
The number of calls is equal to the numbers of directories in the path + one for file name as mentioned here --implicit-dirs semantics

steve-marmalade commented 1 year ago

@Tulsishah , thank you very much for the response.

When I initially read the docs, I did not appreciate:

That list requests would be made even when I am attempting to load a specific file (as opposed to listing the contents of a directory)
- By extension, that there would be a list request for each directory in addition to the file.
That storage.objects.list requests are billed differently from storage.objects.get requests (the former are more expensive)

To summarize, I did not expect there to be significant billing implications to consider when using gcsfuse, but there clearly are. This might be worth highlighting more prominently in the documentation (I certainly would have appreciated it).

ahmedetefy commented 7 months ago

I completely agree with @steve-marmalade 's sentiment on this. We also had significant billing implications with using gcsfuse because the list operations are billed as class A operations which are significantly more costly than class B operations.

This might be worth highlighting more prominently in the documentation (I certainly would have appreciated it).