googleapis / google-cloud-go

Google Cloud Client Libraries for Go.
https://cloud.google.com/go/docs/reference
Apache License 2.0
3.76k stars 1.29k forks source link

firestore: Estimate usage statistics for a firestore client #7518

Open yabberyabber opened 1 year ago

yabberyabber commented 1 year ago

For large applications that depend on firestore, it is often difficult to determine which modules are doing the most firestore operations. The google cloud console interface gives a good overview for how many of each API request types are happening at a global level, but it is difficult to get more granular data on what parts of code are doing the most/least operations.

Describe the solution you'd like I think this problem could be well solved if the firestore.Client type exposed a method that would return the number of read/write/delete operations performed by that client. The interface might look like so:

type ClientStats struct {
  Reads int
  Writes int
  Deletes int
}

func (c *Client) Stats() ClientStats {
    // to be implemented in the firestore.Client type by reading from counters
    // which get updated within the client as it makes requests
}

Consumers of this API could very easily write some glue code to translate the ClientStats type into whatever monitoring framework they use in their application such as prometheus, influxdb, datadog, etc. These ClientStats would be specific to a firestore.Client instance, so granular usage could be determined by creating several firestore.Client instances and passing them into different subsystems.

I'm happy to push a PR for this. I have looked at the source code and I have a pretty good idea of which methods need to increment which counters. I would really like to get sign-on from a maintainer first, though, as I don't know whether there's an alternate implementation that already exists or is on the roadmap.

Describe alternatives you've considered I don't think it is possible to get usage statistics from the API as-is. Please let me know if I am wrong about this, because the easiest code to write is the code that is already written :)

If firestore.Client were an interface (rather than a struct) it would be easy to define my own type which implements the firestore.Client interface and wraps the upstream firestore.Client. My proxy type could maintain usage statistics while deferring all actual API calls to the inner upstream type. Unfortunately, firestore.Client is not an interface, but rather a struct. And most existing code that uses firestore expects the firestore.Client struct type rather than an interface.

I see that there is a clientHook variable in the firestore package. I'm not sure exactly how this is used, but I wonder if it may be possible for me to inject my own hook such that my monitoring code gets executed for every external API call.

meredithslota commented 1 year ago

Hi @yabberyabber — thanks for filing this! We will have to check and see what the recommended solution for this is currently. My first thought would be to integrate Cloud Logging in some way, but I'm not sure if that would meet your needs. https://cloud.google.com/firestore/docs/audit-logging as a starting point.