add bytes written Counter to Prometheus HTTP request listener

[nginthfs]

Background

We currently export metrics for indexing request counts and latencies, but we do not have a metric for throughput. This is important to accurately measure against real running Solr clusters.

Solution

This PR first adds the ability to register a Prometheus counter with the PrometheusExportManager. Next, it adds a Prometheus Counter metric called solr_bench_data_write_total that represents the total number of bytes written to the Solr cluster via UploadDocs.

[patsonluk]

Ah very useful metrics indeed! Thank you for adding this feature 😊

Several thoughts:

1. My guess of the existing totalUploadedDocs being passed as a mutable AtomicLong is to have the outer logic to eventually be able to read it to print in the final json output. Therefore your proposed bytesUploaded follows the same pattern which makes a lot of sense. However, if we are only going to use it for the prometheus metrics, may I suggest not passing that value around (Besides, I sometimes find it a bit hard to track mutable variables being passed into ctor/method, as there's a much bigger scope to track the mutation). Instead, we can change UploadDocs#call to return the "result of the Upload operation", which can be the bytes uploaded. Then we no longer need to embed that metrics via the key but read it from the "UploadDoc operation result". Perhaps we can introduce a new type (PrometheusUploadMetricsListener?) of listener and use it at here that handles specifically Upload metrics as the existing PrometheusHttpRequestDurationListener applies to both indexing and querying.
2. Do you think it's helpful to report the # of docs uploaded via Prometheus as well? If we use the suggestion in 1., then UploadDocs#call can instead return a class of UploadDocResult, which contains 2 fields , bytesUploaded and docsUploaded, we can populate those in that method, and then PrometheusUploadMetricsListener#onExecutionComplete should have both values available for reporting. The added benefit of returning a UploadDocResult class is that it's much more readable than returning a generic long IMHO. 😊

[chatman]

+1, great idea! Will look into it in more detail tomorrow.

On Fri, 2 Feb, 2024, 1:01 am patsonluk, @.***> wrote:

Ah very useful metrics indeed! Thank you for adding this feature 😊

Several thoughts:

1. My guess of the existing totalUploadedDocs being passed as a mutable AtomicLong is to have the outer logic to eventually be able to read it to print in the final json output. Therefore your proposed bytesUploaded follows the same pattern which makes a lot of sense. However, if we are only going to use it for the prometheus metrics, may I suggest not passing that value around (Besides, I sometimes find it a bit hard to track mutable variables being passed into ctor/method, as there's a much bigger scope to track the mutation). Instead, we can change UploadDocs#call to return the "result of the operation", which can be the bytes uploaded. Then we no longer need to embed that metrics via the key but read it from the "UploadDoc operation result". Perhaps we can introduce a new type (PrometheusUploadMetricsListener?) of listener and use it at here https://github.com/fullstorydev/solr-bench/blob/master/src/main/java/org/apache/solr/benchmarks/BenchmarksMain.java#L309 that handles specifically Upload metrics as the existing [PrometheusHttpRequestDurationListener] applies to both indexing and querying.
2. Do you think it's helpful to report the # of docs uploaded via Prometheus as well? If we use the suggestion in 1., then UploadDocs#call can instead return a class of UploadDocResult, which contains 2 fields , bytesUploaded and docsUploaded, we can populate those in that method, and then PrometheusUploadMetricsListener#onExecutionComplete should have both values available for reporting. 😊

— Reply to this email directly, view it on GitHub https://github.com/fullstorydev/solr-bench/pull/91#issuecomment-1922078240, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDCR5FIGCJ2UWL3RNADO5LYRPUR7AVCNFSM6AAAAABCVLT5CCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRSGA3TQMRUGA . You are receiving this because you were assigned.Message ID: @.***>

[nginthfs]

Ah very useful metrics indeed! Thank you for adding this feature 😊

Several thoughts:

1. My guess of the existing totalUploadedDocs being passed as a mutable AtomicLong is to have the outer logic to eventually be able to read it to print in the final json output. Therefore your proposed bytesUploaded follows the same pattern which makes a lot of sense. However, if we are only going to use it for the prometheus metrics, may I suggest not passing that value around (Besides, I sometimes find it a bit hard to track mutable variables being passed into ctor/method, as there's a much bigger scope to track the mutation). Instead, we can change UploadDocs#call to return the "result of the Upload operation", which can be the bytes uploaded. Then we no longer need to embed that metrics via the key but read it from the "UploadDoc operation result". Perhaps we can introduce a new type (PrometheusUploadMetricsListener?) of listener and use it at here that handles specifically Upload metrics as the existing PrometheusHttpRequestDurationListener applies to both indexing and querying.
2. Do you think it's helpful to report the # of docs uploaded via Prometheus as well? If we use the suggestion in 1., then UploadDocs#call can instead return a class of UploadDocResult, which contains 2 fields , bytesUploaded and docsUploaded, we can populate those in that method, and then PrometheusUploadMetricsListener#onExecutionComplete should have both values available for reporting. The added benefit of returning a UploadDocResult class is that it's much more readable than returning a generic long IMHO. 😊

1. That sounds like a good approach! I mostly did it like this since it seemed like an easy way to re-use code that was already there, but I definitely agree with it being hard to track global mutable variables like that, especially across threads.
2. Hmm, I think it could definitely be useful to see the number of docs uploaded! I didn't think about it first since I'm very closely keeping track of how many docs were uploaded for my use case, but it seems like a nice convenience to have, and not too hard to implement.