[Bug Report]: http_requests_queue_items_total is not zero after process entire queue

[tiagodaraujo]

Prerequisites

• [X] I have searched issues to ensure it has not already been reported

Description

The http_requests_queue_items_total metric is being used as a counter to report the total task waiting in the queue.

Sometimes after the queue is empty the gauge reports more than 0 items in the queue.

Steps to reproduce

1. Open the Solution
2. Go to AdaptativeConcurrencyLimiterTests test class
3. Change the assertions in AssertMetrics method as described below
4. Run the GetAsync_WithReducedLimitAndQueueSize_SomeRequestsAreRejected test until failure (use the Test Explorer Run Until Failure)
5. After 5 attempts the tests should fail

Assert.Contains("http_requests_concurrency_items_total{method=\"GET\",priority=\"normal\"} 0", content);
Assert.Contains("http_requests_concurrency_limit_total", content);
Assert.Contains("http_requests_task_processing_time_seconds", content);
Assert.Contains("http_requests_queue_items_total{method=\"GET\",priority=\"normal\"} 0", content);
Assert.Contains("http_requests_queue_limit_total", content);
Assert.Contains("http_requests_queue_time_seconds_sum{method=\"GET\",priority=\"normal\"}", content);
Assert.Contains("http_requests_queue_time_seconds_count{method=\"GET\",priority=\"normal\"}", content);
Assert.Contains("http_requests_queue_time_seconds_bucket{method=\"GET\",priority=\"normal\",le=\"0.0005\"}", content);
Assert.Contains("http_requests_rejected_total{method=\"GET\",priority=\"normal\",reason=\"max_queue_items\"}", content);

Expected behavior

After processing all the tasks the metrics http_requests_queue_items_total and http_requests_concurrency_items_total must be 0 for all methods and priorities

Actual behavior

When the test fails, the http_requests_queue_items_total is 1, however it must be 0.

I saw one test failing which method where http_requests_queue_items_total{method=GET} 1 and http_requests_queue_items_total{method=UNKNOWN} -1

LoadShedding version

1.0.0

[kikofps]

Hi @tiagodaraujo ! Thanks for reporting the issue, we'll investigate and give you feedback.

[tiagodaraujo]

Hi @kikofps, I just opened a pull request to review and fix this issue

https://github.com/Farfetch/loadshedding/pull/23

Thanks in advance

[kikofps]

Hi @tiagodaraujo , Thank you, we're reviewing the pull request.