fix bug in average ring buffer and add negative duration protection to scheduler decision maker

bdoyle0182 commented 1 year ago

Description

The average ring buffer has an edge case where discarding the max and min outliers from the average calculation can result in the average being negative due to the max and min not ever being reset even when the max and min value is ultimately discarded from the ongoing buffer. So the way it's set up it keeps the absolute max and absolute min duration for this function that has ever occurred. I just got rid of the max and min outlier discarding for now because I don't think it's adding much value and to appropriately update the max and min would require a performance hit to the ring buffer. I've also added negative duration buffer protection to the scheduling decision maker so that if such a case were introduced again in the future, the scheduling decision maker can more gracefully handle it and still fallback to scheduling containers when there are stale activations rather than thinking the amount of containers to schedule is negative.

Here is a raw example of how things can go wrong.

There is an edge case on the (sum - max - min) / (size - 2) calculation. The max never gets reset even if the element leaves the buffer. So say you have a random activation that takes 60s to run. The normal case is that all other activations take 50ms. If the buffer size is 10, then the sum should = 500 assuming the 60000 activation has left the buffer. however the max value will still be set to 60000.

(500 - 60000 - 50) / (10 - 2) = -7443ms

Related issue and scope

[ ] I opened an issue to propose and discuss this change (#????)

My changes affect the following components

[ ] API
[ ] Controller
[ ] Message Bus (e.g., Kafka)
[ ] Loadbalancer
[X] Scheduler
[ ] Invoker
[ ] Intrinsic actions (e.g., sequences, conductors)
[ ] Data stores (e.g., CouchDB)
[ ] Tests
[ ] Deployment
[ ] CLI
[ ] General tooling
[ ] Documentation

Types of changes

[X] Bug fix (generally a non-breaking change which closes an issue).
[ ] Enhancement or new feature (adds new functionality).
[ ] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

[X] I signed an Apache CLA.
[X] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[ ] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[ ] I updated the documentation where necessary.

codecov-commenter commented 1 year ago

Codecov Report

Merging #5396 (8f3f04d) into master (4e2dea1) will decrease coverage by 0.99%. The diff coverage is 62.50%.

:exclamation: Current head 8f3f04d differs from pull request most recent head f05bf00. Consider uploading reports for the commit f05bf00 to get more accurate results

@@            Coverage Diff             @@
##           master    #5396      +/-   ##
==========================================
- Coverage   76.77%   75.78%   -0.99%     
==========================================
  Files         240      240              
  Lines       14600    14596       -4     
  Branches      634      672      +38     
==========================================
- Hits        11209    11062     -147     
- Misses       3391     3534     +143

Impacted Files	Coverage Δ
...rg/apache/openwhisk/common/AverageRingBuffer.scala	`28.57% <0.00%> (+1.29%)`	:arrow_up:
...core/scheduler/queue/SchedulingDecisionMaker.scala	`97.02% <71.42%> (-1.94%)`	:arrow_down: