Fail Betterer on expired deadline

[joseph-ravenwolfe]

We currently have a use case where we would like test coverage to increase by a certain deadline. If coverage does not increase, we would like to fail Betterer, prompting an Engineer to increase coverage at this point, before continuing to ship features.

If the goal has been met and the deadline has passed, we would like Betterer to pass with exit code 0 and mark the test as expired.

If the goal not been met, and the deadline has passed, we would like Betterer to fail with exit code 1 until either, the deadline has been increased, or the goal has been met. If the goal has been met, it would go back to expired behavior with success code.

Without Betterer, we would employ this restriction at an organizational level. (but it would need to be a far more manual process)

Does Betterer provide functionality like this?

[joseph-ravenwolfe]

Thinking about this further. I think there are at least 2 additional data points you'd need to capture: the first recorded value for the test, and the most recent deadline. Because, in the example of seeing a number get smaller by a deadline, you need to know:

• Has the deadline passed?
• Has the number gotten smaller at any point since the deadline was set.

Deadline First Set (current date: Nov 1 2023)

Result: 'First Seen'

  exports[`check linters`] = {
+   initValue: `40`,
    value: `40`,
+   deadline: `2024-01-01`'
  };

Betterer Run Before Deadline

Result: 'Doing Better'

  exports[`check linters`] = {
    initValue: `40`,
-   value: `40`,
+   value: `30`,
    deadline: `2024-01-01`'
  };

Betterer Run After Deadline (Pass Case)

Result: 'Deadline Expired'

  exports[`check linters`] = {
    initValue: `40`,
    value: `30`,
    deadline: `2024-01-01`'
  };

Betterer Run After Deadline (Fail Case)

Result: 'Failed: Deadline Reached With No Improvement'

  exports[`check linters`] = {
    initValue: `40`,
    value: `40`,
    deadline: `2024-01-01`'
  };

Deadline Updated

Result: 'First Seen'

  exports[`check linters`] = {
-   initValue: `40`,
+   initValue: `30`,
    value: `30`,
-   deadline: `2024-01-01`'
+   deadline: `2024-02-01`'
  };