phenomnomnominal / betterer

betterer makes it easier to make incremental improvements to your codebase
MIT License
569 stars 38 forks source link

Betterer Allows Tests to Get Worse After Meeting Goal #1181

Open joseph-ravenwolfe opened 7 months ago

joseph-ravenwolfe commented 7 months ago

Describe the bug When setting a goal with Betterer. Once the goal is met (for example, 0 linting errors), Betterer removes the last value from the cache. Once the value has been removed. You are able to make the goal worse (for example, introduce 50 linting errors), causing Betterer to start over and check the number 50 back into the cache.

To Reproduce

export default {
  'check linting': () =>
    new BettererTest({
      test: () => 50,
      constraint: smaller,
      goal: (value: number) => { return value === 0 },
      deadline: new Date('2023/11/01')
    })
};

Run Betterer.

-      test: () => 50,
+     test: () => 0,

Run Betterer, and receive a "Goal Met" message. Results are cleared.

-      test: () => 0,
+     test: () => 50,

Run Betterer, and receive a "for the first time" message. Result is now saved to 50.

Expected behavior Once a goal was met, you would expect that the goal never got worse from that point on. However, the goal can get a lot worse and Betterer appears to allow it.