search

github / scientist

:microscope: A Ruby library for carefully refactoring critical paths.
MIT License
7.47k stars 442 forks source link

Exceptions in compare block shouldn't break normal execution #91

Closed jonathan-nye closed 6 years ago

jonathan-nye commented 6 years ago

The great thing about Scientist is being able to test out new code without breaking your existing functionality. However, I recently discovered that if you have a custom compare block, unhandled errors in that block will get raised up. I feel like comparing your control and candidate should also not break the normal execution if something goes wrong (i.e. you want your experiment to have as little impact as possible). I think this could be fixed by updating these lines to:

if comparator
  begin
    comparator.call(value, other.value)
  rescue StandardError
    false
  end
else
  ....

I don't have time to open a PR for this right now, but was wondering if this was a good addition or not.

jonathan-nye commented 6 years ago

FYI, I got around this by wrapping my comparison block with a begin/rescue and returning false in the rescue.

jonathan-nye commented 6 years ago

Actually, I guess the intention is that you would handle these in your own experiment by overriding the raised method.