Documenting all Score edge cases that break optimization gradients

[rgerkin]

@russelljjarvis Could you give me a list here of examples of all of the kinds of Score edge cases that are messing up optimization. Some of them may have common solutions so I'd like to collect them in one issue. For example:

• [ ] ZScore: Standard deviation is 0
• [ ] RatioScore: Prediction and observation are opposite sign
• [ ] RatioScore: Observaion is 0
• [ ] etc...

See also: russelljjarvis/neuronunit_opt#26, russelljjarvis/neuronunit_opt#27

[russelljjarvis]

I would if I had time, but I am sure that I wont as I am experiencing intense time pressure in other life domains. This could be accomplished by using the logger.info inside except clauses of bridge_judge and bridge_passive in the optimization_management.py file. See multiple other GH issues (maybe the ones you flagged).

[rgerkin]

Is it OK if all these cases just get returned as InsufficientDataScore or ErrorScore or something like that? Then you can choose what optimizer scores to give them (e.g. the worst numerical score). This will cause discontinuities in the objective function, but I don't know that this can be avoided.

[russelljjarvis]

Unfortunately, I don't yet have the infrastructure in place to catch these bad pre-score values, but score.log_norm_score does fail for some score values.

~/git/sciunit/sciunit/suites.py in judge(self, models, skip_incapable, stop_on_error, deep_error, parallel, log_norm)
    161                 for test in self.tests:
    162                     score = self.judge_one(model, test, sm, skip_incapable,
--> 163                                               stop_on_error, deep_error)
    164                     if log_norm:
    165                         if score.get_raw() != 0:

~/git/sciunit/sciunit/suites.py in judge_one(self, model, test, sm, skip_incapable, stop_on_error, deep_error)
    203                                stop_on_error=stop_on_error,
    204                                deep_error=deep_error)
--> 205             log('Score is <a style="color: rgb(%d,%d,%d)">' % score.color()
    206                 + '%s</a>' % score)
    207         sm.loc[model, test] = score

~/git/sciunit/sciunit/scores/base.py in color(self, value)
    113         """Turn the score intp an RGB color tuple of three 8-bit integers."""
    114         if value is None:
--> 115             value = self.norm_score
    116         rgb = Score.value_color(value)
    117         return rgb

~/git/sciunit/sciunit/scores/complete.py in norm_score(self)
    168         """Return 1.0 for a ratio of 1, falling to 0.0 for extremely small
    169         or large values."""
--> 170         score = math.log10(self.score)
    171         cdf = (1.0 + math.erf(score / math.sqrt(2.0))) / 2.0
    172         return 1 - 2*math.fabs(0.5 - cdf)

ValueError: math domain error