Closed drewoldag closed 1 year ago
Patch and project coverage have no change.
Comparison is base (
64b35ec
) 100.00% compared to head (e6d174e
) 100.00%.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
@sschmidt23 As part of this PR, I copied the output file created in the estimation demo notebook into a data directory that this evaluation notebook references. However, the file is quiet large by git repo standards. @eacharles mentioned that you've got some code/experience with stashing larger files at NERSC and then pulling them in when necessary. If that's true, would you mind pointing me in the right direction for this PR, so that I can do the same in this notebook?
I just copied it to NERSC, and I see Eric already updated with a curl to grab it. I did not try running the notebook with this new file yet, though.
I'm crashing in the Evaluator stage in the demo notebook and also in goldenspike. I'm not sure it isn't a problem with my installation or my version of qp though, so I'd be curious of y'all see the same. Traceback below.
File ~/software/DESC/RAIL/src/rail/evaluation/evaluator.py:105, in Evaluator.run(self)
103 out_table = {}
104 for pit_metric in pit_metrics:
--> 105 value = PIT_METRICS[pit_metric]()
107 # The result objects of some meta-metrics are bespoke scipy objects with inconsistent fields.
108 # Here we do our best to store the relevant fields in `out_table`.
109 if isinstance(value, list): # pragma: no cover
File ~/software/DESC/qp/src/qp/metrics/pit.py:167, in PIT.evaluate_PIT_outlier_rate(self, pit_min, pit_max)
151 def evaluate_PIT_outlier_rate(self, pit_min=0.0001, pit_max=0.9999):
152 """Compute fraction of PIT outliers by evaluating the CDF of the distribution in the PIT Ensemble
153 at `pit_min` and `pit_max`.
154
(...)
165 The percentage of outliers in this distribution given the min and max bounds.
166 """
--> 167 return calculate_outlier_rate(self._pit, pit_min, pit_max)[0]
File ~/software/DESC/qp/src/qp/metrics/metrics.py:324, in calculate_outlier_rate(p, lower_limit, upper_limit)
321 except ValueError: #pragma: no cover - unittest coverage for _check_ensemble_is_not_nested is complete
322 logging.warning("Each element in the ensemble `p` must be a single distribution.")
--> 324 outlier_rates = [(dist.cdf(lower_limit) + (1. - dist.cdf(upper_limit)))[0][0] for dist in p]
325 return outlier_rates
File ~/software/DESC/qp/src/qp/metrics/metrics.py:324, in <listcomp>(.0)
321 except ValueError: #pragma: no cover - unittest coverage for _check_ensemble_is_not_nested is complete
322 logging.warning("Each element in the ensemble `p` must be a single distribution.")
--> 324 outlier_rates = [(dist.cdf(lower_limit) + (1. - dist.cdf(upper_limit)))[0][0] for dist in p]
325 return outlier_rates
@eacharles is that the whole error output that you're seeing? I'm running the notebook locally, and am not seeing that message.
@eacharles @sschmidt23 both the goldenspike and evaluation_examples/demo notebooks run after the latest bug fix to qp (and subsequent release of qp v0.8.3).
Fixed references to PIT metric implementations in RAIL, now points to qp implementations. Added output file to estimation_data/data dir.