Use qp-based PIT metrics instead of deprecated RAIL-based PIT metrics

[drewoldag]

Fixed references to PIT metric implementations in RAIL, now points to qp implementations. Added output file to estimation_data/data dir.

[codecov[bot]]

Codecov Report

Patch and project coverage have no change.

Comparison is base (64b35ec) 100.00% compared to head (e6d174e) 100.00%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #353 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 38 38 Lines 2581 2581 ========================================= Hits 2581 2581 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `100.00% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LSSTDESC#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://app.codecov.io/gh/LSSTDESC/RAIL/pull/353?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LSSTDESC) | Coverage Δ | | |---|---|---| | [src/rail/core/utilStages.py](https://app.codecov.io/gh/LSSTDESC/RAIL/pull/353?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LSSTDESC#diff-c3JjL3JhaWwvY29yZS91dGlsU3RhZ2VzLnB5) | `100.00% <ø> (ø)` | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[drewoldag]

@sschmidt23 As part of this PR, I copied the output file created in the estimation demo notebook into a data directory that this evaluation notebook references. However, the file is quiet large by git repo standards. @eacharles mentioned that you've got some code/experience with stashing larger files at NERSC and then pulling them in when necessary. If that's true, would you mind pointing me in the right direction for this PR, so that I can do the same in this notebook?

[sschmidt23]

I just copied it to NERSC, and I see Eric already updated with a curl to grab it. I did not try running the notebook with this new file yet, though.

[eacharles]

I'm crashing in the Evaluator stage in the demo notebook and also in goldenspike. I'm not sure it isn't a problem with my installation or my version of qp though, so I'd be curious of y'all see the same. Traceback below.

File ~/software/DESC/RAIL/src/rail/evaluation/evaluator.py:105, in Evaluator.run(self)
    103 out_table = {}
    104 for pit_metric in pit_metrics:
--> 105     value = PIT_METRICS[pit_metric]()
    107     # The result objects of some meta-metrics are bespoke scipy objects with inconsistent fields.
    108     # Here we do our best to store the relevant fields in `out_table`.
    109     if isinstance(value, list):  # pragma: no cover

File ~/software/DESC/qp/src/qp/metrics/pit.py:167, in PIT.evaluate_PIT_outlier_rate(self, pit_min, pit_max)
    151 def evaluate_PIT_outlier_rate(self, pit_min=0.0001, pit_max=0.9999):
    152     """Compute fraction of PIT outliers by evaluating the CDF of the distribution in the PIT Ensemble
    153     at `pit_min` and `pit_max`.
    154 
   (...)
    165         The percentage of outliers in this distribution given the min and max bounds.
    166     """
--> 167     return calculate_outlier_rate(self._pit, pit_min, pit_max)[0]

File ~/software/DESC/qp/src/qp/metrics/metrics.py:324, in calculate_outlier_rate(p, lower_limit, upper_limit)
    321 except ValueError:  #pragma: no cover - unittest coverage for _check_ensemble_is_not_nested is complete
    322     logging.warning("Each element in the ensemble `p` must be a single distribution.")
--> 324 outlier_rates = [(dist.cdf(lower_limit) + (1. - dist.cdf(upper_limit)))[0][0] for dist in p]
    325 return outlier_rates

File ~/software/DESC/qp/src/qp/metrics/metrics.py:324, in <listcomp>(.0)
    321 except ValueError:  #pragma: no cover - unittest coverage for _check_ensemble_is_not_nested is complete
    322     logging.warning("Each element in the ensemble `p` must be a single distribution.")
--> 324 outlier_rates = [(dist.cdf(lower_limit) + (1. - dist.cdf(upper_limit)))[0][0] for dist in p]
    325 return outlier_rates

[drewoldag]

@eacharles is that the whole error output that you're seeing? I'm running the notebook locally, and am not seeing that message.

[drewoldag]

@eacharles @sschmidt23 both the goldenspike and evaluation_examples/demo notebooks run after the latest bug fix to qp (and subsequent release of qp v0.8.3).