LSSTDESC / rail_attic

Redshift Assessment Infrastructure Layers
MIT License
14 stars 9 forks source link

Not all PITMetaMetrics are calculated when the user calls PIT.evaluate #279

Closed drewoldag closed 1 year ago

drewoldag commented 2 years ago

In the PIT.evaluate method this code will automatically evaluate the PITMetaMetrics that are decorated with the @PITMetaMetric decorator. However, the decorator has a slight logic error.

In the decorator in pit.py we're looking for a evaluate method in the decorated class that has default arguments defined. However, only two of the metametrics have a default arugment. The others don't have input arguments that require default args - those metrics are not included.

As it stands, only PITOutRate and PITAD are evaluated by default while KS and CvM are not. The evaluate methods of KS and CvM do not include arguments other than self.

The action item here is to fix the logic in the decorator such that all the MetaMetrics are included as expected.

drewoldag commented 1 year ago

Now that PIT metrics have been moved from RAIL to qp, the code that evaluated the meta metrics has been refactored as well. A new method, calculate_pit_meta_metrics does the work of calculating all the meta metrics. The user also has the option of calling each meta metric calculation individually.

The code can be seen here: https://github.com/LSSTDESC/qp/blob/main/src/qp/metrics/pit.py#L78