riedgar-ms
commented
4 years ago This is something I've started cogitating on again, in the context of the AzureML - now MLFlow - integration. We do want to enable composability, but also avoid saving lots copies of y_true etc. in the composable pieces.
Then again, we also need an API which allows users to 'mess around in a notebook' without having to set up a bunch of prerequisites. A small example of this is how the dashboard doesn't require model and sensitive feature names, but will generate them itself if invoked without them. In contrast _create_group_metric_set() absolutely requires these names, to minimise the chances of users shooting themselves in the foot.
I've been thinking a bit about this over the weekend, and there are two slightly different sets of changes which are required.
The first is on the level of the entire dashboard JSON. I think that we should merge the
modelNamesandpredictedYportions into a single list of dictionaries, similar to the way the sensitive features are handled. I'm guessing that the current separation is just a historical accident. At this time, we can also decide on the keys we want - basically, camel or snake case. These are fairly mechanical changes, but would impose a very non-zero cost to rollout.The second is the trickier one, and involves the contents of those dictionaries in the
precomputedMetrics(2d) array. I've not got very far with my thoughts yet, but essentially.... if we want to make things less brittle (and also enable third parties to contribute their own views of the statistics), we also need to make sure that we devise something queryable. Basically, a 'view' would list its required statistics, and the 'engine' inside the dashboard would be responsible for matching the available views up with the available statistics. How all that works.... I'm not sure yet.