facebookresearch / dora

Dora is an experiment management framework. It expresses grid searches as pure python files as part of your repo. It identifies experiments with a unique hash signature. Scale up to hundreds of experiments without losing your sanity.
MIT License
262 stars 24 forks source link

Grid Explorer can now display best score #29

Closed louismartin closed 2 years ago

louismartin commented 2 years ago

This allows displaying the best scores in the grid monitoring instead of the last scores.

Behaviour can be defined using:

class MyExplorer(Explorer):
    update_modes = {"train_loss": "min", "val_loss": "min", "val_f1_score": "max"}

    def get_grid_metrics(self):
        return [
            tt.group(
                "metrics",
                [
                    tt.leaf("step", "d"),
                    tt.leaf("train_loss", ".3f"),
                    tt.leaf("val_loss", ".3f"),
                    tt.leaf("val_f1_score", ".1%"),
                ],
            )
        ]

    def process_history(self, history):
        updated_metrics = super().process_history(history)
        return {"metrics": updated_metrics}

The code is backward compatible and the mode will fallback to "last" if not specified.

adefossez commented 2 years ago

I would consider this is up to the final users to set those up in the way they want.

Personally I use nested dictionaries for valid vs. train etc. and your PR wouldn't support that. Also you can imagine endless variations why not average the last 5 epochs ? introduce a standard deviation etc. I don't want to have to worry about all that in Dora. Both here and for grids definition, any utility function belong in the final user code, and the goal of Dora there ### is to stay just a minimum viable product.

louismartin commented 2 years ago

Ok fair I understand the reasoning