[Feature] average performace metric for all validation dataset

[serious-gist]

average performace metric for all validation dataset

Assume the model is train using dataset A and validated against dataset B and dataset C.

I read somewhere either in the documentation or issue being raise in Github (Both of which I am unable to trace back now 😔 😔) that the build-in script outputs performance metric of the dataset B and C seperately.

But instead, I would like have the average performance metric for overall validation dataset alongwith for sepeate dataset. May I know which .py function that I can modify to achieve the desire requirement? or is their anything I have to do in the abinet config.

currently my config looks like this below for

val_evaluator = dict(
    type="MultiDatasetEvaluator"
    metrics=[
        dict(type="WordMetric", mode=["exact", "ignore_case", "ignore_case_symbol"]),
        dict(type="CharMetric"),
    ],
    dataset_prefixes = ['A', 'B']
)

Current: After each epoch:

{"A/word_acc": 0.67, "A/word_acc_ignore_case": .30, "A/word_acc_ignore_case_symbol": 0.2"
"B/word_acc": 0.22, "B/word_acc_ignore_case": 0.33, "B/word_acc_ignore_case_symbol": 0.23}

Wish to have: After all the epochs or maybe each epoch:

``{"mean_of_oveall_dataset/word_acc": 0.67, "mean_of_oveall_dataset/word_acc_ignore_case": .30, "mean_of_oveall_dataset/word_acc_ignore_case_symbol": 0.2"}`
`

[gaotongxiao]

That happens on our roadmap. Thanks for your suggestion!