Open stephenroller opened 3 years ago
This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.
Hi @klshuster and @stephenroller! I've just submitted a PR for this issue: https://github.com/facebookresearch/ParlAI/pull/4894
We have quite a few instances where we have some per-token losses/metrics along with a corresponding mask
And we want a per-batch-example average:
I'd like us to have a helper classmethod in
Metric
calledfrom_mask
:Once this is done, add unit tests for this (test AverageMetric and PPLMetric directly). Checkpoint there.
After you've implemented this, upgrade TorchGeneratorAgent to use your new helper, upgrading the code for loss, ppl, and token_acc.
Example: https://github.com/facebookresearch/ParlAI/blob/67433e376fc361dee5aa045cb6bb2b68d3faa478/parlai/core/torch_generator_agent.py#L711-L727
See if you can find at least one other place who can benefit from upgrading this pattern.