🚀 Hey all, this PR adds a little feature on top of the WandbLogger that I have improved/extended in this PR, #1137. The PR adds the prediction report generated when evaluation.predict=true as W&B Tables.
What is W&B Tables?
It's an interactive way of going through your dataset and model prediction. It is a two dimensional grid of data where each column has a single type of data—think of this as a more powerful DataFrame. Tables support primitive and numeric types, as well as nested lists, dictionaries, and rich media types. Log a Table to W&B, then query, compare, and analyze results in the UI.
One can now keep track of the model prediction result. I found keeping track of the same using json or csv challenging especially when I am revisiting the project after few days.
What more can be done?
So far all good, but W&B Tables has a lot to offer. We can log the actual input question and image instead of question_id and image_id. Any help in this direction would be useful.
Future additions/improvement
Here's a crude road map for this feature.
The following is how I see it progressing, which might change in the future.
[x] Log the prediction report as W&B Tables. The current solution is generic in my opinion.
[ ] Add ground truth answer for the given dataset and see if images are available to be logged as a column.
[ ] Add ability to log media.
I would love to know if you are interested in this. :D
🚀 Hey all, this PR adds a little feature on top of the
WandbLogger
that I have improved/extended in this PR, #1137. The PR adds the prediction report generated whenevaluation.predict=true
as W&B Tables.What is W&B Tables?
It's an interactive way of going through your dataset and model prediction. It is a two dimensional grid of data where each column has a single type of data—think of this as a more powerful DataFrame. Tables support primitive and numeric types, as well as nested lists, dictionaries, and rich media types. Log a Table to W&B, then query, compare, and analyze results in the UI.
(Source)
Modification
This PR adds a
log_prediction_report
to theWandbLogger
. This method is called in thetrainer/core/evaluation_loop.py
file (a 2-3 line addition).Usage
To use this, in the
config/defaults.yaml
do,training.wandb.enabled=true
,training.wandb.log_tables=true
andevaluation.predict=true
.Result
Here's the Tables page that you can check out.
Superpowers
One can now keep track of the model prediction result. I found keeping track of the same using
json
orcsv
challenging especially when I am revisiting the project after few days.What more can be done?
So far all good, but W&B Tables has a lot to offer. We can log the actual input question and image instead of
question_id
andimage_id
. Any help in this direction would be useful.Future additions/improvement
Here's a crude road map for this feature.
The following is how I see it progressing, which might change in the future.
I would love to know if you are interested in this. :D