Diagnostic tools and additional visualizations from "What Actions are Needed for Understanding Human Actions in Videos?" ICCV 2017
Contributor: Gunnar A. Sigurdsson
Please see the tool/, oracles/, and error_analysis/ subdirectories for the visualization tools and attributes for Charades! Let me know if you run into any problems.
To develop better algorithms for understanding activities, it will help to understand how and why they work. Looking solely at a single performance number obscures away from various details about the algorithms. To facilitate this process we are releasing a set of open-source Python scripts to generate detailed visualizations of one or more algorithms. The tool operates on the same file format required for the official Charades evaluation scripts, and therefore only needs a single submission file from each algorithm for analysis and comparison. The tool contains annotations for many attributes, and any subset of those can be selected for visualization. This allows to quickly generate diagnostic plots of one or more algorithms that fit into 1/4 of a page. These plots follow the same structure as the experiments section of the paper. A list of attributes and abbreviations is presented at the end of this section. We note that this analysis could be extended to any dataset, by following our methodology outlined in the paper to compute and collect the same attributes. To highlight the usefulness of the tool we consider use cases below.
First we visualize how individual baselines would look under the diagnostic. Output from the diagnostics tool is presented for multiple baselines on Charades. The same can be done for any algorithm that can do both classification and localization.
Diagnostic Output on Charades includes classification and localization. 0,1,2,3 denotes the degree of each continuous attribute and N,Y denotes if a binary attributes is present (Yes) or absent (No). Dashed lines indicate the overall performance of each algorithm. Blue line is classification and Brown line is localization performance.
If you want your algorith to appear here, you are welcome to send me an email with links to your classification and/or localization submission files on the Charades test set!
To understand the difference between multiple algorithms, or different versions of the same algorithm, the tool can also be used to visualize many different algorithms. A condensed summary of the plots in the paper are presented in below for classification on Charades.
Below we list the abbreviations used in the plots and their meanings. The continuous attributes are grouped into varying degrees of the attribute where 1 is the attribute is low and 4 the attribute is high.