AffordanceNet and AffContext

This is tensorflow AffordanceNet and AffContext implementations. Both are implemented and tested with tensorflow 2.3.

The main objective of both architectures is to identify action affordances, so that they can be used in real robotic applications to understand the diverse objects present in the environment.

Both models have been trained on IIT-AFF and UMD datasets.

Detections on novel image

Example of ground truth affordances compared with the affordance detection results by AffordanceNet and AffContext on the IIT-AFF dataset.

Example of ground truth affordances compared with the affordance detection results by AffordanceNet and AffContext on the UMD dataset.

AffordanceNet simultaneously detects multiple objects with their corresponding classes and affordances. This network mainly consists of two branches: an object detection branch to localise and classify the objects in the image, and an affordance detection branch to predict the most probable affordance label for each pixel in the object.

AffContext correctly predicts the pixel-wise affordances independently of the class of the object, which allows to infer the affordances for unseen objects. The structure of this network is similar to AffordanceNet, but the object detection branch only performs binary classification into foreground and background areas, and it includes two new blocks: an auxiliary task to infer the affordances in the region and a self-attention mechanism to capture rich contextual dependencies through the region.

Results

The results of the tensorflow implementation are contrasted with the values provided in the papers from AffordanceNet and AffContext. However, since the procedure of how the results are processed to obtain the final metrics in both networks may be different, the results are also compared with the values obtained by running the original trained models, but processing the outputs and calculating the measures with the code from this repository. These results are denoted with * in the comparison tables.

Setup guide

Requirements

• Python 3
• CUDA 10.1

Installation

1. Clone the repository into your $AffordanceNet_ROOT folder.

2. Install the required Python3 packages with: pip3 install -r requirements.txt

Testing

1. Download the pretrained weights:

   • AffordanceNet weights trained on IIT-AFF dataset.
   • AffContext weights trained on UMD dataset.

2. Extract the file into $AffordanceNet_ROOT/weights folder.

3. Visualize results for AffordanceNet trained on IIT-AFF dataset:

   python3 affordancenet_predictor.py --config_file config_iit_test

4. Visualize results for AffContext trained on UMD dataset:

   python3 affcontext_predictor.py --config_file config_umd_test

Training

1. Download the IIT-AFF or UMD datasets in Pascal-VOC format following the instructions in AffordanceNet (IIT-AFF) and AffContext(UMD).

2. Extract them into the $AffordanceNet_ROOT/data folder and make sure to have the following folder structure for IIT-AFF dataset:

   • cache/
   • VOCdevkit2012/

The same applies for UMD dataset, but folder names should be cache_UMD and VOCdevkit2012_UMD

3. Run the command to train AffordanceNet on IIT-AFF dataset:

   python3 affordancenet_trainer.py --config_file config_iit_train

4. Run the command to train AffContext on UMD dataset:

   python3 affcontext_trainer.py --config_file config_umd_train

Acknowledgements

This repo used source code from AffordanceNet and Faster-RCNN