Generative Modeling for Multi-task Visual Learning

This is the repository for Generative Modeling for Multi-task Visual Learning, published at ICML 2022.

[Paper] [Slides] [Poster] [Talk]

Dataset architecture

root 
    - rgb 
        - scene_class_0
            - 001.png
            - 002.png
            - ...
        - scene_class_1
    - ss
        - scene_class_0
            - 001.npy
            - ...
    - de
        - scene_class_0
            - 001.npy
            - ...
    - sn
        - scene_class_0
            - 001.png
            - ...
    - msk
        - scene_class_0
            - 001.png
            - ...

Full MGM model training

To run the full model, due to the training mismatch of the GAN and discriminative model, we need to first pre-train the GAN and then train the full model. See train.py and train.sh for sampled training process.

Light-weight MGM model training

The refinement network and the self-supervision networks are the key in the paper. These modeules can help as long as we have some weakly-labelled data. Thus, we provide a light-weight MGM model -- we did not include a generative model in the framework, but assume we have some weakly-labeled data (no matter it is real or synthesized) to train the model.

see MGM_light in mgm.py and train_light.py for a sample training.

Acknowledgement

This work is built on self-attention GAN and taskgrouping repo.

Citation

@inproceedings{bao2022generative,
    Author = {Bao, Zhipeng and Hebert, Martial and Wang, Yu-Xiong},
    Title = {Generative Modeling for Multi-task Visual Learning},
    Booktitle = {ICML},
    Year = {2022},
}