search

rizar / systematic-generalization-sqoop

Code for "Systematic Generalization: What Is Required and Can It Be Learned"
Other
37 stars 11 forks source link

readme

Systematic Generalization: What Is Required and Can It Be Learned

The code used for the experiments in the paper.

Setup

Clone the repo

git clone https://github.com/rizar/systematic-generalization-sqoop.git
cd systematic-generalization-sqoop
export NMN=$PWD

Setup the environment using conda (recommended) and install this as a package in development mode

conda env create -f environment.yml
conda activate sysgen
pip install -e .

if you don't use conda, you can do pip install --user -r requirements.txt

Download all versions of SQOOP dataset from here and unpack it. Let $DATA be the location of the data on your system.

Running Experiments

In the examples below we are using SQOOP with #rhs/lhs=1, other versions can be used by changing --data_dir.

FiLM

scripts/train/film_flatqa.sh --data_dir $DATA/sqoop-variety_1-repeats_30000 --checkpoint_path model.pt\
 --num_iterations 200000

MAC

scripts/train/mac_flatqa.sh --data_dir $DATA/sqoop-variety_1-repeats_30000 --checkpoint_path model.pt\
 --num_iterations 100000

Conv+LSTM

scripts/train/convlstm_flatqa.sh --data_dir $DATA/sqoop-variety_1-repeats_30000 --checkpoint_path model.pt\
 --num_iterations 200000

RelNet

scripts/train/rel_flatqa.sh --data_dir $DATA/sqoop-variety_1-repeats_30000 --checkpoint_path model.pt\
 --num_iterations 500000

NMN-Tree, NMN-Chain, NMN-Chain-Shortcut

scripts/train/shnmn_flatqa.sh --data_dir $DATA/sqoop-variety_1-repeats_30000\
 --hard_code_tau --tau_init tree --hard_code_alpha --alpha_init correct\
 --num_iterations 50000 --checkpoint_path model.pt

For a different layout use --tau_init=chain or --tau_init=chain_shortcut. For a different module, use --use_module=find, the default is Residual. Make sure to train for 200000 iterations if you use Find.

Stochastic-N2NMN

 scripts/train/shnmn_flatqa.sh --data_dir $DATA/sqoop-variety_1-repeats_30000\
  --shnmn_type hard --model_bernoulli 0.5 --hard_code_alpha --alpha_init=correct\
  --num_iterations 200000 --checkpoint_path model.pt

--model_bernoulli is the initial probability of the model being a tree.

Attention-N2NMN

scripts/train/shnmn_flatqa.sh --data_dir $DATA/sqoop-variety_1-repeats_30000\
 --hard_code_tau --tau_init tree --use_module=find --num_iterations 200000\
 --checkpoint_path model.pt

Citation

Bahdanau, D., Murty, S., Noukhovitch, M., Nguyen, T. H., de Vries, H., & Courville, A. (2018). Systematic Generalization: What Is Required and Can It Be Learned?. ICLR 2019

(the first two authors contributed equally)

@inproceedings{sysgen2019,
    title = {Systematic Generalization: What Is Required and Can It Be Learned?},
    booktitle = {International Conference on Learning Representations},
    author = {Bahdanau, Dzmitry and Murty, Shikhar and Noukhovitch, Michael and Nguyen, Thien Huu and Vries, Harm de and Courville, Aaron},
    year = {2019},
    url = {https://openreview.net/forum?id=HkezXnA9YX},
}

Acknowledgements.

This code is based on the reference implementation for "FiLM: Visual Reasoning with a General Conditioning Layer" by Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, Aaron Courville (AAAI 2018) which was based on the reference implementation for "Inferring and Executing Programs for Visual Reasoning" by Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Fei-Fei Li, Larry Zitnick, Ross Girshick (ICCV 2017)