Original PyTorch implementation of ORCA proposed in the paper "Cross-Modal Fine-Tuning: Align then Refine". ORCA is developed for effectively solving ML problems in diverse modalities using large-scale pretrained transformers. It adapts to a target task via an align-then-refine workflow: given the target input, ORCA first learns an embedding network that aligns the embedded feature distribution with the pretraining modality. The pretrained model is then fine-tuned on the embedded data to exploit the knowledge shared across modalities.
This repo specifically supports
The Docker image needed for each task can be found in the configuration files under the ./src/configs
directory. Then, run ./src/startup-hook.sh
to install the dependencies.
./src/datasets
python3 ./src/main.py --config ./src/configs/task.yaml
Place the corresponding implementation in ./src/embedders.py
and complete the get_tgt_model
function.
./src/data_loaders.py
and complete the get_data
function in ./src/task_configs.py
../src/utils.py
and complete the get_metric
function in ./src/task_configs.py
.get_config
function in ./src/task_configs.py
../src/configs
.If you find this project helpful, please consider citing our paper:
@inproceedings{shen2023orca,
author = {Shen, Junhong and Li, Liam and Dery, Lucio M. and Staten, Corey and Khodak, Mikhail and Neubig, Graham and Talwalkar, Ameet},
title = {Cross-Modal Fine-Tuning: Align then Refine},
publisher = {ICML},
year = {2023},
url = {https://arxiv.org/abs/2302.05738}
}
Thanks!