Multi-Task Baselines

[MuMuJun97]

SOON

• https://eval.ai/web/challenges/challenge-page/1275/leaderboard/3235
• DUET:
• GBE (raw baseline)

---

REVERIE

---

R2R

---

CVDN

• https://arxiv.org/pdf/2003.00443.pdf

CVDN | Not use R2R metrics

• https://arxiv.org/pdf/2110.13309.pdf

[MuMuJun97]

Ours

Single-Task on Val_unseen, Model bloom-560M

• R2R 8GPU*8 batch size, Epoch 30, lr 5e-05

  sr: 49.91, oracle_sr: 59.18, spl: 43.91, nDTW: 57.14, SDTW: 41.53, CLS: 55.90
  action_steps: 6.15, steps: 6.15, lengths: 12.14, nav_error: 5.46, oracle_error: 3.56

• R2R 8GPU*8 batch size, Epoch 20, lr 3e-05

  sr: 47.75, oracle_sr: 57.23, spl: 40.95, nDTW: 54.41, SDTW: 39.09, CLS: 53.31
  action_steps: 6.67, steps: 6.67, lengths: 13.14, nav_error: 5.58, oracle_error: 3.59

• REVERIE 8GPU*8 batch size, Epoch 20, lr 3e-05

  sr: 27.38, oracle_sr: 38.55, spl: 19.81, nDTW: 35.15, SDTW: 19.42, CLS: 36.76
  action_steps: 8.07, steps: 8.07, lengths: 15.78, nav_error: 8.05, oracle_error: 4.97

• CVDN 8GPU*8 batch size, Epoch 20, lr 3e-05

  sr: 32.41, oracle_sr: 47.46, spl: 24.89, nDTW: 39.48, SDTW: 23.09, CLS: 39.88
  action_steps: 9.10, steps: 9.10, lengths: 17.84, nav_error: 7.42, oracle_error: 5.08

• SOON 8GPU*8 batch size, Epoch 20, lr 3e-05

  sr: 17.61, oracle_sr: 22.10, spl: 15.92, nDTW: 30.46, SDTW: 14.89, CLS: 36.76
  action_steps: 11.62, steps: 11.62, lengths: 21.31, nav_error: 13.59, oracle_error: 9.10

  Multi-Tasks on Val_unseen, Model bloom-560M

• 8GPU*8 batch size, Epoch 20, lr 3e-05

  # R2R
  sr: 42.78, oracle_sr: 53.70, spl: 35.54, nDTW: 49.94, SDTW: 33.72, CLS: 49.23
  action_steps: 7.26, steps: 7.26, lengths: 14.39, nav_error: 5.90, oracle_error: 3.83
  # REVERIE
  sr: 26.57, oracle_sr: 37.23, spl: 19.48, nDTW: 35.29, SDTW: 18.78, CLS: 36.56
  action_steps: 8.54, steps: 8.54, lengths: 17.00, nav_error: 7.69, oracle_error: 4.90
  # SOON 
  sr: 23.81, oracle_sr: 27.61, spl: 22.17, nDTW: 37.30, SDTW: 20.71, CLS: 41.97
  action_steps: 10.27, steps: 10.27, lengths: 19.34, nav_error: 13.21, oracle_error: 8.76
  # CVDN
  sr: 32.48, oracle_sr: 49.17, spl: 25.93, nDTW: 42.02, SDTW: 24.41, CLS: 43.81
  action_steps: 7.49, steps: 7.49, lengths: 15.08, nav_error: 7.97, oracle_error: 5.10