OA Long Context Developer Meeting

[andreaskoepf]

When & Where

• Time: Sunday, July 9, 2023: 7pm UTC (9 pm CEST, 12 pm PDT)
• Place: General Voice Channel of the OA discord (inivite)

Long-context Techniques

• ALiBi
• BPT
• Landmarks
• Gist Tokens
• LongNet
• Focus Transformer
• Position interpolation, SuperHOT, NTK-Aware Scaled RoPE

We would like to hear your opinion:

• did you already evaluate one of the listed techniques or do you plan to work on one of them?
• if you read one or multiple of the papers, what do you think regarding:
  • main advantages/disadvantages
  • required training compute
  • inference implementation complexity

Evaluation

In order to properly compare the different long-context techniques we should agree on some evaluation metrics to use.

Candidate metrics:

• Perplexity: we should agree on a base-model, dataset and context lengths for evaluation (ideally using the same test-split).
• Pass-key retrieval accuracy: described in the Landmarks paper (page 10)
• max context possible on 80 GB GPU
• token/s on A100

Ideally we would use the same evaluation code or at least if the evaluation code would be made available open-source to repeat the experiments.