[Baselines] [Metrics] Benchmarking embedding alignment with GPTs 🤖 and CLIPs 📎

[Smith42]

Benchmarking embedding alignment with GPTs 🤖 and CLIPs 📎

Contacts: Mike (S), Marc Participants:

Goals and deliverable

1. Developing AstroPT to the point where it can ingest and generate embeddings from Francois' & Liam's galaxy image/spectrum aligned dataset that was used in AstroCLIP, and is now in the *Pile.
   • We can likely generate embeddings autoregressively via "astro-sentences" in a similar way that is done in Bai+2023. The galaxy/spectrum pairs will be aligned in the embedding space through being in proximity with each other at pretraining time.
2. Once we have the embeddings, we would want to define and test a good metric for how well each method does. This could be a simple downstream task like classification, or maybe some linear probe dark magick? This task could then be used as a standard benchmark in the AstroPile project.
3. 🚀 stretch goal 🚀 do we see a scale (in terms of trainable parameters) where AstroPT outperforms AstroCLIP?

Resources needed

We'd probably need a fair amount of GPUs for pretraining 😄, plus some enthusiastic people to help get this working

Rough checklist

• [X] Get AstroPT ready to ingest AstroPile data like AstroCLIP's galaxy image/spectrum pairs
• [ ] Pretrain an AstroPT on a dataset of galaxy image + spectra "visual sentences"
• [ ] Extract embeddings from AstroPT and AstroCLIP models for same data
• [ ] Define a metric to compare the embedding spaces
• [ ] Is there a parameter crossover point where one method suddenly "beats" the other?

[Smith42]

I have some code that can extract embeddings from AstroPT now, and equivalent code is available in AstroCLIP. Next to do is get the code ready to ingest gal image/spectrum pairs from here, eta ~few hours 😄

Some thoughts about embedding space metrics:

• testing on discriminative downstream tasks
• embedding space information density via compressibility or generative tasks
• probably also want a raw PCA as a baseline embedding generator comparison here

[Smith42]

FYI to download Francois' AstroCLIP dataset I use the following code:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="EiffL/AstroCLIP",
    repo_type="dataset",
    local_dir="./",
    local_dir_use_symlinks=False,
    cache_dir="/raid/data/cache",
)

Then the parquets can be loaded with pandas

[Smith42]

Some foundation model benchmark papers that are knocking about in Earth Observation for inspo:

• GEOBench: https://arxiv.org/abs/2306.03831/
• PhilEOBench: https://phileo-bench.github.io/