gLM2 Model Fine-Tuning and Embedding Evaluation

[wconnell]

Objectives

Train fine-tuned versions of gLM2:

• [x] Version 0 (v0): Initial fine-tuning and qualitative eval (completed).
• [x] Version 1 (v1): Fine-tuning with augmented training data accounting for plasmid circularity and long context.
• [x] Evaluate gLM2 plasmid embeddings and various methods

Tasks

Model Fine-Tuning (v1):

• [x] Augment training data to include plasmid circularity and long-context information.
• [x] Fine-tune gLM2 with the augmented dataset.
• [x] Document the fine-tuning process, hyperparameters, and any challenges.

Embedding Extraction

• [x] Ensure embeddings are stored in a standardized format for evaluation pipeline/script

Baseline Methods

• See other workstream

Evaluation Metric

• [x] Define and calculate evaluation metrics such as NMI, ARI
• [x] Apply metrics to assess payload gene clustering and other features.

Analysis

• [x] Compare embeddings from different versions and baseline methods.
• [x] Interpret results to determine the effectiveness of fine-tuning and embedding methods.
• [ ] Polaris Challenge Task: Share OpenPlasmid & evaluation metrics/methods

[wconnell]

I created two evaluations, one for common CDS features and one for common payload genes. In these evaluations we expect the embeddings to cluster by these labels. We take the embeddings, apply Kmeans clustering (number clusters is number of label categories), and then calculate the NMI and ARI between clusters and categories.

Results

Model	Eval	NMI	ARI
gLM2_150-pretrained	CDS-curated-features	0.1950	0.1156
gLM2_150-finetuned	CDS-curated-features	0.2098	0.1203
gLM2_150-finetuned-augment	CDS-curated-features	0.2152	0.1510
plasmidGPT	CDS-curated-features	0.1097	0.0609
one-hot-encoding	CDS-curated-features	0.1806	0.0517
gLM2_150-pretrained	common-entrez-gene	0.2429	0.0802
gLM2_150-finetuned	common-entrez-gene	0.3070	0.1564
gLM2_150-finetuned-augment	common-entrez-gene	0.3072	0.1372
plasmidGPT	common-entrez-gene	0.3024	0.1056
one-hot-encoding	common-entrez-gene	0.2581	0.0952

pretrained PCA

finetuned PCA