Finetuning on windows machine

[mscherrmann]

Hi,

I pretrained a mosaic-bert version on a linux server. In a next step I like to finetune the model locally on a windows 11 machine. I tried to run your finetuning script but unfortunately it does nto seem to work on a windows machine.

Another try was to convert the model to the Huggingface format. I tried this approach using the model "mosaicml/mosaic-bert-base" on Huggingface first, after allowing for "BertForSequenceClassification" in the config file. This works, but it seems that the training does not converge, in contrast to the original Huggingface model "bert-base-uncased". Do you have any explanations or recommendations for that?

This is the code:

#%% Imports
from datasets import load_dataset
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification
from torch.utils.data import DataLoader
from torch.optim import AdamW
from transformers import get_scheduler
import torch
import evaluate

#%% Begin by loading some random dataset (i.e. the Yelp Reviews dataset):
dataset = load_dataset("yelp_review_full")

#%% Tokenize
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets = tokenized_datasets.remove_columns(["text"])
tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
tokenized_datasets.set_format("torch")

#%% As we are only interested in the comparison between hf bert and mosaic bert, finetune on a smaller dataset
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)
eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)

#%% Load model & train
model =AutoModelForSequenceClassification.from_pretrained('original-mosaic-bert', trust_remote_code=True, num_labels=5) # Finetune mosaic-bert model, but losses do not converge
# AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5) # This finetunes the original huggingface model with converging losses
optimizer = AdamW(model.parameters(), lr=5e-5)
num_epochs = 3
num_training_steps = num_epochs * len(train_dataloader)
lr_scheduler = get_scheduler(
    name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
)
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
model.to(device)

model.train()
for epoch in range(num_epochs):
    for batch in train_dataloader:
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()

        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()
        print(loss)

[mvpatel2000]

Can you please provide the trace for the original run with our fine tunint script as well?

[mscherrmann]

The problem of your code on a windows machine seems to be that the triton package is not available for on windows. However, do you think it is possible to finetune using torch and transformers?

[mvpatel2000]

It should be possible. Triton is used for faster flash-attn, but it can be replaced with torch ops. I recommend trying it out!

Unfortunately, we would not be able to help you debug your script in this case as it's outside the scope of things we support

[mscherrmann]

Ok, I see.