search

time-series-foundation-models / lag-llama

Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting
Apache License 2.0
1.08k stars 121 forks source link

How to finetune on Custom Loss function #38

Open gopal86 opened 3 months ago

gopal86 commented 3 months ago

Hi, I wanted to fine tune the model on my own dataset however with my own custom loss. Could you give an example on how to do that? Thanks

ashok-arjun commented 2 months ago

Hi @gopal86. What kind of loss functions would you like to finetune it with?

jinxzed commented 2 months ago

Hi, I would like to train on weighted mape loss function. Kindly share an example

ashok-arjun commented 2 months ago

You would have to replace the last layer with an appropriate layer for the loss function, and then finetune the model.

You could try finetuning just the last layer, or the entire model.

gopal86 commented 2 months ago

Hi @gopal86. What kind of loss functions would you like to finetune it with?

I have certain constraints which I would like to impose. Could you give an example on how to fine-tune any custom loss function?

ashok-arjun commented 2 months ago

When you finetune on any other function, you will have to change the last layer to whatever is appropriate.

For instance, if you would like to perform classification, you could put a linear layer with softmax at the end after the embeddings, and then use the cross entropy loss.

If you would like to perform regression, you could put a linear layer at the end with a single output, and use the MSE loss.

When training, you could train the full model or just the last layer.

An idea: To start with, you could get the embeddings from the pretrained model, and then train a simple statistical classifier/regressor on the embeddings.

jinxzed commented 1 month ago

Hi,

On Wed, 17 Apr 2024 at 02:07, Arjun Ashok @.***> wrote:

When you finetune on any other function, you will have to change the last layer to whatever is appropriate.

For instance, if you would like to perform classification, you could put a linear layer with softmax at the end after the embeddings, and then use the cross entropy loss.

If you would like to perform regression, you could put a linear layer at the end with a single output, and use the MSE loss.

When training, you could train the full model or just the last layer.

An idea: To start with, you could get the embeddings from the pretrained model, and then train a simple statistical classifier/regressor on the embeddings.

— Reply to this email directly, view it on GitHub https://github.com/time-series-foundation-models/lag-llama/issues/38#issuecomment-2059883779, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOTJPURAHHWCKUOCHCPS4XTY5WDZ7AVCNFSM6AAAAABFJAR3MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJZHA4DGNZXHE . You are receiving this because you commented.Message ID: @.***>

jinxzed commented 1 month ago

Hi, Can you share an example on how to get the pretrained model embeddings for the lag-llama model.

On Wed, 17 Apr 2024 at 02:07, Arjun Ashok @.***> wrote:

When you finetune on any other function, you will have to change the last layer to whatever is appropriate.

For instance, if you would like to perform classification, you could put a linear layer with softmax at the end after the embeddings, and then use the cross entropy loss.

If you would like to perform regression, you could put a linear layer at the end with a single output, and use the MSE loss.

When training, you could train the full model or just the last layer.

An idea: To start with, you could get the embeddings from the pretrained model, and then train a simple statistical classifier/regressor on the embeddings.

— Reply to this email directly, view it on GitHub https://github.com/time-series-foundation-models/lag-llama/issues/38#issuecomment-2059883779, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOTJPURAHHWCKUOCHCPS4XTY5WDZ7AVCNFSM6AAAAABFJAR3MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJZHA4DGNZXHE . You are receiving this because you commented.Message ID: @.***>