Problems on fine-tuning on my own data

time-series-foundation-models / lag-llama

Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting

Apache License 2.0

1.09k stars 121 forks source link

Problems on fine-tuning on my own data #29

Open FuZixin opened 3 months ago

FuZixin commented 3 months ago

I'm trying to fine-tune the model using my own data. When using the fine-tuned model to make predictions, I find that lag-llama gives all zero value predictions, which does not reproduce the fine-tuning effect in the demo. I followed the following steps:

Call _from_longdataframe to convert the data from the dataframe format to the standard format.
The data contains 7000 second-level data points. I use the first 4000 points for fine-tuning training, and the remaining 3000 data points as model input. The task is to predict the last 60 data points.
The prediction result is shown in the following figure. The result provided by lag-llama is 60 zero values.

What is the cause of this, or Which step did I do wrong?

ashok-arjun commented 3 months ago

Hi @FuZixin, thanks for the detailed description of the issue.

Can you share a notebook with the data if possible? If the data cannot be shared, can you please share images of the data plotted? It seems to me from the attached image that the data itself is constant across time (at around -6).
Can you please let me know the zero-shot performance / forecasts of the model?

Thanks, Arjun

FuZixin commented 3 months ago

Thank you for your reply:

I've been able to run non-zero predictions by modifying _nonnegative_predsamples=False in the fine-tuning code; I'm speculating that it may be due to the fact that I'm using samples with all negative values?
Sorry I can't provide the dataset directly, here's a sample of the data, I hope it helps: our fine-tuning training set is a time series with 4 thousand data points, with a 1 second acquisition interval, and with 3-bit precision.

Q: However, the predictions based on the fine-tuned model show a large offset from the true value, we are currently working on this issue. 20240321-173711(WeLinkPC) If you have any comments or opinions, please feel free to communicate with us :)

ashok-arjun commented 2 months ago

I see, that's very useful to know.

We have updated the repo with best practices for finetuning. Maybe that could help.

simona-0 commented 1 day ago

Hello @FuZixin , have you maybe set nonnegative_pred_samples=True when you called LagLlamaEstimator? By setting this True you only allow the model to give positive predictions.