Finetune problem

[lukc12138]

I am trying to use the command you provided to finetune on my own dataset(15000 rows ,almost as big as ETTh1) and I found a serious problem that the result goes much worse after finetuning. The results are below. I don't know how to fix this and why this happened. Maybe you could help me? Thanks a lot. This is the zero-shot result using pretrained model.

This is the zero-shot result after finetuning.

[gorold]

@lukc12138 could you edit the bug report according to the template such that it is reproducible? Not too sure what is going on from just the plots.

[lukc12138]

@lukc12138 could you edit the bug report according to the template such that it is reproducible? Not too sure what is going on from just the plots.

Thanks for the reply. I found why this happened. It is because I turned the .ckpt file into .safetensors with the code below:

import torch
import os
import safetensors
from typing import Dict, List, Optional, Set, Tuple
from safetensors.torch import _find_shared_tensors, _is_complete, load_file, save_file

loaded = torch.load('./epoch=2-step=60.ckpt')
if "state_dict" in loaded:
    loaded = loaded["state_dict"]
# print(type(loaded))
safetensors.torch.save_file(loaded, 'model.safetensors')

And I use the model.safetensors with the config.json below:

{
  "attn_dropout_p": 0.0,
  "d_model": 384,
  "distr_output": {
    "_target_": "uni2ts.distribution.mixture.MixtureOutput",
    "components": [
      {
        "_target_": "uni2ts.distribution.student_t.StudentTOutput"
      },
      {
        "_target_": "uni2ts.distribution.normal.NormalFixedScaleOutput",
        "scale": 0.001
      },
      {
        "_target_": "uni2ts.distribution.negative_binomial.NegativeBinomialOutput"
      },
      {
        "_target_": "uni2ts.distribution.log_normal.LogNormalOutput"
      }
    ]
  },
  "dropout_p": 0.2,
  "max_seq_len": 512,
  "num_layers": 6,
  "patch_sizes": [
    8,
    16,
    32,
    64,
    128
  ],
  "scaling": true
}

Is there anything wrong?Or is there a way to save the model as safetensors when finetuning?

[gorold]

No need to convert the model to safetensors, just use a config file like this: https://github.com/SalesforceAIResearch/uni2ts/blob/7f9a34ec5b6a55d831baa54aca77e80044706277/cli/conf/eval/model/moirai_lightning_ckpt.yaml

[lukc12138]

No need to convert the model to safetensors, just use a config file like this: https://github.com/SalesforceAIResearch/uni2ts/blob/7f9a34ec5b6a55d831baa54aca77e80044706277/cli/conf/eval/model/moirai_lightning_ckpt.yaml

Thanks for your help. It worked now.