full parameters training with multi gpu

[wangz1200]

Hi, Thank you very much for disseminating openly your work and code. I want to train the model with full parameters(T5 and Moment head). Could I tran it with multi gpus. Thanks a lot.

[mononitogoswami]

Hi, Thanks for your interest in MOMENT. You can certainly train all the parameters of the model, even in a single GPU, depending on batch size. We trained MOMENT on one A6000 GPU with 49 GiB RAM (without multi GPU training) with a batch size of 1024 time series. More details can be found in the Reproducibility Statement in the paper.

When you download MOMENT, by default, the patch embedding layer and the transformer backbone (T5) are frozen. You can unfreeze them using the following code:

from momentfm import MOMENTPipeline

model = MOMENTPipeline.from_pretrained(
    "AutonLab/MOMENT-1-large", 
    model_kwargs={
        'task_name': 'reconstruction',
        'forecast_horizon': 192,
        'head_dropout': 0.1,
        'weight_decay': 0,
        'freeze_encoder': False, # Freeze the patch embedding layer. True by default.
        'freeze_embedder': False, # Freeze the transformer encoder. True by default.
        'freeze_head': False, # False by default
    },
)

You can certainly train the model on multiple GPUs using PyTorch modules such as DDP. Since we did not have to perform multi GPU training, we do not have code for this currently.

Let us know if you have any more questions, and thanks again for your interest in MOMENT!

[wangz1200]

Thank you for your reply. I appreciate it!

[mononitogoswami]

Let us know how MOMENT works for you! For now, I'll close this issue. Feel free to open it again if you have any questions!

[wangz1200]

Thanks again for your reply. Now,I had trained it with full parameters. And I saved the pretrained models(encoder mdoel and head mdoel). Maybe my dataset is futures data, it is hard to forcast trend. I want to combined with real data researching model ability.