The official code for "TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (ICLR 2024)". TEMPO is one of the very first open source Time Series Foundation Models for forecasting task v1.0 version.
Thanks for this amazing work. I am interested in reproducing the results of your paper but I am getting torch.cuda.OutOfMemoryError
Currently I am trying to run it on Google Colab with a T4 GPU.
Do you think using one more GPU can solve this problem? If not, what else can I try?
Below is the complete output mesage:
/bin/bash: line 2: fg: no job control
mkdir: cannot create directory ‘logs/TEMPO/loar_revin_100_percent_1_prompt_equal_1/’: File exists
mkdir: cannot create directory ‘logs/TEMPO/loar_revin_100_percent_1_prompt_equal_1/ettm2_pmt1_no_pool_TEMPO_6’: File exists
logs/TEMPO/loar_revin_100_percent_1_prompt_equal_1/ettm2_pmt1_no_pool_TEMPO_6/test_336_96_lr0.001.log
Namespace(model_id='etth1_TEMPO_6_prompt_learn_336_96_100', checkpoints='./lora_revin_6domain_checkpoints_1/', task_name='long_term_forecast', prompt=1, num_nodes=1, seq_len=336, pred_len=96, label_len=168, decay_fac=0.5, learning_rate=0.001, batch_size=256, num_workers=0, train_epochs=10, lradj='type3', patience=5, gpt_layers=6, is_gpt=1, e_layers=3, d_model=768, n_heads=4, d_ff=768, dropout=0.3, enc_in=7, c_out=1, patch_size=16, kernel_size=25, loss_func='mse', pretrain=1, freeze=1, model='TEMPO', stride=8, max_len=-1, hid_dim=16, tmax=20, itr=1, cos=1, equal=1, pool=False, no_stl_loss=False, stl_weight=0.001, config_path='./configs/multiple_datasets.yml', datasets='ETTm1,ETTh2,ETTm2,electricity,traffic,weather', target_data='ETTh1', use_token=0, electri_multiplier=1, traffic_multiplier=1, embed='timeF')
['ETTm1', 'ETTh2', 'ETTm2', 'electricity', 'traffic', 'weather']
ETTm1
dataset: ett_m
train 238903
val 79975
ETTh2
dataset: ett_h
self.enc_in = 7
self.data_x = (8640, 7)
train 57463
self.enc_in = 7
self.data_x = (3216, 7)
val 19495
ETTm2
dataset: ett_m
train 238903
val 79975
electricity
dataset: custom
train 5771901
val 814377
traffic
dataset: custom
train 10213838
val 1431782
weather
dataset: custom
train 765576
val 108675
ETTm1
dataset: ett_m
train 238903
ETTh2
dataset: ett_h
self.enc_in = 7
self.data_x = (8640, 7)
train 57463
ETTm2
dataset: ett_m
train 238903
electricity
dataset: custom
train 5771901
traffic
dataset: custom
train 10213838
weather
dataset: custom
train 765576
Way1 1251978
self.enc_in = 7
self.data_x = (3216, 7)
test 19495
trainable params: 308736 || all params: 82207488 || trainable%: 0.38
0% 0/4891 [00:01<?, ?it/s]
Traceback (most recent call last):
File "/content/TEMPO/main_multi_6domain_release.py", line 292, in
outputs, loss_local = model(batch_x, ii, seq_trend, seq_seasonal, seq_resid) #+ model(seq_seasonal, ii) + model(seq_resid, ii)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, *kwargs)
File "/content/TEMPO/models/TEMPO.py", line 446, in forward
x = self.gpt2_trend(inputs_embeds =x_all).last_hidden_state
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/peft/peft_model.py", line 642, in forward
return self.get_base_model()(*args, *kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 1116, in forward
outputs = block(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, *kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 651, in forward
feed_forward_hidden_states = self.mlp(hidden_states)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, *kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 572, in forward
hidden_states = self.act(hidden_states)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, *kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/activations.py", line 56, in forward
return 0.5 input (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) (input + 0.044715 * torch.pow(input, 3.0))))
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 460.00 MiB. GPU
Dear authors,
Thanks for this amazing work. I am interested in reproducing the results of your paper but I am getting
torch.cuda.OutOfMemoryError
Currently I am trying to run it on Google Colab with a T4 GPU. Do you think using one more GPU can solve this problem? If not, what else can I try? Below is the complete output mesage:
/bin/bash: line 2: fg: no job control mkdir: cannot create directory ‘logs/TEMPO/loar_revin_100_percent_1_prompt_equal_1/’: File exists mkdir: cannot create directory ‘logs/TEMPO/loar_revin_100_percent_1_prompt_equal_1/ettm2_pmt1_no_pool_TEMPO_6’: File exists logs/TEMPO/loar_revin_100_percent_1_prompt_equal_1/ettm2_pmt1_no_pool_TEMPO_6/test_336_96_lr0.001.log Namespace(model_id='etth1_TEMPO_6_prompt_learn_336_96_100', checkpoints='./lora_revin_6domain_checkpoints_1/', task_name='long_term_forecast', prompt=1, num_nodes=1, seq_len=336, pred_len=96, label_len=168, decay_fac=0.5, learning_rate=0.001, batch_size=256, num_workers=0, train_epochs=10, lradj='type3', patience=5, gpt_layers=6, is_gpt=1, e_layers=3, d_model=768, n_heads=4, d_ff=768, dropout=0.3, enc_in=7, c_out=1, patch_size=16, kernel_size=25, loss_func='mse', pretrain=1, freeze=1, model='TEMPO', stride=8, max_len=-1, hid_dim=16, tmax=20, itr=1, cos=1, equal=1, pool=False, no_stl_loss=False, stl_weight=0.001, config_path='./configs/multiple_datasets.yml', datasets='ETTm1,ETTh2,ETTm2,electricity,traffic,weather', target_data='ETTh1', use_token=0, electri_multiplier=1, traffic_multiplier=1, embed='timeF') ['ETTm1', 'ETTh2', 'ETTm2', 'electricity', 'traffic', 'weather'] ETTm1 dataset: ett_m train 238903 val 79975 ETTh2 dataset: ett_h self.enc_in = 7 self.data_x = (8640, 7) train 57463 self.enc_in = 7 self.data_x = (3216, 7) val 19495 ETTm2 dataset: ett_m train 238903 val 79975 electricity dataset: custom train 5771901 val 814377 traffic dataset: custom train 10213838 val 1431782 weather dataset: custom train 765576 val 108675 ETTm1 dataset: ett_m train 238903 ETTh2 dataset: ett_h self.enc_in = 7 self.data_x = (8640, 7) train 57463 ETTm2 dataset: ett_m train 238903 electricity dataset: custom train 5771901 traffic dataset: custom train 10213838 weather dataset: custom train 765576 Way1 1251978 self.enc_in = 7 self.data_x = (3216, 7) test 19495 trainable params: 308736 || all params: 82207488 || trainable%: 0.38 0% 0/4891 [00:01<?, ?it/s] Traceback (most recent call last): File "/content/TEMPO/main_multi_6domain_release.py", line 292, in
outputs, loss_local = model(batch_x, ii, seq_trend, seq_seasonal, seq_resid) #+ model(seq_seasonal, ii) + model(seq_resid, ii)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, *kwargs)
File "/content/TEMPO/models/TEMPO.py", line 446, in forward
x = self.gpt2_trend(inputs_embeds =x_all).last_hidden_state
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/peft/peft_model.py", line 642, in forward
return self.get_base_model()(*args, *kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 1116, in forward
outputs = block(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, *kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 651, in forward
feed_forward_hidden_states = self.mlp(hidden_states)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, *kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 572, in forward
hidden_states = self.act(hidden_states)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, *kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/activations.py", line 56, in forward
return 0.5 input (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) (input + 0.044715 * torch.pow(input, 3.0))))
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 460.00 MiB. GPU