calgarylaoxu
commented
3 months ago you're getting a CUDA out-of-memory error, use a GPU server or switch to Google Colab instead will solve the issue
Open jianlao8 opened 4 months ago
calgarylaoxu
commented
3 months ago you're getting a CUDA out-of-memory error, use a GPU server or switch to Google Colab instead will solve the issue
Hi, I ran the script and then get the following error. I follow the suggestion to set the environment variable "export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" at the terminal session, but it got the same error.
return self.df["dividend"][-1] Progress |███████------------------------------------------------| 13.04% [Elapsed: 0:00:21 ETA: 0:02:24] Portfolio Val: 169,231.522024-02-25 19:00:23,447: root: ERROR: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 3.94 GiB of which 21.06 MiB is free. Including non-PyTorch memory, this process has 3.88 GiB memory in use. Of the allocated memory 3.41 GiB is allocated by PyTorch, and 13.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-02-25 19:00:23,451: root: ERROR: Traceback (most recent call last): File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 856, in run self._run_trading_session() File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 807, in _run_trading_session self._on_trading_iteration() File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 276, in func_output result = func_input(self, *args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 300, in func_output result = func_input(self, *args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 87, in func_output return func_input(self, args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 378, in _on_trading_iteration on_trading_iteration() File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/tools/decorators.py", line 62, in func_output frame, result = call_function_get_frame(func_input, *args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/tools/decorators.py", line 30, in call_function_get_frame result = func(*args, *kwargs) File "/home/kenlao/python/Code Trading/tradingbot.py", line 50, in on_trading_iteration probability, sentiment = self.get_sentiment() File "/home/kenlao/python/Code Trading/tradingbot.py", line 45, in get_sentiment probability, sentiment = estimate_sentiment(news) File "/home/kenlao/python/Code Trading/finbert_utils.py", line 14, in estimate_sentiment result = model(tokens["input_ids"], attention_mask=tokens["attention_mask"])[ File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1564, in forward outputs = self.bert( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1013, in forward encoder_outputs = self.encoder( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 607, in forward layer_outputs = layer_module( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward self_attention_outputs = self.attention( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 427, in forward self_outputs = self.self( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, **kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 325, in forward attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2)) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 3.94 GiB of which 21.06 MiB is free. Including non-PyTorch memory, this process has 3.88 GiB memory in use. Of the allocated memory 3.41 GiB is allocated by PyTorch, and 13.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Exception in thread MLTrader: Traceback (most recent call last): File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 856, in run self._run_trading_session() File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 807, in _run_trading_session self._on_trading_iteration() File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 276, in func_output result = func_input(self, *args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 300, in func_output result = func_input(self, *args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 87, in func_output return func_input(self, args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 378, in _on_trading_iteration on_trading_iteration() File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/tools/decorators.py", line 62, in func_output frame, result = call_function_get_frame(func_input, *args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/tools/decorators.py", line 30, in call_function_get_frame result = func(*args, *kwargs) File "/home/kenlao/python/Code Trading/tradingbot.py", line 50, in on_trading_iteration probability, sentiment = self.get_sentiment() File "/home/kenlao/python/Code Trading/tradingbot.py", line 45, in get_sentiment probability, sentiment = estimate_sentiment(news) File "/home/kenlao/python/Code Trading/finbert_utils.py", line 14, in estimate_sentiment result = model(tokens["input_ids"], attention_mask=tokens["attention_mask"])[ File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1564, in forward outputs = self.bert( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1013, in forward encoder_outputs = self.encoder( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 607, in forward layer_outputs = layer_module( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward self_attention_outputs = self.attention( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 427, in forward self_outputs = self.self( File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, **kwargs) File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 325, in forward attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2)) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 3.94 GiB of which 21.06 MiB is free. Including non-PyTorch memory, this process has 3.88 GiB memory in use. Of the allocated memory 3.41 GiB is allocated by PyTorch, and 13.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/home/kenlao/miniconda3/envs/py310/lib/python3.10/site-packages/lumibot/strategies/strategy_executor.py", line 869, in run raise RuntimeError("Exception encountered, stopping BackTest.") from e RuntimeError: Exception encountered, stopping BackTest.
Creating trades plot...
Creating indicators plot...
Creating tearsheet... (py310) (base) kenlao@ken