search

brjathu / LART

Code repository for the paper "On the Benefits of 3D Pose and Tracking for Human Action Recognition", (CVPR 2023)
https://github.com/brjathu/LART
249 stars 32 forks source link

CUDA OUT OF MEMORY #21

Closed Lxzzzzzzzzzzz closed 7 months ago

Lxzzzzzzzzzzz commented 7 months ago

Can 4090 run the demo? (lart) lxz@a4061-MS-7E06:~/projects/LART$ python scripts/demo.py video.source="assets/jump.mp4" +half=True [02/26 14:54:37] INFO No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate' acceleratesupport.py:17 [2024-02-26 14:54:38,556][pytorch_lightning.utilities.migration.utils][INFO] - Lightning automatically upgraded your loaded checkpoint from v1.8.1 to v2.2.0.post0. To apply the upgrade to your files permanently, run python -m pytorch_lightning.utilities.upgrade_checkpoint ../../.cache/4DHumans/logs/train/multiruns/hmr2/0/checkpoints/epoch=35-step=1000000.ckpt WARNING: You are using a SMPL model, with only 10 shape coefficients. [2024-02-26 14:54:40,603][phalp.trackers.PHALP][INFO] - Loading Predictor model... [2024-02-26 14:54:40,730][phalp.trackers.PHALP][INFO] - Loading Detection model... [2024-02-26 14:54:44,187][detectron2.checkpoint.detection_checkpoint][INFO] - [DetectionCheckpointer] Loading from https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_vitdet_h/f328730692/model_final_f05665.pkl ... [2024-02-26 14:54:45,620][detectron2.checkpoint.detection_checkpoint][INFO] - [DetectionCheckpointer] Loading from https://dl.fbaipublicfiles.com/detectron2/COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x/139653917/model_final_2d9806.pkl ... [2024-02-26 14:54:45,709][phalp.trackers.PHALP][INFO] - Setting up Visualizer... [2024-02-26 14:54:45,954][phalp.utils.io][INFO] - Number of frames: 171 [2024-02-26 14:54:46,181][phalp.trackers.PHALP][INFO] - Setting up DeepSort... [2024-02-26 14:54:46,181][phalp.trackers.PHALP][INFO] - Saving tracks at : outputs//results/jump Tracking : jump ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% eta : 0:00:00 time elapsed : 0:00:49 [2024-02-26 14:55:36,588][pytorch_lightning.utilities.migration.utils][INFO] - Lightning automatically upgraded your loaded checkpoint from v1.8.1 to v2.2.0.post0. To apply the upgrade to your files permanently, run python -m pytorch_lightning.utilities.upgrade_checkpoint ../../.cache/4DHumans/logs/train/multiruns/hmr2/0/checkpoints/epoch=35-step=1000000.ckpt WARNING: You are using a SMPL model, with only 10 shape coefficients. [2024-02-26 14:55:38,351][main][INFO] - Loading Predictor model... [2024-02-26 14:55:38,651][phalp.trackers.PHALP][INFO] - Loading Detection model... [2024-02-26 14:55:41,734][detectron2.checkpoint.detection_checkpoint][INFO] - [DetectionCheckpointer] Loading from https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_vitdet_h/f328730692/model_final_f05665.pkl ... [2024-02-26 14:55:43,150][detectron2.checkpoint.detection_checkpoint][INFO] - [DetectionCheckpointer] Loading from https://dl.fbaipublicfiles.com/detectron2/COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x/139653917/model_final_2d9806.pkl ... [2024-02-26 14:55:43,309][phalp.trackers.PHALP][INFO] - Setting up Visualizer... [2024-02-26 14:55:44,810][slowfast.visualization.predictor][INFO] - Start loading model weights. [2024-02-26 14:55:44,810][slowfast.utils.checkpoint][INFO] - Loading network weights from /home/lxz/.cache/phalp/ava/mvit.pyth. missing keys: [] unexpected keys: [] [2024-02-26 14:55:45,121][slowfast.visualization.predictor][INFO] - Finish loading model weights Error executing job with overrides: ['video.source=assets/jump.mp4', '+half=True'] Traceback (most recent call last): File "/home/lxz/projects/LART/scripts/demo.py", line 103, in main lart_model.postprocessor.run_lart(pkl_path) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/phalp/visualize/postprocessor.py", line 102, in run_lart final_visuals_dic = self.post_process(final_visuals_dic, save_fast_tracks=self.cfg.post_process.save_fast_tracks, video_pkl_name=video_pkl_name) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/phalp/visualize/postprocessor.py", line 36, in post_process smoothed_fasttrack = self.phalp_tracker.pose_predictor.smooth_tracks(fasttrack, moving_window=True, step=32, window=32) File "/home/lxz/projects/LART/lart/utils/wrapper_phalp.py", line 226, in smooth_tracks fast_track = self.add_slowfast_features(fast_track) File "/home/lxz/projects/LART/lart/utils/wrapper_phalp.py", line 203, in add_slowfastfeatures task = SlowFastWrapper(t_, cfg, list_of_all_frames, midbbox, video_model, center_crop=center_crop) File "/home/lxz/projects/LART/lart/utils/wrapper_pyslowfast.py", line 61, in SlowFastWrapper task = video_model(task) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/visualization/predictor.py", line 110, in call preds, feats = self.model(inputs, bboxes) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/models/video_model_builder.py", line 1239, in forward x, thw = blk(x, thw) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/fairscale/nn/checkpoint/checkpoint_activations.py", line 171, in _checkpointed_forward return original_forward(module, *args, *kwargs) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/models/attention.py", line 547, in forward x_block, thw_shape_new = self.attn(x_norm, thw_shape) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/models/attention.py", line 407, in forward attn = cal_rel_pos_spatial( File "/home/lxz/anaconda3/envs/lart/lib/python3.10/site-packages/slowfast/models/attention.py", line 112, in cal_rel_pos_spatial attn[:, :, sp_idx:, sp_idx:].view(B, -1, q_t, q_h, q_w, k_t, k_h, k_w) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.25 GiB. GPU 0 has a total capacty of 23.62 GiB of which 4.13 GiB is free. Including non-PyTorch memory, this process has 18.56 GiB memory in use. Of the allocated memory 17.68 GiB is allocated by PyTorch, and 428.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

vtn98 commented 7 months ago

Hi @Lxzzzzzzzzzzz , did you find a solution to this?

Lxzzzzzzzzzzz commented 7 months ago

Hi @Lxzzzzzzzzzzz , did you find a solution to this? 换个视频就行，别用那个例子