search

mkocabas / VIBE

Official implementation of CVPR2020 paper "VIBE: Video Inference for Human Body Pose and Shape Estimation"
https://arxiv.org/abs/1912.05656
Other
2.86k stars 550 forks source link

VIBE is running, but it only uses the CPU (not the GPU) and then crashes as it runs out of memory (64GB..) #225

Open richardmking opened 2 years ago

richardmking commented 2 years ago

Hello, I have followed the instruction from Carlos Barreto for installing the VIBE code on a windows PC, and got it to run ... But, it only gets as far a starting the multi person tracking section using the CPU, not the GPU then it starts to consume the RAM memory until it uses it all up then crashes

Can you give me any clue as to what I have done wrong or how to fix it?

Thanks đź‘Ť Richard

Configuration of my PC Windows 10 CPU = i7 with 64GB of RAM GPU 0 = Intel UHD Graphics 750 GPU 1 = NVIDIA GeForce RTX 3060 Ti

LOG from Command Prompt: This is triggered when the CPU runs out of the 64GB of memory, (then it dies a horrible death!) NOTE: I put in 1 debug print statement to check what the variable device was set to, it is cuda NOT cpu...

(venv_vibe) D:\MOCAP\VIBE>python demo_alter.py --vid_file sample_video.mp4 --output_folder output/ --display RMK DEBUG demo_alter.py device = cuda Running "ffmpeg -i sample_video.mp4 -f image2 -v error /tmp\sample_video_mp4/%06d.png" Images saved to "/tmp\sample_video_mp4" Input video number of frames 300 Running Multi-Person-Tracker 1%|â–Š | 1/100 [10:13<16:52:29, 613.64s/it]Traceback (most recent call last): File "demo_alter.py", line 408, in main(args) File "demo_alter.py", line 96, in main tracking_results = mot(image_folder) File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\multi_person_tracker\mpt.py", line 284, in call trackers = self.run_tracker(dataloader) File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\torch\autograd\grad_mode.py", line 49, in decorate_no_grad return func(*args, *kwargs) File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\multi_person_tracker\mpt.py", line 78, in run_tracker predictions = self.detector(batch) File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\torch\autograd\grad_mode.py", line 49, in decorate_no_grad return func(args, **kwargs) File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\yolov3\yolo.py", line 70, in call detections = non_max_suppression(detections, self.conf_thres, self.nms_thres) File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\yolov3\utils\utils.py", line 260, in non_max_suppression detections = detections[~invalid] RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 51016 bytes. Buy new RAM!

Exception ignored in: <function tqdm.del at 0x0000020942B6D708> Traceback (most recent call last): File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\tqdm_tqdm.py", line 931, in del self.close() File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\tqdm_tqdm.py", line 1133, in close self._decr_instances(self) File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\tqdm_tqdm.py", line 496, in _decr_instances cls.monitor.exit() File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\site-packages\tqdm_monitor.py", line 52, in exit self.join() File "C:\Users\richa\anaconda3\envs\venv_vibe\lib\threading.py", line 1041, in join raise RuntimeError("cannot join current thread") RuntimeError: cannot join current thread

CyWang-Jade commented 2 years ago

I have the same problem, have you solved it? Can you tell me the solution?Thank you very much.

richardmking commented 2 years ago

Hi Cy,

I’m afraid that I never managed to find the reason for the issue. I didn’t get any reply or help from github so I have had to abandon it.

This was my first attempt to use a machine learning type program and so don’t really know where to start to try to fix it by myself..

I may try to come back to it at a future date, but I have put it to one side at the moment. I hope you have better luck than me.

Regards

Richard

From: CyWang @.> Sent: 22 February 2022 05:37 To: mkocabas/VIBE @.> Cc: richardmking @.>; Author @.> Subject: Re: [mkocabas/VIBE] VIBE is running, but it only uses the CPU (not the GPU) and then crashes as it runs out of memory (64GB..) (Issue #225)

I have the same problem, have you solved it? Can you tell me the solution?Thank you very much.

— Reply to this email directly, view it on GitHubhttps://github.com/mkocabas/VIBE/issues/225#issuecomment-1047442690, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AWR43FSQMFCOG6D6USWODQLU4MOJHANCNFSM5IMRFXCA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.Message ID: @.**@.>>

lakpa-tamang9 commented 2 years ago

The main reason your program is not utilizing GPU is that the torch is not enabled with cuda toolkit. While installing torch and torchvision use conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch . As you are creating a new virtual environment for the project, torch should be installed with cuda toolkit. Note: The cudatoolkit version is according to the version of your installed cuda. You can check your version using nvidia-smi command in linux.

tscftw commented 2 years ago

The main reason your program is not utilizing GPU is that the torch is not enabled with cuda toolkit. While installing torch and torchvision use conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch. As you are creating a new virtual environment for the project, torch should be installed with cuda toolkit. Note: The cudatoolkit version is according to the version of your installed cuda. You can check your version using nvidia-smi command in linux.

I tried this fix on both windows and linux and it didn't work for me, when running the sample video my pc takes way too long and my gpu is still not being utilized, any help?

fursink commented 1 year ago

conda create -n venv_vibe python=3.7 conda activate venv_vibe conda install pytorch torchvision torchaudio pytorch-cuda=11.6 cudnn -c pytorch -c nvidia conda install -c anaconda git install_conda.bat prepare_data.bat

--------install_conda.bat------------

pip install mkl intel-openmp pip install numpy==1.17.5 pip install git+https://github.com/giacaglia/pytube.git --upgrade pip install -r requirements_alter.txt

--------requirements_alter.txt------------ tqdm yacs h5py numpy scipy numba smplx gdown PyYAML joblib pillow trimesh pyrender progress filterpy matplotlib tensorflow-gpu tensorboard scikit-image scikit-video opencv-python llvmlite pyglet==1.5.27 git+https://github.com/mattloper/chumpy.git git+https://github.com/mkocabas/yolov3-pytorch.git git+https://github.com/mkocabas/multi-person-tracker.git

edit file and reinstall Work on 3080ti