search

YuShen0118 / Garment_Generation

64 stars 11 forks source link

Gradient flow computing cause GPU out of memories #1

Open Hoang1802 opened 4 years ago

Hoang1802 commented 4 years ago

Hi guys, first i want to thank you for public code and dataset in spec garment generation.

I try to run this code but something went wrong. When i run shell file scripts/batch_process_singlethread.sh, i meet this error:

Traceback (most recent call last):
  File "prepare_data.py", line 92, in <module>
    singleObjWithId(sys.argv[1]+'/', sys.argv[2]+'/', int(sys.argv[3]),int(sys.argv[4]),int(sys.argv[5]))
  File "prepare_data.py", line 86, in singleObjWithId
    dispMap, dpMap, legalMap = compute_maps(objFileName, bodyFileName)
  File "/home/pham.minh.hoang/Work/ducvn/Garment_Generation/util/compute_maps.py", line 81, in compute_maps
    x_i_final, _ = gradient_flow( SamplesLoss("sinkhorn", p=2, blur=.01), x_i_new, Y_j, lr=0.5, do_a=False )
  File "/home/pham.minh.hoang/Work/ducvn/Garment_Generation/util/compute_maps.py", line 40, in gradient_flow
    L_αβ = loss(a/a.sum(), x_i, b/b.sum(), y_j)
  File "/home/pham.minh.hoang/Work/ducvn/3d_fashion/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/pham.minh.hoang/Work/ducvn/3d_fashion/lib/python3.6/site-packages/geomloss/samples_loss.py", line 237, in forward
    verbose = self.verbose )
  File "/home/pham.minh.hoang/Work/ducvn/3d_fashion/lib/python3.6/site-packages/geomloss/sinkhorn_samples.py", line 44, in sinkhorn_tensorized
    C_xx, C_yy = ( cost( x, x.detach()), cost( y, y.detach()) ) if debias else (None, None)  # (B,N,N), (B,M,M)
  File "/home/pham.minh.hoang/Work/ducvn/3d_fashion/lib/python3.6/site-packages/geomloss/sinkhorn_samples.py", line 28, in <lambda>
    2 : (lambda x,y : squared_distances(x,y) / 2),
  File "/home/pham.minh.hoang/Work/ducvn/3d_fashion/lib/python3.6/site-packages/geomloss/utils.py", line 38, in squared_distances
    D_xy = torch.matmul( x, y.permute(0,2,1) )  # (B,N,D) @ (B,D,M) = (B,N,M)
RuntimeError: CUDA out of memory. Tried to allocate 16.00 GiB (GPU 0; 10.91 GiB total capacity; 4.81 MiB already allocated; 9.70 GiB free; 6.00 MiB reserved in total by PyTorch)

I checked gpu memories by cmd nvidia-smi, and this file consumes less memory than i thought:

    0   N/A  N/A     19122      C   python                           1031MiB

Then please help me solve this problem. Thank you !!!

My system:

P/S: can you add requirements.txt to control version of package in repo

YuShen0118 commented 3 years ago

Sorry about replying so late, I just notice this question. I made a mistake about the code version. The prepare_data.py is a later method we used than the one in our ECCV2020 paper, so you don't have to use it now. Instead, you can use reconstruction/new_enc.cpp (you may need a C++ compiler to compile it). See the updated code and README file. Your environment is similar to mine, which should be good. And thanks for your suggestion about adding requirements.txt to control the version of packages in the repo, will do that soon.