search

JDAI-CV / Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving

Virtural try-on under arbitrary poses
MIT License
217 stars 65 forks source link

how to test on new images #5

Closed trinanjan12 closed 4 years ago

trinanjan12 commented 4 years ago

I have a person's image wearing a top. This is my reference image. Also, I have the segmentation map of the reference image. I take one more image which is my target image. And I have created 18 points key points from the open pose of the target image. Now I have an in-shop clothe(top), whose binary mask I have calculated using OpenCV. I need to wrap this clothe on the target image(whose key points I have generated). Haven't read the full paper yet

  1. What all things I need to generate to test on my own data?
  2. Do I need to resize all the images to 256x192 before feeding it to the network
  3. in the demo_txt file I am using the below format reference image.jpg target_pose_keypoints.json clothe_item.jpg

and I am getting below error while running demo.sh. I ran the demo.sh with the example settings and it generated images. So the code is working. The only thing is I am not able to test for my own images.

Thanks in advance

Traceback (most recent call last): File "demo.py", line 185, in forward(opt, paths, 4, opt.forward_save_path) File "demo.py", line 109, in forward warped_cloth = warped_image(gmm, result) File "/media/trinanjan/BACKUP/Trinanjan/current_project/virtual_try_on/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving/utils/warp_image.py", line 28, in warped_image agnostic = torch.cat([source_parse_shape, im_h, target_pose_embedding], 1) RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 256 and 1100 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:87

trinanjan12 commented 4 years ago

This issue is solved. I had to resize the images(ref,target,product) to (h=256,w=192) and calculate.The binary mask, segmentation map, pose key points should be created for the resized images.

j-sieger commented 4 years ago

I have resized the images but while loading the dataset i am getting the below error. Image at entry 0 is taken from official dataset. image at entry 4 is my own image. so facing an issue with my images they are not same shape datasets img shape.

Traceback (most recent call last): File "demo.py", line 192, in forward(opt, paths, 4, forward_save_path) File "demo.py", line 108, in forward for i, result in enumerate(val_dataloader): File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 345, in next data = self._next_data() File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data) File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise() File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 395, in reraise raise self.exc_type(msg) RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop data = fetcher.fetch(index) File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch return self.collate_fn(data) File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate return {key: default_collate([d[key] for d in batch]) for key in elem} File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in return {key: default_collate([d[key] for d in batch]) for key in elem} File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate return torch.stack(batch, 0, out=out) RuntimeError: stack expects each tensor to be equal size, but got [1, 256, 192] at entry 0 and [1, 256, 192, 3] at entry 4

trinanjan12 commented 4 years ago

Looks like your data loader has an issue as it is failing in the getitem. try creating a Jupyter notebook only for the dataloder to see if you can confirm the below shapes(visualization also might help)

Not sure if this will help or not if you just run the dataloader and try to check the data coming it will look something like this for batch of 1

This is train dataloder

source_parse (20, 256, 192) target_parse (20, 256, 192) source_parse_vis torch.Size([3, 256, 192]) target_parse_vis torch.Size([3, 256, 192]) source_pose_embedding (18, 256, 192) target_pose_embedding (18, 256, 192) target_pose_loc 18 source_image torch.Size([3, 256, 192]) target_image torch.Size([3, 256, 192]) cloth_image torch.Size([3, 256, 192]) cloth_parse torch.Size([1, 256, 192]) source_parse_shape torch.Size([1, 256, 192]) im_h torch.Size([3, 256, 192]) im_c torch.Size([3, 256, 192]) source_image_name 5 target_image_name 5 cloth_image_name 5 warped_cloth_image torch.Size([3, 256, 192]) warped_cloth_name 5 warped_cloth_path 25 source_img_path 27 target_img_path 27 target_pose_path 45 target_parse_path 31 source_parse_vis_path 35 target_parse_vis_path 35 target_pose_img (256, 192, 3) warped_cloth_parse torch.Size([1, 256, 192]) target_parse_cloth (256, 192)

j-sieger commented 4 years ago

@trinanjan12 : Thanks for the quick reply. I found that issue is with my binary masked cloth. added the debuggers. def parse_cloth(self, path, downsample=False):
cloth_parse = Image.open(path) print("cloth_parse",cloth_parse.size) cloth_parse_array = np.array(cloth_parse) print("cloth_parse_array",cloth_parse_array.shape) cloth_parse = (cloth_parse_array == 255).astype(np.float32) # 0 | 1 cloth_parse = cloth_parse[np.newaxis, :] print("cloth_parse2", cloth_parse.shape) ----- output for dataset mask is: cloth_parse (192, 256) cloth_parse_array (256, 192) cloth_parse2 (1, 256, 192) ----- output for my masked cloths: cloth_parse (192, 256) cloth_parse_array (256, 192, 3) cloth_parse2 (1, 256, 192, 3) i will try to modify the way of creating my binary masked images.

trinanjan12 commented 4 years ago

Cloth mask are binary images (0/1). something like this I think

prod_images = glob.glob(resize_img_path + '/clothimage/{}.png'.format('product')) for index,img_path in tqdm.tqdm(enumerate(prod_images), total = len(prod_images)): image = plt.imread(img_path) image_copy = image.copy() white_pixels_mask = np.all(image == [255, 255, 255], axis=-1) non_white_pixels_mask = np.any(image != [255, 255, 255], axis=-1)
image_copy[white_pixels_mask] = [0,0,0] image_copy[non_white_pixels_mask] = [1,1,1] cv2.imwrite(resize_img_path +'/clothmask/product{}_mask.png'.format(index+1), non_white_pixels_mask255)

j-sieger commented 4 years ago

Thank you so much. this is so simple. I have used grabcut and many other things to get the binary mask and still have issue with the shape. but your is working fine but the mask is not so clear

trinanjan12 commented 4 years ago

I haven't found any models yet which works only on product images. grabcut/segmentation methods are not working well on the product images. I haven't found any deep learning model that was trained only with product images for segmentation. Let me know if you have a luck with that. My function is a brute force way of doing it if the background is white.

j-sieger commented 4 years ago

Yes. Even i am facing same issue. Even My code is also not working for production images. I will let you know if i found something useful.

bhavesh2799 commented 4 years ago

@j-sieger @trinanjan12 Can anyone tell what is the issue here. I have resized all the images to (256, 192). Currently I am thinking there are two possible regions of error: Openpose json file, which used BODY25 model with face and hands model, or there is some internal code error.

Also in demo.txt, first 4 sets of images are their example images, and the last one is my custom image.

`/content/drive/My Drive/Eroom-models-working/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving Running forward

trinanjan12 commented 4 years ago

@bhavesh2799 Haven't worked on it for some time. I am attaching a file that I had used for data preparation. Just running the file may not work. I am attaching for you reference purpose. It has a script to convert 25 keypoints to 18 keypoints. https://colab.research.google.com/drive/1IM84ldwD36XTEPN90PVenzj2R_Q1lUad

bhavesh2799 commented 4 years ago

@bhavesh2799 Haven't worked on it for some time. I am attaching a file that I had used for data preparation. Just running the file may not work. I am attaching for you reference purpose. It has a script to convert 25 keypoints to 18 keypoints. https://colab.research.google.com/drive/1IM84ldwD36XTEPN90PVenzj2R_Q1lUad

@trinanjan12 This works for me. But came to another error in running demo.sh

/content/drive/My Drive/Eroom-models-working/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving Running forward

trinanjan12 commented 4 years ago

Hard to say. A good idea is to go through line by line the data loader file. This kind of problem happens if you don't prepare the data exactly the way the repo had suggested. For example the keypoint files should have exact Json structure others it will show error, as it is being index with exact string key.

bhavesh2799 commented 4 years ago

Hard to say. A good idea is to go through line by line the data loader file. This kind of problem happens if you don't prepare the data exactly the way the repo had suggested. For example the keypoint files should have exact Json structure others it will show error, as it is being index with exact string key.

Yeah, I checked it. Earlier I tried converting the openpose json output to coco format through the openpose model, and the format was quite different from desired. So, now I am using the script mentioned in https://colab.research.google.com/drive/1IM84ldwD36XTEPN90PVenzj2R_Q1lUad by you. But it's showing index error here.

# convert 25 keypoints to 18 keypoints
import numpy as np
import json
key_18_path = '/content/drive/My Drive/coco-json'
export_path = '/content/drive/My Drive/Eroom-models-working/my-input-source-images/'
for index,file_name in enumerate(key_25_files):
    key_25,key_18 = [],[]
    x = {"version":1.3,"people":[{"person_id":[-1],"pose_keypoints_2d":[],"face_keypoints_2d":[],"hand_left_keypoints_2d":[],"hand_right_keypoints_2d":[],"pose_keypoints_3d":[],"face_keypoints_3d":[],"hand_left_keypoints_3d":[],"hand_right_keypoints_3d":[]}]}
    with open(file_name,'r') as f:
        key_25 = np.array(json.load(f))
    key_18 = key_25[0:8]
    key_18 = np.append(key_18,key_25[9:19])

    x_axis = key_18[::3]
    y_axis = key_18[1::3]
    z_axis = key_18[2::3]

    extension = '.jpg'
    img_file_name =  export_path +os.path.basename(file_name).split('_keypoint')[0]+extension
    img_orig = Image.open(img_file_name)

    x_ratio = (192/img_orig.size[0])
    y_ratio = (256/img_orig.size[1])

    small_x_axis = x_axis * x_ratio
    small_y_axis = y_axis * y_ratio
    small_key_18 = list(zip(small_x_axis,small_y_axis,z_axis))

    small_key_18 = np.ravel(np.array(small_key_18))
    x['people'][0]['pose_keypoints_2d'] = small_key_18.tolist()
    with open( key_18_path + '/{}'.format(os.path.basename(file_name)),'w') as f:
        json.dump(x,f)
IndexError                                Traceback (most recent call last)
<ipython-input-10-8e14d543bcdd> in <module>()
      9     with open(file_name,'r') as f:
     10         key_25 = np.array(json.load(f))
---> 11     key_18 = key_25[0:8]
     12     key_18 = np.append(key_18,key_25[9:19])
     13 

IndexError: too many indices for array
trinanjan12 commented 4 years ago

@bhavesh2799 could be a problem with the openpose keypoints. Again the colab was for your reference. I have worked on it long back, so not sure if both of our scenarios same. Checkout openpose keypoints mapping https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/output.md. Also I don't think you have to use openpose only. I think you can use detectron2 also for keypoints extraction. Detectron2 is much easier.

bhavesh2799 commented 4 years ago

@trinanjan12 , could you please tell where could be the error. I updated the json to coco format. So I am pretty sure, that there isn't any error wrt json file. All images are of size (256,192,3). I don't understand why it is showing size as [1,256,192,3]. I have rechecked the sizes of all images, it works fine if it's just their inference data, but fails with my dataset.

First 4 examples in demo.txt are the inference samples, and the last is my custom sample.

1LJ21D005-G11@10=person_half_front.jpg AD121D0G7-A11@10=person_half_front_keypoints.json CO121D08G-O11@10=cloth_front.jpg test
1VJ21D02K-Q11@8=person_half_front.jpg AD121D0F0-G12@10=person_half_front_keypoints.json AD121D0HY-Q11@18=cloth_front.jpg test
4HI21D003-C11@10=person_half_front.jpg EV421EA51-A11@9=person_whole_front_keypoints.json 1VJ21D02B-Q11@10.1=cloth_front.jpg test
A0F21D012-K11@8=person_half_front.jpg EV421DAIY-E11@8=person_half_front_keypoints.json AS141D04K-Q11@16.1=cloth_front.jpg test
10resized.jpg 10resized_keypoints.json s-1resized.jpg test
/content/drive/My Drive/Eroom-models-working/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving
Running forward
+ CUDA_VISIBLE_DEVICES=0 python demo.py --batch_size_v 80 --num_workers 4 --forward_save_path demo/forward
Namespace(G_GAN=1, G_VGG=1, G_nn=1, batch_size_t=128, batch_size_v=80, beta1=0.5, dataroot=False, dataset='MPV', dataset_mode='regular', decay_iters=10, epoch=200, face_L1=10, face_gan=3, face_img_L1=1, face_residual=False, face_vgg=1, fine_height=256, fine_width=192, forward='normal', forward_save_path='demo/forward', gan_mode='lsgan', gpu_ids=[0, 1, 2, 3], grid_size=5, init_gain=0.02, init_type='normal', input_nc_D_app=6, input_nc_D_face=6, input_nc_D_parsing=56, input_nc_G_app=26, input_nc_G_face=6, input_nc_G_parsing=36, isdemo=False, isval=False, joint=False, joint_G_parsing=1, joint_all=False, joint_parse_loss=False, lambda_L1=1, lr=0.0002, mask=1, mask_tvloss=False, momentum=0.9, n_layers_D=3, ndf=64, netD_app='resnet_blocks', netD_face='resnet_blocks', netD_parsing='basic', netG_app='treeresnet', netG_face='treeresnet', netG_parsing='unet_256', ngf=64, no_dropout=False, norm='instance', num_workers=4, output_nc_app=4, output_nc_face=3, output_nc_parsing=20, pool_size=100, print_freq=10, resume_D_app='', resume_D_face='', resume_D_parse='', resume_G_app='pretrained_checkpoint/app.tar', resume_G_face='pretrained_checkpoint/face.tar', resume_G_parse='pretrained_checkpoint/parsing.tar', resume_gmm='pretrained_checkpoint/step_009000.pth', save_epoch_freq=1, save_time=False, size=(256, 192), start_epoch=0, suffix='', train_mode='parsing', use_gmm=False, val_freq=200, warp_cloth=False, weight_decay=0.0001)
initialization method [normal]
initialization method [normal]
initialize network with normal
initialize network with normal
initialize network with normal
====================
====================
====================
====================
==>loaded model
Traceback (most recent call last):
  File "demo.py", line 184, in <module>
    forward(opt, paths, 4, opt.forward_save_path)
  File "demo.py", line 106, in forward
    for i, result in enumerate(val_dataloader):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 73, in default_collate
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 73, in <dictcomp>
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [1, 256, 192] at entry 0 and [1, 256, 192, 3] at entry 4

I don't understand why it is showing it this why with my custom image.

trinanjan12 commented 4 years ago

@bhavesh2799 the error says it is expecting a image of batch_size,h,w and it is getting a image of batch_size,h,w,num_channel. I think the error is because the image expected to be a binary map of the cloth image but instead it is getting a rgb image. Try following the exact procedure that they have mentioned with naming and folder convention as the folder names and the id's of images are indexed directly in the files.

bhavesh2799 commented 4 years ago

@trinanjan12 I tried doing that. It is returning the same error. Is there a chance of internal code error. I checked the sample dataset images, none of them are binary. Or I used cv2 to resize images to 256,192, can that be a cause of this error

trinanjan12 commented 4 years ago

@bhavesh2799 aren't the mask images binary. https://github.com/JDAI-CV/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving/blob/master/dataset/cloth_mask/1VJ21D02B-Q11%4010.1%3Dcloth_front_mask.png

bhavesh2799 commented 4 years ago

nope, they have three channels

bhavesh2799 commented 4 years ago

@trinanjan12 proof

trinanjan12 commented 4 years ago

@bhavesh2799 if you check the parse cloth function in demo dataset it is actually filtering out the mask. So if you have set paths correctly it should be taken care of. This could be some other problem.

bhavesh2799 commented 4 years ago

@trinanjan12 It's really strange. I have checked the dataset multiple times. It doesn't seem to work with new images, but works fine with their own inference images. There maybe some issue with the Dataloader, I don't know actually... If you have any working colab implementation, or know anyone who has tested successfully on custom dataset, please tell

trinanjan12 commented 4 years ago

@bhavesh2799 I had made it work with their existing code. So I don't think it's a problem with the code of this repo. Mostly dataloader. I don't have a working copy of the same code, I had modified a lot of things for my own work, so that may not work for you! If you need help regarding other virtual tryon repo checkout papers with code. you could follow these people https://github.com/Yagami360, https://github.com/minar09 Good luck!

bhavesh2799 commented 4 years ago

@trinanjan12 Finally solved it. It's working now. Problem was with binary masks itself. But it generates really strange output, and why there is a woman's face on all images. error Can we use high resolution images here instead of 256,192 error2

trinanjan12 commented 4 years ago

@bhavesh2799 Congrats! 1.you can't use it for bigger resolution, most of the vton system available works for small resolution.

  1. The face part blends 2 faces from the source and target image using a face network, check out the paper 3.the better the output of predicted segmentation map the better the gan output 4.Qulality of the output depends on the data trained and the difficulty of poses.
bhavesh2799 commented 4 years ago

@trinanjan12 Can we somehow disable the generation network, if I am using the same image as source and target, so at least it works optimally in warping cloth on target image?

amrahsmaytas commented 4 years ago

@trinanjan12 Can we somehow disable the generation network, if I am using the same image as source and target, so at least it works optimally in warping cloth on target image?

congratulations @bhavesh2799 ,

  1. the strange cloth shape is because of the keypoint modification which results in strange wrapping cloth, so take care of joints/points while making any changes in them.

  2. not sure much about face , but, there are other methods which helps preserve the face and body , you can checkout https://github.com/switchablenorms/DeepFashion_Try_On (for reference)

  3. could you please share the modified code here or to my mail (greetsatyamsharma@gmail.com) as i am also facing a lot of issues while testing the repo on custom images ( in colab,its tough, man)

  4. Can any of you , both , guide me clear steps on how to test it up on custom images @trinanjan12 @bhavesh2799

Thanks in advance, satyam.