Test on own images

[ghost]

How to test the results on my own image(s)?

[lartpang]

You can try these steps:

1. change the item https://github.com/lartpang/MINet/blob/master/code/utils/config.py#L20 to MINet_VGG16@<some chars>
2. make a MINet_VGG16_<some chars>(@ -> _) folder in the output folder, and make a 'pth` folder in it.
3. download the pretrained parameters of our model, like "MINet_VGG16.pth" and put it into the output folder as mentioned here: https://github.com/lartpang/MINet/tree/master/code#if-you-want-to-test-the-trained-model-again
4. and then, you may modify some code. (when I check my code in https://github.com/lartpang/MINet/blob/master/code/utils/solver.py#L63-L68, I find some mistakes, and I will fix them soon :< )

You need change these codes:

# https://github.com/lartpang/MINet/blob/master/code/utils/solver.py#L63-L68
        if self.args["resume"]:
            self.resume_checkpoint(load_path=self.path["final_full_net"], mode="all")
        else:
            self.start_epoch = 0
        self.end_epoch = self.args["epoch_num"]
        self.only_test = self.start_epoch == self.end_epoch

to:

# https://github.com/lartpang/MINet/blob/master/code/utils/solver.py#L63-L68
        if self.args["resume"]:
            self.start_epoch = self.args["epoch_num"]
        else:
            self.start_epoch = 0
        self.end_epoch = self.args["epoch_num"]
        self.only_test = self.start_epoch == self.end_epoch

5. in https://github.com/lartpang/MINet/blob/master/code/utils/config.py#L21, change the item resume to 'True`
6. modify the dict: https://github.com/lartpang/MINet/blob/master/code/utils/config.py#L30-L37 to only contain your data and you must enble that the contained path include the two folders: Image & Mask. (And I have implemented some functions to get the path list from the file, you can read the code in https://github.com/lartpang/MINet/blob/master/code/utils/imgs/create_rgb_datasets_imgs.py, and you can try it. :)

NOTE: Currently, I only released the code to test images which have the paired masks. But it is easy to modify the code in https://github.com/lartpang/MINet/blob/master/code/utils/imgs/create_rgb_datasets_imgs.py and https://github.com/lartpang/MINet/blob/master/code/utils/solver.py to test your images without masks.

Please use this code more often and your suggestions will make it better.

[ghost]

Thanks, will try out and let you know :)

[ghost]

H @lartpang, I did try out the code with the changes suggested by you. But I think there is a problem with the checkpoint file or in the code of loading the checkpoint as I did get the following errors-

RuntimeError: Error(s) in loading state_dict for CPLightMINet_Res50: size mismatch for trans.conv2.h2h_0.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 256, 3, 3]). size mismatch for trans.conv2.h2h_0.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.m2m_0.weight: copying a param with shape torch.Size([256, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 512, 3, 3]). size mismatch for trans.conv2.m2m_0.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.l2l_0.weight: copying a param with shape torch.Size([256, 1024, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 1024, 3, 3]). size mismatch for trans.conv2.l2l_0.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnh_0.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnh_0.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnh_0.running_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnh_0.running_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_0.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_0.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_0.running_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_0.running_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnl_0.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnl_0.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnl_0.running_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnl_0.running_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.h2h_1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.h2h_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.h2m_1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.h2m_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.m2h_1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.m2h_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.m2m_1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.m2m_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.m2l_1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.m2l_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.l2m_1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.l2m_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.l2l_1.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.l2l_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnh_1.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnh_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnh_1.running_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnh_1.running_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_1.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_1.running_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_1.running_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnl_1.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnl_1.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnl_1.running_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnl_1.running_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.h2m_2.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.h2m_2.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.l2m_2.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.l2m_2.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.m2m_2.weight: copying a param with shape torch.Size([256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv2.m2m_2.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_2.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_2.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_2.running_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.bnm_2.running_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv2.m2m_3.weight: copying a param with shape torch.Size([64, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.h2h_0.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 512, 3, 3]). size mismatch for trans.conv3.h2h_0.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.m2m_0.weight: copying a param with shape torch.Size([512, 1024, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 1024, 3, 3]). size mismatch for trans.conv3.m2m_0.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.l2l_0.weight: copying a param with shape torch.Size([512, 2048, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 2048, 3, 3]). size mismatch for trans.conv3.l2l_0.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnh_0.weight: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnh_0.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnh_0.running_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnh_0.running_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_0.weight: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_0.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_0.running_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_0.running_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnl_0.weight: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnl_0.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnl_0.running_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnl_0.running_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.h2h_1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.h2h_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.h2m_1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.h2m_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.m2h_1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.m2h_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.m2m_1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.m2m_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.m2l_1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.m2l_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.l2m_1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.l2m_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.l2l_1.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.l2l_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnh_1.weight: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnh_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnh_1.running_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnh_1.running_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_1.weight: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_1.running_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_1.running_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnl_1.weight: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnl_1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnl_1.running_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnl_1.running_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.h2m_2.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.h2m_2.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.l2m_2.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.l2m_2.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.m2m_2.weight: copying a param with shape torch.Size([512, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3]). size mismatch for trans.conv3.m2m_2.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_2.weight: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_2.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_2.running_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.bnm_2.running_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]). size mismatch for trans.conv3.m2m_3.weight: copying a param with shape torch.Size([64, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 64, 3, 3])

[lartpang]

Hi, the reason should be that you doesn't change the item https://github.com/lartpang/MINet/blob/master/code/utils/config.py#L20.

The value of this item use @ to separate the name of the model class in the folder network from the custom additional name.

The pretrained weight file I provided corresponds to these two models in https://github.com/lartpang/MINet/blob/master/code/network/MINet.py .

So, you should set the model class name before @ of the value of the item to MINet_VGG16 or MINet_Res50.

I did not provide the pretrained parameters of other models. Of course, I can share it with you if you want.

[lartpang]

@karnabhc I have updated the code, you can try it.

[ghost]

I see an error in https://github.com/lartpang/MINet/blob/master/code/utils/pipeline_ops.py#L5 where

import torch.optim.optimizer as optim

should be replaced by import torch.optim as optim

[ghost]

Also, as you mentioned

Added some backbone models using ws+gn which can further improve the generalization performance of the model. If there is a need, I can provide the pretrained parameter file of the corresponding improved version of MINet.

I would like to know what other models can be used and their corresponding pretrained parameters.

[lartpang]

I have pretrained some models with BigBatch+SomeTrainTrick+CheckpointFeature

1. VGG-16
2. ResNet-50
3. ResNet-WS+GN-50 (batchsize = 16, https://github.com/lartpang/MINet/blob/master/code/network/WSGNLightMINet.py#L11). It has the best performance, so I update it to the google drive: https://drive.google.com/drive/folders/16yTcf_m-ehnhWgXlN6hbZpBKMy6lYIQQ?usp=sharing (CPLMINet_WSGNRes50_40-0.025-Poly) Because I use the batchsize 32, I don't try the *-101 backbone limited by my device.

[lartpang]

I see an error in https://github.com/lartpang/MINet/blob/master/code/utils/pipeline_ops.py#L5 where

import torch.optim.optimizer as optim

should be replaced by import torch.optim as optim

I import it for type checking: https://github.com/lartpang/MINet/blob/master/code/utils/pipeline_ops.py#L42. And I can run the code without any error...

[ghost]

HI @lartpang I faced some issues, what should be the exp_name in config.py and the changes in /code/network/init.py corresponding to the above checkpoint file?

[lartpang]

For example: exp_name: MINet_VGG16@e_40_lr_0.025_opti_f3trick_sche_Poly /network/init.py: from .MINet import * (Because the model MINet_VGG16 is in MINet.py) checkpoint path: /output/MINet_VGG16_e_40_lr_0.025_opti_f3trick_sche_Poly/pth/state_final.pth

so:

1. you can set the exp_name to CPLMINet_WSGNRes50@40-0.025-Poly, the code will use the model https://github.com/lartpang/MINet/blob/master/code/network/WSGNLightMINet.py#L11.
2. and then, you need add from .WSGNLightMINet import * in the __init__.py and put the parameter file into /output/CPLMINet_WSGNRes50_40-0.025-Poly/pth/

[ghost]

Thanks, it worked!