problem loading pretrained weights

[djbeymer]

Hi @jyhjinghwang ,

The SegSort README indicates that the weights from ResNet101.v1 from the Tensorflow-Slim project can be used as weights for the SegSort network. When I try loading the ResNet101.v1 model from Tensorflow-Slim and running pyscripts/inference/inference.py, I get the following error NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Tensor name "resnet_v1_101/block5/conv2/BatchNorm/beta" not found in checkpoint files /home/beymer/SegSort/snapshots/imagenet/trained/resnet_v1_101.ckpt [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Is the ResNet101.v1 model from Tensorflow-Slim compatible with the SegSort model?

Thanks, David

[jyhjinghwang]

Hi David,

Block5 does not exist in the ImageNet-pretrained model, which is why here we exclude 'block5'. I'm not entirely certain what caused this problem on your end but I believe some flags could fix it.

Best, Jyh-Jing

[djbeymer]

Hi @jyhjinghwang

Are there any pre-trained weight for the entire network? And in this github repo, are there implementations for the pixel sorting and segment sorting stages?

Thanks, David

[jyhjinghwang]

Hi David,

We only have the network after fully trained and do not have a partially initialized checkpoint.

And yes, the pixel sorting (KMeans) is implemented in common_utils.py and segment sorting is in train_utils.py. You could follow the comment of each function to get a picture of how it's done.

Best, Jyh-Jing

[limai1990]

Hi David,

We only have the network after fully trained and do not have a partially initialized checkpoint.

And yes, the pixel sorting (KMeans) is implemented in common_utils.py and segment sorting is in train_utils.py. You could follow the comment of each function to get a picture of how it's done.

Best, Jyh-Jing

Hi @jyhjinghwang,

Is pixel sorting not used in unsupervised segmentation of the implementation? The implementation used the over-segmentation results from HED as the cluster labels directly?

Moreover, the paper described that memory banks were used, are there implementations for memory banks?

Thanks, Li

[jyhjinghwang]

Hi Li,

The pixel sorting part is only used during inference in the unsupervised setting. While it is also possible to incorporate into training, I didn't see performance improvements then. So yes, HED oversegmentations are treated as ground truth masks for training.

Yes, the memory banks are implemented. Please check out the code in train_utils.py.

Best, Jyh-Jing

[limai1990]

Hi Li,

The pixel sorting part is only used during inference in the unsupervised setting. While it is also possible to incorporate into training, I didn't see performance improvements then. So yes, HED oversegmentations are treated as ground truth masks for training.

Yes, the memory banks are implemented. Please check out the code in train_utils.py.

Best, Jyh-Jing

Hi @jyhjinghwang,

Thank you for your reply.

The memory bank is not used in the implementation of unsupervised segmentation training? _train_segsortunsup.py uses the loss of _add_unsupervised_segsortloss instead of _add_segsortloss. That is, using memory bank in unsupervised segmentation does not improve the performance?

Thanks, Li

[jyhjinghwang]

Hi Li,

For the unsupervised part, I didn't tune much with the memory bank and I think a couple of mIoU points wouldn't make a difference either. IMHO, this work is to demonstrate how this flexible formulation (SegSort, combining metric learning and discriminative clustering) can be used for unsupervised segmentation and set up a simple baseline for followups. I believe there's still a large room for performance improvement. :)

Hope this answer helps.

Best, Jyh-Jing

[limai1990]

Hi Li,

For the unsupervised part, I didn't tune much with the memory bank and I think a couple of mIoU points wouldn't make a difference either. IMHO, this work is to demonstrate how this flexible formulation (SegSort, combining metric learning and discriminative clustering) can be used for unsupervised segmentation and set up a simple baseline for followups. I believe there's still a large room for performance improvement. :)

Hope this answer helps.

Best, Jyh-Jing

Thank you :)