search

VCIP-RGBD / DFormer

[ICLR 2024] DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation
https://yinbow.github.io/Projects/DFormer/index.html
MIT License
142 stars 24 forks source link

Evaluation with my own dataset #14

Closed dongho-Han closed 6 months ago

dongho-Han commented 6 months ago

Hello. Thanks for sharing your nice work! I'm trying to run my own dataset with your model, but got stuck by some points. Could you give any suggestions?

(1) I'm trying to run DFormer on my own captured rgb videos. Therefore, first, I made depth images using Depth Estimation model. However, as your model also requires labels for metric evaluation, I don't have any ideas for that. Could you give any easy way to make label image?

(2) In this code, https://github.com/VCIP-RGBD/DFormer/blob/2aa25e362807b1027bddb3046a96cf1c8ec89cbf/utils/val_mm.py#L105 you used palette for saving the image. Would you give any descriptions for (i) what the palette is(for which purpose), (ii) how can I make the palette for my own dataset?

Thanks!

yinbow commented 6 months ago

Thanks for your attention! (1) This repository is used for semantic segmentation. So the labels here are the semantic maps for the images. Take a label (semantic segmentation map) from NYUDepthV2 as example(shown in the following image), the objects that belong to different category are labeled with different values, e.g., 1 for table, 2 for chair ....

So if you want to make label, we recommend you to use the labelme. Set the category number and get the corresponding mask for different categories. And the background (not belong to any categories you set) is set to 0.

0

(2) (i) For visualization, different categories are colored in different colors. For example, the first item (RGB value) in palette is used to color the first category of the dataset. (ii) You can allocate arbitrary color for your categories. The number of different colors is equal to the categories number for your dataset. Also, you can use the palette from other datasets. We provide a palette in below, you can use part of them for your dataset.

A palette template ` PALETTE = [[120, 120, 120], [180, 120, 120], [6, 230, 230], [80, 50, 50], [4, 200, 3], [120, 120, 80], [140, 140, 140], [204, 5, 255], [230, 230, 230], [4, 250, 7], [224, 5, 255], [235, 255, 7], [150, 5, 61], [120, 120, 70], [8, 255, 51], [255, 6, 82], [143, 255, 140], [204, 255, 4], [255, 51, 7], [204, 70, 3], [0, 102, 200], [61, 230, 250], [255, 6, 51], [11, 102, 255], [255, 7, 71], [255, 9, 224], [9, 7, 230], [220, 220, 220], [255, 9, 92], [112, 9, 255], [8, 255, 214], [7, 255, 224], [255, 184, 6], [10, 255, 71], [255, 41, 10], [7, 255, 255], [224, 255, 8], [102, 8, 255], [255, 61, 6], [255, 194, 7], [255, 122, 8], [0, 255, 20], [255, 8, 41], [255, 5, 153], [6, 51, 255], [235, 12, 255], [160, 150, 20], [0, 163, 255], [140, 140, 140], [250, 10, 15], [20, 255, 0], [31, 255, 0], [255, 31, 0], [255, 224, 0], [153, 255, 0], [0, 0, 255], [255, 71, 0], [0, 235, 255], [0, 173, 255], [31, 0, 255], [11, 200, 200], [255, 82, 0], [0, 255, 245], [0, 61, 255], [0, 255, 112], [0, 255, 133], [255, 0, 0], [255, 163, 0], [255, 102, 0], [194, 255, 0], [0, 143, 255], [51, 255, 0], [0, 82, 255], [0, 255, 41], [0, 255, 173], [10, 0, 255], [173, 255, 0], [0, 255, 153], [255, 92, 0], [255, 0, 255], [255, 0, 245], [255, 0, 102], [255, 173, 0], [255, 0, 20], [255, 184, 184], [0, 31, 255], [0, 255, 61], [0, 71, 255], [255, 0, 204], [0, 255, 194], [0, 255, 82], [0, 10, 255], [0, 112, 255], [51, 0, 255], [0, 194, 255], [0, 122, 255], [0, 255, 163], [255, 153, 0], [0, 255, 10], [255, 112, 0], [143, 255, 0], [82, 0, 255], [163, 255, 0], [255, 235, 0], [8, 184, 170], [133, 0, 255], [0, 255, 92], [184, 0, 255], [255, 0, 31], [0, 184, 255], [0, 214, 255], [255, 0, 112], [92, 255, 0], [0, 224, 255], [112, 224, 255], [70, 184, 160], [163, 0, 255], [153, 0, 255], [71, 255, 0], [255, 0, 163], [255, 204, 0], [255, 0, 143], [0, 255, 235], [133, 255, 0], [255, 0, 235], [245, 0, 255], [255, 0, 122], [255, 245, 0], [10, 190, 212], [214, 255, 0], [0, 204, 255], [20, 0, 255], [255, 255, 0], [0, 153, 255], [0, 41, 255], [0, 255, 204], [41, 0, 255], [41, 255, 0], [173, 0, 255], [0, 245, 255], [71, 0, 255], [122, 0, 255], [0, 255, 184], [0, 92, 255], [184, 255, 0], [0, 133, 255], [255, 214, 0], [25, 194, 194], [102, 255, 0], [92, 0, 255]]`

I hope this can help you. If you have any further questions, feel free to communicate with us.

dongho-Han commented 6 months ago

Thanks for help! I think your answer gives me much information. I'll try and let you know if I have any problem :)