Open scott870430 opened 3 years ago
Thanks for you interesting.
It seems we missed a script file.
I have added it to our code in the new commit.
Please use the new script file to generate voc_2012_val_instance.json, voc_2012_train_instance.json, sbd_9118_instance.json
:
python3 projects/WSL/tools/convert_voc2012_and_sbd_instance.py
We use the precomputed MCG proposal.
The roi_heads is for instance segmentation and sem_seg_head is for semanctic segmentation.
To perform instance segmentation, we need adjust the configure file by removing sem_seg_head
and changing Dataset
to TRAIN: ('voc_2012_train_instance', 'sbd_9118_instance')
and TEST: ('voc_2012_val_instance',)
.
And it should also set PS_ON: False
, which turns off panoptic segmentation.
We will plan to update new configure files for instance segmentation only.
Thanks, I will try it.
Another question, what is SP_ON
mean in config?
The MCG segmentation proposals are based on superpixels.
Thus, we need to deal with superpixel by setting SP_ON: True
.
Thank you for the reply!
SP_ON
work in train_net.py
And I checked wsl.data build_detection_train_loader
and detectron2.data build_detection_train_loader
, I didn't find any different of them. And also parameter LOAD_PROPOSALS
in config didn't update in add_wsl_config
? So the model will not use proposal in training?
Sorry, I am new in detectron2 project framework, maybe I misunderstood some places...
Is superpixel means proposal in your project? As far as I know, the proposal provided by mcg is called superpixel. Or you did other operating for superpixel?
Will you provide pre-trained weight for panoptic segmentation / instance segmentation?
build_detection_train_loader
functions are the same in detectron2
and wsl
. But load_proposals_into_datasets
functions are different. And load_proposals_into_datasets
is used by get_detection_dataset_dicts
, and get_detection_dataset_dicts
is called by _train_loader_from_config
, and _train_loader_from_config
is need by build_detection_train_loader
.
We use proposals during training. LOAD_PROPOSALS
is set to True
in the parent configure file PascalVOC-Detection/oicr_WSR_18_DC5_1x.yaml
.Thank you for the reply!
Why use superpixel instead of proposal from mcg?
Which GPU did you use for training? And how much time is usually spent on training?
I try to reproduce your work. I run the code on three 2080ti GPUs, and it looks like it will take nearly a week.
I also encountered some errors like Original error was: PyCapsule_Import could not import module "datetime"
, BlockingIOError: [Errno 11] Resource temporarily unavailable
...
Is this caused by insufficient performance of my machine?
Hi,
Thank you for the reply!
It seems that I can have the same results as you when I reach 5000 iterations. However, I encountered `runtimeerror: can't start new thread
at 6000 iterations when testing... (The above errors also occurred during testing or start testing)
It seems that the thread of my server is not enough for testing? Or the program has too many threads and it crashes. I didn’t find any relevant information on detectron2’s github.
Is there any solution, or can these errors be avoided?
Thanks.
About instance segmentation, what do you mean we need adjust the configure file by removing sem_seg_head
?
In config remove the SEM_SEG_HEAD
?
MODEL:
META_ARCHITECTURE: "GeneralizedMCNNWSL"
MASK_ON: True
#SEM_SEG_HEAD:
# NAME: "TwoClassHead"
# IN_FEATURES: ["res5",]
# NUM_CLASSES: 2
ROI_HEADS:
NAME: "JTSMROIHeads"
ROI_BOX_HEAD:
POOLER_TYPE: "MOIPool"
ROI_MASK_HEAD:
NAME: "MaskRCNNConvUpsampleWSLHead"
#NAME: "MaskRCNNConvUpsampleHead"
BACKBONE:
#NAME: "build_wsl_resnet_backbone"
NAME: "build_wsl_resnet_v2_backbone"
FREEZE_AT: 5
After removed SEM_SEG_HEAD, I got error
File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/build.py", line 21, in build_model
model = META_ARCH_REGISTRY.get(meta_arch)(cfg)
File "/home/chcheng/research/JTSM/detectron2/config/config.py", line 181, in wrapped
explicit_args = _get_args_from_config(from_config_func, *args, **kwargs)
File "/home/chcheng/research/JTSM/detectron2/config/config.py", line 238, in _get_args_from_config
ret = from_config_func(*args, **kwargs)
File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/meta_arch/mcnn.py", line 99, in from_config
"sem_seg_head": build_sem_seg_head(cfg, backbone.output_shape()),
File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/semantic_seg.py", line 100, in build_sem_seg_head
return SEM_SEG_HEADS_REGISTRY.get(name)(cfg, input_shape)
File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/semantic_seg.py", line 130, in __init__
1, int(np.log2(feature_strides[in_feature]) - np.log2(self.common_stride))
KeyError: 'p2'
Thanks.
Thank you for the reply!
It seems that I can have the same results as you when I reach 5000 iterations. However, I encountered
`runtimeerror: can't start new thread
at 6000 iterations when testing... (The above errors also occurred during testing or start testing)It seems that the thread of my server is not enough for testing? Or the program has too many threads and it crashes. I didn’t find any relevant information on detectron2’s github.
Is there any solution, or can these errors be avoided?
Thanks.
I don't have this error, maybe you can try to reduce the number of threads.
About instance segmentation, what do you mean
we need adjust the configure file by removing sem_seg_head
? In config remove theSEM_SEG_HEAD
?MODEL: META_ARCHITECTURE: "GeneralizedMCNNWSL" MASK_ON: True #SEM_SEG_HEAD: # NAME: "TwoClassHead" # IN_FEATURES: ["res5",] # NUM_CLASSES: 2 ROI_HEADS: NAME: "JTSMROIHeads" ROI_BOX_HEAD: POOLER_TYPE: "MOIPool" ROI_MASK_HEAD: NAME: "MaskRCNNConvUpsampleWSLHead" #NAME: "MaskRCNNConvUpsampleHead" BACKBONE: #NAME: "build_wsl_resnet_backbone" NAME: "build_wsl_resnet_v2_backbone" FREEZE_AT: 5
After removed SEM_SEG_HEAD, I got error
File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/build.py", line 21, in build_model model = META_ARCH_REGISTRY.get(meta_arch)(cfg) File "/home/chcheng/research/JTSM/detectron2/config/config.py", line 181, in wrapped explicit_args = _get_args_from_config(from_config_func, *args, **kwargs) File "/home/chcheng/research/JTSM/detectron2/config/config.py", line 238, in _get_args_from_config ret = from_config_func(*args, **kwargs) File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/meta_arch/mcnn.py", line 99, in from_config "sem_seg_head": build_sem_seg_head(cfg, backbone.output_shape()), File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/semantic_seg.py", line 100, in build_sem_seg_head return SEM_SEG_HEADS_REGISTRY.get(name)(cfg, input_shape) File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/semantic_seg.py", line 130, in __init__ 1, int(np.log2(feature_strides[in_feature]) - np.log2(self.common_stride)) KeyError: 'p2'
Thanks.
Sorry, it is my wrong.
To only perform instance segmentation for PASCAL VOC, you can try to set PS_ON`` to
False, and leave
SEM_SEG_HEAD``` unchanged.
More mofidies are need for MS COCO.
Thank you for the reply!
Is there any way to reduce the number of threads from the code or I need to add code for reduce threads?
Thanks, I will try instance segmentation for VOC.
Regarding instance segmentation on MS COCO, it need to remove SEM_SEG_HEAD in config? Btw, will you upload config for instance segmentation recently?
You should change the codes in this line.
For MS COCO, you also need to set SEM_SEG_HEAD
as PASCAL VOC.
I will plan to upload new config.
Thank you for the reply!
Is there any way to reduce the number of threads from the code or I need to add code for reduce threads?
Thanks, I will try instance segmentation for VOC.
Regarding instance segmentation on MS COCO, it need to remove SEM_SEG_HEAD in config? Btw, will you upload config for instance segmentation recently?
Thanks, I will try it.
SEM_SEG_HEAD
is not use in instance segmentation, but for MS COCO, I need to set SEM_SEG_HEAD
as PASCAL VOC?
Am i misunderstanding something?
Thanks!
Yes, another way is remove semantic segmentation part from mcnn.py.
Thanks, I will try it.
SEM_SEG_HEAD
is not use in instance segmentation, but for MS COCO, I need to setSEM_SEG_HEAD
as PASCAL VOC? Am i misunderstanding something?Thanks!
Hi, following your instance segmentation setting, I got error with
File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 799, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/meta_arch/mcnn.py", line 217, in forward
images, features, proposals, gt_instances, gt_sem_seg, superpixels
File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/roi_heads_jtsm.py", line 540, in forward
losses = self._forward_box(features, proposals)
File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/roi_heads_jtsm.py", line 657, in _forward_box
losses = self.box_predictor.losses(predictions, proposals, self.gt_classes_img_oh)
File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 692, in losses
gt_classes_img_oh,
File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 404, in losses
return {"loss_cls": self.binary_cross_entropy_loss()}
File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 361, in binary_cross_entropy_loss
self.predict_probs_img(), self.gt_classes_img_oh, reduction="mean"
File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/functional.py", line 2886, in binary_cross_entropy
"Please ensure they have the same size.".format(target.size(), input.size())
ValueError: Using a target size (torch.Size([1, 20])) that is different to the input size (torch.Size([1, 21])) is deprecated. Please ensure they have the same size.
It seems that the model will still output 21 categories (including semantic?) but the ground truth only 20 categories (for instance segmentation?).
In here, the box prediction will add num_classes_stuff - 1
, should modify
SEM_SEG_HEAD:
NAME: "TwoClassHead"
IN_FEATURES: ["res5",]
NUM_CLASSES: 2
NUM_CLASSES
to 1? After modified to one, the code will work, but I am not sure about that the modify is correct?
Some questions about the architecture, the panoptic mining branch in paper is implement in ROI_HEADS in code? What is pgt and mist meaning in here?
For instance segmentation, can I remove this line for speed up training?
Something about Panoptic Segmentation question, as I know, the stuff class in Panoptic Segmentation amorphous regions of similar texture or material such as grass, sky, road.
Thanks.
Hi, following your instance segmentation setting, I got error with
File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 799, in forward output = self.module(*inputs[0], **kwargs[0]) File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/meta_arch/mcnn.py", line 217, in forward images, features, proposals, gt_instances, gt_sem_seg, superpixels File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/roi_heads_jtsm.py", line 540, in forward losses = self._forward_box(features, proposals) File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/roi_heads_jtsm.py", line 657, in _forward_box losses = self.box_predictor.losses(predictions, proposals, self.gt_classes_img_oh) File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 692, in losses gt_classes_img_oh, File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 404, in losses return {"loss_cls": self.binary_cross_entropy_loss()} File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 361, in binary_cross_entropy_loss self.predict_probs_img(), self.gt_classes_img_oh, reduction="mean" File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/functional.py", line 2886, in binary_cross_entropy "Please ensure they have the same size.".format(target.size(), input.size()) ValueError: Using a target size (torch.Size([1, 20])) that is different to the input size (torch.Size([1, 21])) is deprecated. Please ensure they have the same size.
It seems that the model will still output 21 categories (including semantic?) but the ground truth only 20 categories (for instance segmentation?).
In here, the box prediction will add
num_classes_stuff - 1
, should modifySEM_SEG_HEAD: NAME: "TwoClassHead" IN_FEATURES: ["res5",] NUM_CLASSES: 2
NUM_CLASSES
to 1? After modified to one, the code will work, but I am not sure about that the modify is correct?
I think the modification is right. But you should carefull check whether stuff classes are involved during mining, as it will affect the mined results for thing classes. We prefer to manually remove the segmantic part to performan instance segmentation.
Some questions about the architecture, the panoptic mining branch in paper is implement in ROI_HEADS in code?
Yes. ROI_HEADS has both panoptic panoptic mining branch and instance segmentation branch. The default semantic segmentation branch is SemSegFPNHead.
What is pgt and mist meaning in here?
pgt is pseudo-ground-truth and mist is multiple instance self-training.
For instance segmentation, can I remove this line for speed up training?
This line can be removed.
Something about Panoptic Segmentation question, as I know, the stuff class in Panoptic Segmentation amorphous regions of similar texture or material such as grass, sky, road.
- In your work, you only recognize background, not the grass, sky... etc. in pascal voc 2012 dataset?
Yes.
- Is it because of the lack of relevant annotations that lead you to set up like this?
Yes.
- In pascal voc 2012, are all pixels other than instance classified as background?
Yes.
Thanks.
You are welcome
Thank you for the reply!
We prefer to manually remove the segmantic part to performan instance segmentation.
If I don't remove the semantic part, it will still compute loss for semantic? Does the modification for instance segmentation just change the calculation method of evaluation? (Panoptic Segmentation -> instance segmentation?)
If I want to remove the semantic segmentation part, is there anything else that needs to be removed besides this line?
Thanks.
I think removing semantic segmentation part in mcnn.py is enough.
Hi, some question about architecture, I found mask_head
and mask_refinery_0
in architecture, these two layers are also MaskRCNNConvUpsampleWSLHead
, and their architecture is the same... (the config you provided also has these two layers)
Why we need these two layer?
Thanks.
Hi, some question about architecture, I found
mask_head
andmask_refinery_0
in architecture, these two layers are alsoMaskRCNNConvUpsampleWSLHead
, and their architecture is the same... (the config you provided also has these two layers) Why we need these two layer? Thanks.
Hi,
mask_refinery_0
is used to refine mask results, which takes mask predictions from mask_head
as supervison.
Is mask_refinery_0
the instance segmentation branch?
And where is mask_head
in architecture?
Isn't the instance segmentation branch take output from grabcut as supervision?
Hi, I found that the training process program will also be interrupted in the config you provided.
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 1800000ms for recv operation to complete
Why is this happening? Thanks.
Is
mask_refinery_0
the instance segmentation branch? And where ismask_head
in architecture? Isn't the instance segmentation branch take output from grabcut as supervision?
Hi, mask_refinery_0
and mask_head
are both in the instance segmentation branch.
And mask_refinery_0
takes the predictions of mask_head
as supervision.
We can also add more refinement.
Hi, I found that the training process program will also be interrupted in the config you provided.
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 1800000ms for recv operation to complete
Why is this happening? Thanks.
I haven't met such issues. It seems to be the problem in distributed training.
Hi, I am interested in your project. I ran into some problems:
following README.md
I get error FileNotFoundError: [Errno 2] No such file or directory: 'datasets/VOC_SBD/annotations/voc_2012_val_instance.json' Also, there are only
voc_2012_val_panoptic.json, voc_2012_train_panoptic.json, sbd_9118_panoptic.json
, three files in datasets/VOC_SBD/annotations/ folder. Am i missing something?Is proposal_generator use in Panoptic Segmentation? Or use the precomputed_proposal (MCG?)
Is roi_heads for instance segmentation and sem_seg_head for semanctic segmentation? If I want to perform instance segmentation, I just need to remove the sem_seg_head?
Thank you for your help!