VCTree SGCls +Visual Error Out at Start: Initialize Singleton in Main If Not Using Graft

zhanwenchen commented 1 year ago

The cause is the changed singleton workflow for graft is affecting the semantic vgstats workflow. Resolution:

[x] Fix visual script to address this and rerun the job. No need to create a ticket for this. Just mention this issue in the commit.
[x] Modify the VGStats workflow to accommodate semantic-only. It would be complicated to create a ticket for this given there's an activate branch and pull request for #50 (PR: #86). Just mention this issue in the commit.

The error message is:

Traceback (most recent call last):
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 401, in <module>
    main()
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 394, in main
    model = train(cfg, local_rank, args.distributed, logger)
                                                                                                                                                                                                                                   447,3         78%
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 394, in main
    model = train(cfg, local_rank, args.distributed, logger)
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 60, in train
    model = build_detection_model(cfg)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/detector/detectors.py", line 10, in build_detection_model
    return meta_arch(cfg)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 31, in __init__
    self.roi_heads = build_roi_heads(cfg, self.backbone.out_channels)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/roi_heads.py", line 89, in build_roi_heads
    roi_heads.append(("relation", build_roi_relation_head(cfg, in_channels)))
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/relation_head.py", line 105, in build_roi_relation_head
    return ROIRelationHead(cfg, in_channels)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/relation_head.py", line 33, in __init__
    self.predictor = make_roi_relation_predictor(cfg, feat_dim)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/roi_relation_predictors.py", line 691, in make_roi_relation_predictor
    return func(cfg, in_channels)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/roi_relation_predictors.py", line 329, in __init__
    vg_stats = VGStats()
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/data/vg_stats.py", line 12, in __call__
    cls._instances[cls] = super().__call__(*args, **kwargs)
  File "<string>", line 10, in __init__
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/data/vg_stats.py", line 28, in __post_init__
    raise ValueError('None')
ValueError: None
Traceback (most recent call last):
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 401, in <module>
    main()
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 394, in main
    model = train(cfg, local_rank, args.distributed, logger)
  File "/localtmp/pct4et/relaug/tools/relation_train_net.py", line 60, in train
    model = build_detection_model(cfg)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/detector/detectors.py", line 10, in build_detection_model
    return meta_arch(cfg)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 31, in __init__
    self.roi_heads = build_roi_heads(cfg, self.backbone.out_channels)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/roi_heads.py", line 89, in build_roi_heads
    roi_heads.append(("relation", build_roi_relation_head(cfg, in_channels)))
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/relation_head.py", line 105, in build_roi_relation_head
    return ROIRelationHead(cfg, in_channels)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/relation_head.py", line 33, in __init__
    self.predictor = make_roi_relation_predictor(cfg, feat_dim)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/roi_relation_predictors.py", line 691, in make_roi_relation_predictor
    return func(cfg, in_channels)
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/modeling/roi_heads/relation_head/roi_relation_predictors.py", line 329, in __init__
    vg_stats = VGStats()
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/data/vg_stats.py", line 12, in __call__
    cls._instances[cls] = super().__call__(*args, **kwargs)
  File "<string>", line 10, in __init__
  File "/localtmp/pct4et/relaug/maskrcnn_benchmark/data/vg_stats.py", line 28, in __post_init__
    raise ValueError('None')
ValueError: None

zhanwenchen commented 1 year ago

Is this a problem with code versions? The last commit for local code is +x for scripts on branch 50-add-graft-v1-to-vctree, which is the exactly the same as the branch HEAD. So this is not the cause.

zhanwenchen commented 1 year ago

Why didn't other runs suffer from the same problem? Is it because they weren't actually using the graft dataset?

zhanwenchen commented 1 year ago

Why didn't other runs suffer from the same problem? Is it because they weren't actually using the graft dataset?

It's because it's not actually using the graft, only semantic:

export USE_SEMANTIC=True
export USE_GRAFT=False

zhanwenchen commented 1 year ago

Rerunning the original +visual without the fixed code.

zhanwenchen commented 1 year ago

For the workflow, it's currently

At model init, the graftaugmenter needs an instance of the dataset. So do it in rels

zhanwenchen / relaug

VCTree SGCls +Visual Error Out at Start: Initialize Singleton in Main If Not Using Graft #87