How to add your depth network as a head in main detectron2?

[Dheeru66k]

Hi i also want to add box head and Sematic Seg Head, to make it to do multitask learning along with depth, how can i do that ? What all things i needed from your repo so that i can create a new network based which also perform depth estimation, semantic segmentation and depth estimatiion?

[zzzxxxttt]

If you plan to do supervised depth estimation along with detection or segmentation, I suggest modifying the official detectron2 repo by copy&paste depth training related components from this repo into it. If you need to perform self-supervised depth estimation training, integrating the detection/segmentation part into this repo may be a little bit easier, but a lot of effort is still needed.

[Dheeru66k]

Basically i am trying to do Depth, Semantic and Obj Detection and if have mapped all the depth related components into detectron2 local clone as you said

i have copied "DepthDecoder", "ResnetEncoder" and "DepthresNet" and "SupDepth" model and some other relative imports such as geometry.

Now if want to use "GeneralizedRCNN" architecture instead if your "SupDepth" how can i achieve that along with Maskhead and Box head, because they are added as "ROIHeads" in GeneralizedRCNN. And "GeneralzedRCNN" uses "resnet_fpn_backbone" and your model uses "DepthResNet"., i am a bit confused here.

in RCNN architecture and YAML config looks likes this after changes

but this is calling again a backbone named Depth_net, it means my Architecture is having 2 backbones one for depth and one for Detection and Segmentation, which doesnt make sense for me ,will this can achieve my goal?

and in DepthNet i have registered as backbone and gave some inputs for to be called from config as below

Should i add depth data as a ROIHead or as can i do same things as i did above, can you please guide me with this ?