PeterVennerstrom
commented
2 years ago LoadMultiChannelImageFromFiles will load both image files and stack them. This will likely be compatible with some pipeline functions, but not others.
There might be some functions like resize that are compatible with a [h, w, 6] shape, but not [h, w, 3, 2] so you may want to change stack to concatenate in LoadMultiChannelImageFromFiles.
Some functions like PhotoMetricDistortion will require a [h, w, 3] input. If you are using function with this requirement a possible solution would be to decorate with a function that splits the 6 channel image into two 3 channel images and sends the appropriate one through the transform function then concatenates them after.
Maenansyn
YihanWang-chengdu
nsnlz
WuTao-CS
dagezhuang
Troy-peng-0327
nikky4D
qqq1521902442
jinuncle
realkanchi
unnamed0216
vlaminckaxel
KiZemmy
zmaomia
goodday-94
JINAOLONG
YangParky
I adapted Faster R-CNN to be bi-modal (colour images and an aligned second 3-channel 2D sensor data). For this I duplicated the ResNet50-backbone and use a splitted forward-function. Given input x for the function is a list of two elements. x[0] is forwarded into the first ResNet50-backbone, x[1] is forwarded into the second branch. Afterwards is fusioned by an element-wise max-operation and then continued as regular Faster R-CNN.
The model structure created looks as expected, loading an adapted checkpoint works as well, but running images through the network does not work as the pipeline is not made for two images per instance and interprets it as a batch. Adapting the first-level methods (e.g. inference_detector()) does not work as the problem is deeply rooted inside the base-code of the models. So I figured out that I have to adapt the training- and test-pipelines.
But the custom pipeline tutorial doesn't explain how I can achieve that or at least I can't derive it from its content. During research in the issues here I came across issue #1861 where it was asked how to change backbone input from 3 to 6 channels. This seems to be a suitable solution for my problem to have [6, width, height]-shaped image tensors as input instead of [2, 3, width, height]. During the forward-function of my custom model I could then split it into 2 [3, 320, 240] tensors and use it as before and the batch-problem would not occur.
But neither in this issue nor in #6977 nor #4182 I can find a description about what to do and I haven't found the solution for creating such a custom pipeline by myself. That would be a great help. I also don't know how the img_norm_cfg get its values and how I would need to adopt this for 6 channels:
img_norm_cfg = dict( mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)The annotations-json includes only the colour images (and annotations only refer to colour images therefore), but they are working for both modals as they are aligned and a pixel in one modal refers to the same point in the real world as the pixel at the same position in the second modal image.
Summary: The new custom-backbone for 2 three-channel 2D-data is registered in registry and the adapted model is built by only adapting the backbone-dict in the config. Checkpoints are adapted accordingly. Data set is a custom data set with 2 same-sized images per instance. The reason for it not working is most likely an un-adapted pipeline config which I don't know how to customize correctly.