Implemented weighted-multi_input-[shortcut] layer with weights-normalization

[AlexeyAB]

Implemented weighted-multi_input-[shortcut] layer with weights-normalization, added:

New [shortcut] can:

• can take more than 2 input layers for adding: from = -2, -3 (and -1 by default)

• can multiply by weights:

  • the different input layers: weight per_feature (per_layer)
  • or the different input layers and channels: weight per_channel

• can normalize weights by using: avg_relu or softmax

The simplest example: yolov3-tiny_3l_shortcut_multilayer_per_feature_softmax.cfg.txt

---

• Fusion-layer for BiFPN for EfficientDet network: https://github.com/AlexeyAB/darknet/issues/4346

  [shortcut]
  from= -2, -3, 6            # means: -1, -2, -3, 6 - relative/absolute indexes of input layers
  weights_type= per_feature  # none (default), per_feature, per_channel
  weights_normalizion= relu  # none (default), relu, softmax
  activation= linear         # linear (default), leaky, rely, logistic, swish, mish, ...

---

• Original residual-connection:

  [shortcut]
  from= -5           # means: -1, -5 - relative indexes of input layers
  activation= linear

---

---

https://arxiv.org/abs/1911.09070v1

---

https://arxiv.org/abs/1911.09070v1

[Kyuuki93]

@AlexeyAB @WongKinYiu I made a network has 3 BiFPN blocks with P3~P5, just take a look for sure, darknet53-bifpn3.cfg.txt but even set input size to 320*320 and subdivision=32will get this error

CUDA Error Prev: an illegal memory access was encountered
CUDA Error Prev: an illegal memory access was encountered: Resource temporarily unavailable
darknet: ./src/utils.c:297: error: Assertion `0` failed.
Aborted (core dumped)

[AlexeyAB]

@Kyuuki93 Try to use the latest commit. I set

batch=64
subdivisions=16
width=320
height=320

and trained your cfg-file for 100 iterations successfully.

[AlexeyAB]

@Kyuuki93

There is bug in your cfg: use weights_type=per_feature instead of weights_type=per_feture

[AlexeyAB]

@Kyuuki93

Also you should not specify -1 layer, since it is set by default for all [shortcut] layers.

Just use

[shortcut]
from=61
weights_type=per_feature
weights_normalizion=relu
activation=leaky

instead of

[shortcut]
from=-1,61
weights_type=per_feature
weights_normalizion=relu
activation=leaky

[Kyuuki93]

@AlexeyAB cuda error was lead by weights_type=per_feture, my mistake.

Modified .cfg is here darknet53-bifpn3.cfg.txt which look like this

[Kyuuki93]

Comparison on my dataset, all cases with same training settings and used MS COCO detector pre-trained weights, only different in backbone

Model	AP@.5	AP@.75	precision(.7)	recall(.7)	inference time(416x416)
yolov3	91.79%	63.09%	0.95	0.71	13.25ms
csresnext50-panet	92.80%	64.16%	0.96	0.67	15.61ms
darknet53-bifpn3(P3-5)	91.74%	63.48%	0.95	0.71	15.25ms

ALL network has SPP-layer, inference time test on RTX 2080Ti, BiFPN block use

weights_type=per_feature
weights_normalizion=relu

darkenet53-bifpn3-spp got very similar performance with yolov3-spp, I think the reason could be

• ability of BiFPN*3(P3-5) was very close to FPN in yolov3-spp
• my dataset was too small to show different of two FPN

the option of next step could be

• darknet53-bifpn*N(P3-5)
• darknet53-bifpn*N(P2-5)
• darknet53-bifpn*N(P3-7)
• csresnext50-panet-bifpn*N(Px-Px)

but recently my machines was occupied by other task, I will try further experiment when GPUs got free

[AlexeyAB]

@Kyuuki93 Yes,

• Try to use more than 3xP
• Also try to use 3-5 BiFPN blocks

since BiFPN is not very expensive

[AlexeyAB]

@Kyuuki93 @WongKinYiu I just fixed weights_normalizion=softmax for [shortcut] layer. https://github.com/AlexeyAB/darknet/commit/14172d42b68cf9c81ca1150020475f3e79c82fab#diff-0c461530f46c81f7013a6eaec297ebcfR135

weights_normalizion=relu remains the same as earlier.

[WongKinYiu]

@AlexeyAB

Start re-train ASFF models now.

[AlexeyAB]

@WongKinYiu [shortcut] doesn't affect ASFF. And [shortcut] weights_normalizion=softmax even doesn't affect BiFPN.

• default ASFF uses [scale_channels] scale_wh=1
• default BiFPN uses [shortcut] weights_normalizion=relu

[AlexeyAB]

@Kyuuki93 @WongKinYiu Also I don't know what result of BiFPN (weights_normalizion=softmax) will be better, with max_val=0 or without it, between these 2 lines in these 2 places:

• https://github.com/AlexeyAB/darknet/blob/14172d42b68cf9c81ca1150020475f3e79c82fab/src/blas_kernels.cu#L705-L706
• https://github.com/AlexeyAB/darknet/blob/14172d42b68cf9c81ca1150020475f3e79c82fab/src/blas_kernels.cu#L781-L782

So you can test both cases on two small datasets.

[AlexeyAB]

@Kyuuki93 Hi,

Do you get any progress in BiFPN and BiFPN+ASFF?

[Kyuuki93]

Do you get any progress in BiFPN and BiFPN+ASFF?

Sorry, can't work this days because new year and new virus in china ...

[glenn-jocher]

@Kyuuki93 @AlexeyAB I'm interested in implementing a BiFPN head on top of darknet in https://github.com/ultralytics/yolov3 using https://github.com/AlexeyAB/darknet/files/4048909/darknet53-bifpn3.cfg.txt and benchmarking on COCO.

EfficientDet paper https://arxiv.org/pdf/1911.09070.pdf mentions 3 summation methods: "scalar (per-feature), a vector (per-channel), or a multi-dimensional tensor (per-pixel)". It seems they select scalar/per-feature for their implementation. Then I assume to add multiple 4D tensors, say of shape 16x256x13x13, would we have 2 scalar weights if done 'per_feature', and 2 vector weights of shape 1x256x1x1 if done 'per_channel'?

Also, I'm surprised softmax on the weights imparts such a slowdown (1.3X) in their paper, have you guys also observed this?

[AlexeyAB]

@glenn-jocher

Then I assume to add multiple 4D tensors, say of shape 16x256x13x13, would we have 2 scalar weights if done 'per_feature', and 2 vector weights of shape 1x256x1x1 if done 'per_channel'?

Yes.

1. per_feature (per input layer) - 1 float value for each input layer
2. per_channel - 1 float value per each channel from each layer

---

Also, I'm surprised softmax on the weights imparts such a slowdown (1.3X) in their paper, have you guys also observed this?

Only for training. And only for weighted-shortcut-layer. As result:

• for training ~1-10%
• for detection == 0% (since normalization can be done during initialization) https://github.com/AlexeyAB/darknet/blob/653eceb1a72641cf978734dc15257f8273f1ac94/src/network.c#L1112-L1162

---

Did you write a paper for Mosaic data augmentation?

[glenn-jocher]

@AlexeyAB ah I see. Actually no, I haven't had time to write a mosaic paper. It's too bad, because the results are pretty clear that it helps significantly.

Another thing I was wondering is I see in the BiFPN cfg only the head used weighted shortcuts, while the darknet53 backbone does not. If it helps the head then it may help the backbone as well no? Though to keep expectations in check, the EfficientDet paper only shows a pretty small +0.45 mAP bump from weighted vs non-weighted head.

One difference though is that currently the regular non-weighted shortcut layers effectively have weights=1 that sum to 2, 3 etc depending on the number of inputs. Isn't this a bit strange then that the head must be constrained to sum the weights to 1?

[AlexeyAB]

@glenn-jocher

If it helps the head then it may help the backbone as well no?

I have the same thoughts. We can replace each shortcut with a weighted-shortcut. May be it will get another +0.5 AP.

---

Isn't this a bit strange then that the head must be constrained to sum the weights to 1?

Batch-normalization try to do the same - moves the most values to range [0 - 1] - it increases mAP, speeds up the training, makes training more stable... https://medium.com/@ilango100/batch-normalization-speed-up-neural-network-training-245e39a62f85

[glenn-jocher]

@AlexeyAB ah of course, the BN after the Conv2d() following a shortcut will do this automatically, it completely slipped my mind. Good example. Ok, then I've taken @Kyuuki93's cfg, touched it up a bit (fixed the 'normalization' typo, reverted it to yolov3-spp.cfg default anchors and 80-class configuration, and implemented weighted shortcuts for all shortcut layers.)

I will experiment with a few different weighting techniques using my typical 27 epoch coco results and post the results later on this week hopefully. I suppose the tests should be:

1. yolov3-spp.cfg results (default)
2. yolov3-spp.cfg (all shortcuts = weighted shortcuts)
3. darknet53-bifpn3.cfg (all shortcuts in backbone and head = weighted shortcuts)

[AlexeyAB]

@glenn-jocher Also try to use this cfg-file that is made more similar to https://github.com/xuannianz/EfficientDet/blob/ccc795781fa173b32a6785765c8a7105ba702d0b/model.py

4. csresnext50-bifpn-optimal.cfg.txt (or try to use BiFPN-head from this cfg-file with darknet53 backbone)

Just for fair comparison, set all parameters in [net] and [yolo] to the same values in all 4 models.

[glenn-jocher]

@AlexeyAB thanks for the cfg! I've been having some problems with the route layers on these csresnext50 cfgs, maybe I can just copy the BiFPN part and attach it to darknet53, ah but the shortcut layer numbers will be different... ok maybe I'll just try and use it directly. Let's see...

[AlexeyAB]

@glenn-jocher Yes, you should change route layers and the first from= in weighted [shortcut] layers. Tomorow I will attach csdarknet53-bifpn-optimal.cfg

[WongKinYiu]

@AlexeyAB @glenn-jocher

I will get some free gpus in about 2~4 days. If you want to try weighted shortcut in backbone, i can train it on imagenet.

[AlexeyAB]

@WongKinYiu @glenn-jocher

Yes, it will be nice.

Try to train 2 Classifiers with weighted-shortcut layers:

1. csdarknet53-ws.cfg.txt (relu)

2. csresnext50-ws.cfg.txt (relu)

3. csresnext50-ws-mi2.cfg.txt (softmax) - preferably for training (multi-input weighted shortcut-layers with softmax normalization)

Which are similar to https://github.com/WongKinYiu/CrossStagePartialNetworks/blob/master/imagenet/results.md with mosaic, cutmix, mish, label smooth

---

Try to train these 2 models 416x416 with my BiFPN module which is more like a module by reference: https://github.com/xuannianz/EfficientDet/blob/ccc795781fa173b32a6785765c8a7105ba702d0b/model.py

1. csdarknet53-bifpn-optimal.cfg.txt

2. csresnext50-bifpn-optimal.cfg.txt

For comparison with: https://github.com/ultralytics/yolov3/issues/698#issuecomment-587378576

---

• Can you add AP | AP50 | AP75 for CSPResNeXt-50 optimal without any features?

• What is the Generic in this table?

• Also what is the difference between RFB and RFBN? https://github.com/WongKinYiu/CrossStagePartialNetworks/blob/master/coco/results.md#cspresnext-50-optimal

• And why RFB degrades AP? Is it because RFB without batch-normalization, while RFBN with one?

Model	Size	fps	AP	AP50	AP75	APS	APM	APL	cfg	weight
PANet-(SPP, CIoU)	512×512	44	42.4	64.4	45.9	23.2	45.5	55.3	-	-
PANet-(SPP, RFB, CIoU)	512×512	-	41.8	62.7	45.1	22.7	44.3	55.0	-	-

[WongKinYiu]

• Can you add AP | AP50 | AP75 for CSPResNeXt-50 optimal without any features? still training.

• What is the Generic in this table? it means apply learning rate, momentum... which is searched by generic algorithm. https://github.com/ultralytics/yolov3/issues/392

• Also what is the difference between RFB and RFBN? https://github.com/WongKinYiu/CrossStagePartialNetworks/blob/master/coco/results.md#cspresnext-50-optimal And why RFB degrades AP? Is it because RFB without batch-normalization, while RFBN with one? yes.

[AlexeyAB]

What is the Generic in this table? it means apply learning rate, momentum... which is searched by generic algorithm. https://github.com/ultralytics/yolov3/issues/392

So it is about the optimal Hyperparameters that are obtained by the Genetic algorithm.

[WongKinYiu]

yes

[glenn-jocher]

@AlexeyAB @WongKinYiu ah yes, this is a "Genetic Algorithm", not "Generic". The current hyperparameters I found with it are tuned for yolov3-spp.cfg though, and it's unclear to me how well they generalize to other architectures/datasets. This could be one of the reasons why yolov3-spp trains well on ultralytics/yolov3 in comparison to others like csresnext-panet etc.

I completed the 3 runs (27 COCO epochs). Weighted shortcut only helped very slightly, and BiFPN actually worsened my results slightly. In general all 3 were extremely similar.

	Name	mAP @0.5	mAP @0.5:0.95	Comments
64	yolov3-spp.cfg	50.2	30.9	baseline 63M
65	yolov3-spp.cfg	50.3	31.0	weighted shortcuts 63M
66	darknet53-bifpn3.cfg	49.7	30.8	weighted shortcuts 70M
67	csdarknet53-bifpn-optimal.cfg			results pending 53M

I created a PyTorch weighted shortcut layer for this test. I did not normalize my shortcut weights with softmax or the "fast fusion" method from the paper, instead I used a simple sigmoid method to address the possible instability. I don't know if this difference affects things significantly or not. I suppose I should have just done a strict softmax. I can do that in the future.

class weightedFeatureFusion(nn.Module):  # weighted sum of 2 or more layers https://arxiv.org/abs/1911.09070
    def __init__(self, layers):
        super(weightedFeatureFusion, self).__init__()
        self.n = len(layers) + 1 # number of layers
        self.layers = layers  # layer indices
        self.w = torch.nn.Parameter(torch.zeros(self.n))  # layer weights

    def forward(self, x, outputs):
        w = torch.sigmoid(self.w) * (2 / self.n)  # sigmoid weights (0-1)
        if self.n == 2:
            return x * w[0] + outputs[self.layers[0]] * w[1]
        elif self.n == 3:
            return x * w[0] + outputs[self.layers[0]] * w[1] + outputs[self.layers[1]] * w[2]
        else:
            raise ValueError('weightedFeatureFusion() supports up to 3 layer inputs, %g attempted' % self.n)

UPDATE: @AlexeyAB I will run csdarknet53-bifpn-optimal.cfg to compare with above. I can't run csresnext50-bifpn-optimal.cfg yet due to the resnext bug with the grouped convolutions in my repo, but I'll try to get that sorted out this week or next week as well.

[glenn-jocher]

I looked at the actual fusion weights for yolov3-spp.cfg using the above process, and nothing seems out of the ordinary. They don't quite sum to 1 but not sure if it really matters.

    for x in list(model.parameters()):
        if len(x) == 2:
            print(torch.sigmoid(x).detach())

tensor([0.68042, 0.24355])
tensor([0.43275, 0.43802])
tensor([0.63490, 0.33800])
tensor([0.38976, 0.61641])
tensor([0.45976, 0.52418])
tensor([0.49271, 0.47769])
tensor([0.48430, 0.46104])
tensor([0.48992, 0.43543])
tensor([0.48655, 0.41508])
tensor([0.47568, 0.38939])
tensor([0.42831, 0.35085])
tensor([0.40784, 0.60700])
tensor([0.45018, 0.53021])
tensor([0.46398, 0.47831])
tensor([0.48237, 0.43581])
tensor([0.49701, 0.41102])
tensor([0.49047, 0.40242])
tensor([0.47470, 0.38543])
tensor([0.47063, 0.34986])
tensor([0.42593, 0.57570])
tensor([0.46498, 0.51103])
tensor([0.49873, 0.47298])
tensor([0.53441, 0.46724])

[glenn-jocher]

@AlexeyAB I tried csdarknet53-bifpn-optimal.cfg.txt, but I'm getting the same error related to the channel sizes as I saw in resnext. I don't think this a bug, I think I simply need to update my code per https://github.com/ultralytics/yolov3/issues/698#issuecomment-570427687, which I still haven't done yet.

Also there is a typo in 'weights_normalizion'. I also saw the new model is reduced in size, 53M params vs I think 63M for yolov3-spp.cfg, which is more in line with the efficientdet paper, as the old darknet53-bifpn cfg I used actually increased the parameter count.

UPDATE: I realized the error message is actually about the shortcut layers. When shortcutting layers -1 to 85 I get these shapes: -1, torch.Size([16, 1024, 26, 26]) 85, torch.Size([16, 512, 26, 26])

This error is coming from line 835 in your cfg:

[WongKinYiu]

Also there is a typo in 'weights_normalizion'.

maybe it becuz it is defined as weights_normalizion in parser.c and blas.c.

[AlexeyAB]

@glenn-jocher

Also there is a typo in 'weights_normalizion'.

What do you mean?

---

I did not normalize my shortcut weights with softmax or the "fast fusion" method from the paper, instead I used a simple sigmoid method to address the possible instability.

In ASFF the Softmax works much better than ReLU/Logistic, so may be we can try to train weights_normalizion=softmax in BiFPN too, since it doesn't affect on inference speed https://github.com/AlexeyAB/darknet/issues/4382#issuecomment-564823931

But logically, softmax should give a greater advantage to ASFF than to BiFPN (since in ASFF the multipliers are different for each inference and the sum of multipliers is different every time, so they should be normalized dynamically). So this is not a priority.

---

Weighted shortcut only helped very slightly

Since new shortcut-layer not only weighted, but also multi-input, then we can fuse all shortcut outputs with the same resolution before each subsambling.

I will try to make such model and compare FPS:

---

I also saw the new model is reduced in size, 53M params vs I think 63M for yolov3-spp.cfg

I will run csdarknet53-bifpn-optimal.cfg to compare with above. I can't run csresnext50-bifpn-optimal.cfg yet due to the resnext bug with the grouped convolutions in my repo, but I'll try to get that sorted out this week or next week as well.

I used grouped-conv in the csdarknet53-bifpn-optimal.cfg model.

---

UPDATE: I realized the error message is actually about the shortcut layers. When shortcutting layers -1 to 85 I get these shapes: -1, torch.Size([16, 1024, 26, 26]) 85, torch.Size([16, 512, 26, 26])

This error is coming from line 835 in your cfg:

This is a PRN (partial residual connections) when two tensors with different number of channels are added elementwise.

[AlexeyAB]

@WongKinYiu

About my suggestion:

Since new shortcut-layer not only weighted, but also multi-input, then we can fuse all shortcut outputs with the same resolution

I added csresnext50-ws-mi2.cfg.txt - preferably for training (multi-input weighted shortcut-layers with softmax normalization)

There: https://github.com/AlexeyAB/darknet/issues/4662#issuecomment-587490873

So may be better to train:

2. csresnext50-ws.cfg.txt

and

3. csresnext50-ws-mi2.cfg.txt

whitout 1. csdarknet53-ws.cfg.txt if you don't have enough GPUs.

---

On RTX 2070

• csresnext50.cfg.txt - 68.0 FPS
• csresnext50-ws.cfg.txt - 67.5 FPS
• csresnext50-ws-mi2.cfg.txt - 67.5 FPS

[WongKinYiu]

@AlexeyAB

OK.

[glenn-jocher]

This is a PRN (partial residual connections) when two tensors with different number of channels are added elementwise.

@AlexeyAB ah I see, PRN. Output always matches input shape. Ok perfect, if I can fix this then cspresnext and your new darknet53 cfgs will both work. For the 3 cases:

• Case A larger 'from' should be maxpooled to fit? Or perhaps downsampled using nearest value? Or we could also simply slice the feature from 0 to the same size and input (we lose some data this way).
• Case B @WongKinYiu already explained this to me, the solution is to zero pad (all zeros on the right side, following the existing features) the smaller 'from' feature to match the larger one right?
• Case C is already handled correctly.

[AlexeyAB]

@glenn-jocher

Case A larger 'from' should be maxpooled to fit? Or perhaps downsampled using nearest value? Or we could also simply slice the feature from 0 to the same size and input (we lose some data this way).

There is no maxpool/downsampling. Just take the first C channels.

Or we could also simply slice the feature from 0 to the same size and input (we lose some data this way).

Yes.

[glenn-jocher]

@AlexeyAB @WongKinYiu I've updated my shortcut layer now to correctly handle all new situations: weighted and non-weighted, from smaller features (by zero pad right) and from larger features (by slicing), for any number of input layers.

I should be able to train the 2 new cfg files now. Do these parameter and layer counts seem correct?

• yolov3-spp.cfg (for reference): 225 layers, 6.29987e+07 parameters
• csdarknet53-bifpn-optimal.cfg: 338 layers, 4.99935e+07 parameters
• csresnext50-bifpn-optimal.cfg: 299 layers, 4.25729e+07 parameters

[WongKinYiu]

@glenn-jocher great!

[glenn-jocher]

I noticed that the shortcut zeropad operations in my implementation are slower (about 100X slower on CPU) than the slicing operations. The times (seconds) below are for 1000 shortcut operations for darknet53-bifpn (for different shortcut layers). I tried a different method: instead of padding the smaller 'from' feature, I added the 'from' feature to a sliced input, which should be mathematically equivalent. I didn't see any GPU speedup though, so this looks like its not a primary cause of slowdown with these new models, but the change did help reduce GPU RAM usage a bit.

zeropad 0.2164621353149414
zeropad 0.6748669147491455
zeropad 2.6399528980255127
slice 0.0072019100189208984
slice 0.006873130798339844
slice 0.006844043731689453
slice 0.006783962249755859
slice 0.007230997085571289
zeropad 0.1068871021270752
zeropad 0.6746640205383301
slice 0.0068128108978271484
zeropad 0.1016230583190918

The new models are training now, should be done in a day or two. Unfortunately they are quite a bit slower than yolov3-spp, about 2.5X slower (22 min epoch vs 50-60 min epoch) with multi-scale. My command is here.

python3 train.py --data coco2014.data --img-size 416 608 --epochs 27 --batch 12 --accum 6 --weights '' --device 0 --cfg csdarknet53-bifpn-optimal.cfg --nosave --name 67 --multi

[AlexeyAB]

Do these parameter and layer counts seem correct?

Yes.

[glenn-jocher]

@AlexeyAB I'm seeing poor results here after 15 epochs for csresnext50-bifpn-optimal.cfg (results67 below), so I've cancelled the training.

You have the anchor order for both csresnext50-bifpn-optimal.cfg and csdarknet53-bifpn-optimal.cfg the same as yolov3-spp.cfg, with mask=0,1,2 for the last output layer. Perhaps this order should be reversed as in darknet53-bifpn3.cfg?

I've also seperately confirmed that the majority of the slowdown is due to the grouped convolutions. When I set groups=1 for all convolutions in csdarknet53-bifpn-optimal.cfg my parameter count increases (from 50M to 57M), but the training speeds improves from 51min/epoch to 31min/epoch.

ID	Name	mAP@ 0.5	mAP@ 0.5:0.95	Comments
64	yolov3-spp.cfg	50.2	30.9	baseline 63M
65	yolov3-spp.cfg	50.3	31.0	weighted shortcuts 63M
66	darknet53-bifpn3.cfg	49.7	30.8	weighted shortcuts 70M
67	csdarknet53-bifpn-optimal.cfg	-	-	cancelled 50M
69	csdarknet53-bifpn-optimal.cfg	41.5	23.8	groups=1 57M
70	csresnext50-bifpn-optimal.cfg	44.0	25.9	groups=1 94M

[AlexeyAB]

@glenn-jocher

I'm seeing poor results here after 15 epochs for csresnext50-bifpn-optimal.cfg (results67 below), so I've cancelled the training.

Is it better than darknet53-bifpn3.cfg ? It would be nice to compare with darknet53-bifpn3.cfg so we will understand whether we are moving in the right direction.

What does it mean result64 - 67 on your charts?

I've also seperately confirmed that the majority of the slowdown is due to the grouped convolutions. When I set groups=1 for all convolutions in csdarknet53-bifpn-optimal.cfg my parameter count increases (from 50M to 57M), but the training speeds improves from 51min/epoch to 31min/epoch.

Try to train csresnext50-bifpn-optimal.cfg with groups=1

---

You have the anchor order for both csresnext50-bifpn-optimal.cfg and csdarknet53-bifpn-optimal.cfg the same as yolov3-spp.cfg, with mask=0,1,2 for the last output layer.

Yes.

Perhaps this order should be reversed as in darknet53-bifpn3.cfg?

No.

[glenn-jocher]

@AlexeyAB I've added a table to https://github.com/AlexeyAB/darknet/issues/4662#issuecomment-589316861 to explain the run numbers. darknet53-bifpn3.cfg is there as run 66.

Ok, yes I will re-run csdarknet53-bifpn-optimal.cfg with groups=1 for all convolutions, and I will try csresnext50-bifpn-optimal.cfg with groups=1 after that.

[AlexeyAB]

@glenn-jocher

Case A larger 'from' should be maxpooled to fit? Or perhaps downsampled using nearest value? Or we could also simply slice the feature from 0 to the same size and input (we lose some data this way).

We do not get all the information from the from= But in the main path we don't lose any information. So actually we dont lose any information.

I tried a different method: instead of padding the smaller 'from' feature, I added the 'from' feature to a sliced input, which should be mathematically equivalent.

You should not crop layer -1 otherwise you will lose information and the accuracy will drop dramatically.

Just note that A and B are not equivalent:

[glenn-jocher]

@AlexeyAB yes don't worry, I'm 99% sure I'm handling the new shortcuts correctly. The change I made was to Case B. Rather than zero-pad from as we first talked about, I add from to a sliced input. Only the sliced values in input are affected, the rest of input remains unchanged (same as adding zeros to it).

The original pad pseudocode would be:

output = input * weight0 + [from , zero_padding] * weight1

And the updated pseudocode that avoids padding looks this for ch from channels.

output = input * weight0
output[:, 0:ch] = output[:, 0:ch] + from * weight1

The actual code looks like this.

       # Fusion
        nc = x.shape[1]  # input channels
        for i in range(self.n - 1):
            a = outputs[self.layers[i]]  # feature to add
            ac = a.shape[1]  # feature channels
            dc = nc - ac  # delta channels

            # Adjust channels
            if dc > 0:  # slice input
                x[:, :ac] = x[:, :ac] + (a * w[i + 1] if self.weight else a)
            elif dc < 0:  # slice feature
                x = x + (a[:, :nc] * w[i + 1] if self.weight else a[:, :nc])
            else:  # same shape
                x = x + (a * w[i + 1] if self.weight else a)

[glenn-jocher]

@AlexeyAB @WongKinYiu runs are all done. None of the updates helped the results unfortunately. The higher the groupings the fewer parameters, but also the worse the results apparently.

I only did 10% coco training here, so its possible full training might return a different results, but typically 10% training results correlate fairly well with full training results. As a reference, run 64 acheives 50.2 mAP@0.5 at 10% of training (table below), and 61.5mAP@0.5 mAP with full training.

I'm pretty confused then. If the bifpn cfgs are correct, then we are unable to replicate the improvement in the literature, at least on the ultralytics repo. But with this darknet repo, grouped convolutions do show improvement?

ID	Name	mAP@ 0.5	mAP@ 0.5:0.95	Comments
64	yolov3-spp.cfg	50.2	30.9	baseline 63M
65	yolov3-spp.cfg	50.3	31.0	weighted shortcuts 63M
66	darknet53-bifpn3.cfg	49.7	30.8	weighted shortcuts 70M
67	csdarknet53-bifpn-optimal.cfg	-	-	cancelled 50M
69	csdarknet53-bifpn-optimal.cfg	41.5	23.8	groups=1 57M
70	csresnext50-bifpn-optimal.cfg	44.0	25.9	groups=1 94M

[WongKinYiu]

@glenn-jocher Hello,

I start training them few days ago, will get results after 4 weeks.

[AlexeyAB]

@glenn-jocher @WongKinYiu

If the bifpn cfgs are correct, then we are unable to replicate the improvement in the literature, at least on the ultralytics repo.

Perhaps automatic differentiation in Ultralytics/Pytorch incorrectly performs back propagation for normalization in your weighted-shortcut layers.

I used the same gradient as in the ASFF: https://arxiv.org/pdf/1911.09516v2.pdf

https://github.com/AlexeyAB/darknet/blob/6fb817f68b471da21c04f8bd5ffef0d3c4d67c30/src/blas.c#L181-L187

Then it performs such back propagation: https://github.com/AlexeyAB/darknet/blob/6fb817f68b471da21c04f8bd5ffef0d3c4d67c30/src/blas.c#L216-L217

---

Also will waiting for results of ASFF(softmax), that shows improvement for small custom datasets: https://github.com/AlexeyAB/darknet/issues/3874#issuecomment-561064425 (just this is strange that ASFF + RFB(batchnorm=0) shows the best AP75 accuracy, while RFB(batchnorm=1) degrades AP75)

But on MSCOCO the ASFF(softmax) leads to Nan: https://github.com/AlexeyAB/darknet/issues/3772#issuecomment-580518150

[glenn-jocher]

@AlexeyAB I forgot you were coding gradients yourself, that's very impressive!

Incorrect gradients can always be an issue. I've seen problems arise with in-place operations (pytorch can't backprop these properly), but I made sure to avoid these in my shortcut module, and I also tested the new setup with anomaly detection once to check, by setting torch.autograd.set_detect_anomaly(True) before running (no anomalies were found). But the new shortcut operations actually help a bit on yolov3-spp.cfg (see table below), so at least for same shape weighted shortcut operations everything is going well.

ID	Name	mAP@ 0.5	mAP@ 0.5:0.95	Comments
64	yolov3-spp.cfg	50.2	30.9	baseline 63M
65	yolov3-spp.cfg	50.3	31.0	weighted shortcuts 63M

The main mAP decreases appear to come from groups>1 in the convolutions, but also from the new heads. I suppose a simple test would be for me to take yolov3-spp.cfg, and change all the convolutions to groups=2, groups=4, groups=8 and add these results to the table? That would isolate the effect of increased groups.

[AlexeyAB]

@WongKinYiu

I fixed gradient for BiFPN and weighted-shortcut. So try to re-run training of these models with new code: https://github.com/AlexeyAB/darknet/issues/4662#issuecomment-587490873

[AlexeyAB]

@WongKinYiu

• 3 fixes for BiFPN / weighted-[shortcut]: https://github.com/AlexeyAB/darknet/commit/cfa40fe890d0fc7d100cfec2ee4976bf9838cb30 and https://github.com/AlexeyAB/darknet/commit/3cb9125b95a860694cfafa5b3408aa6123749e1e and https://github.com/AlexeyAB/darknet/commit/f6baa62c9b6151b9f615a1e56434d237553fd4af

• 1 fix for ASFF: https://github.com/AlexeyAB/darknet/commit/c83c21459b3afeb84758dcc602fc612cf2139a69

---

@glenn-jocher Did you try to use ASFF on ultralytics/Pytorch? https://github.com/ruinmessi/ASFF