Closed LukeAI closed 4 years ago
@LukeAI hi, thanks for the feedback! Off the top of my head I think we may not support some of the layers there (https://github.com/ultralytics/yolov3/issues/631#issuecomment-563224735). Do you have an exact *.cfg file that you saw improvements with?
Is this complementary to Gaussian YOLO, can they both be used togethor? So this would be a replacement of the darknet53 backbone with a ReseXt50 backbone?
It's too bad @WongKinYiu didn't do the modifications directly in this repo :)
@WongKinYiu I'd like to implement this cfg in ultralytics/yolov3: https://github.com/WongKinYiu/CrossStagePartialNetworks/blob/master/cfg/csresnext50-panet-spp.cfg
The only new field I see is 'groups' in the convolution layers. Are there other new fields I didn't see? https://github.com/WongKinYiu/CrossStagePartialNetworks/blob/ff762e58750a2261d64855ac9c3a3ea1a993a24a/cfg/csresnext50-panet-spp.cfg#L383-L390
Do you know where I would slot groups into the PyTorch nn.Conv2d()
module?
https://github.com/ultralytics/yolov3/blob/07c1fafba832ef83fca70576e04cef48686f72a1/models.py#L22-L33
@LukeAI @WongKinYiu I've added import of 'groups' into the Conv2d() definition in https://github.com/ultralytics/yolov3/commit/3bfbab7afd5850b4f21b73dd3184374f47eb1d98. Is this sufficient to run CSPResNeXt50-PANet-SPP? @LukeAI can you git pull
this repo and try with the cfg?
yolov3-spp.cfg has 17 unique fields in it's cfg:
17 ['type', 'batch_normalize', 'filters', 'size', 'stride', 'pad', 'activation', 'from', 'layers', 'mask', 'anchors', 'classes', 'num', 'jitter', 'ignore_thresh', 'truth_thresh', 'random']
csresnext50-panet-spp.cfg has 18 unique fields. It seems group is the only newcomer. Ok, so this repo should now fully support csresnext50-panet-spp.cfg @LukeAI.
18 ['type', 'batch_normalize', 'filters', 'size', 'stride', 'pad', 'activation', 'layers', 'groups', 'from', 'mask', 'anchors', 'classes', 'num', 'jitter', 'ignore_thresh', 'truth_thresh', 'random']
@WongKinYiu I am getting an error in 3 of the shortcut layers when running csresnext50-panet-spp.cfg. They are trying to add tensors of torch.Size([1, 64, 104, 104])
from -4 layers previous with incorrectly sized tensors of torch.Size([1, 128, 104, 104])
.
# models.py line 260:
elif mtype == 'shortcut':
try:
x = x + layer_outputs[int(mdef['from'])]
except:
print(i, x.shape, layer_outputs[int(mdef['from'])].shape)
x = layer_outputs[int(mdef['from'])]
# excepted layers:
# 8 torch.Size([1, 128, 104, 104]) torch.Size([1, 64, 104, 104])
# 12 torch.Size([1, 128, 104, 104]) torch.Size([1, 64, 104, 104])
# 16 torch.Size([1, 128, 104, 104]) torch.Size([1, 64, 104, 104])
Change filters=128
to filters=64
on csresnext50-panet-spp.cfg lines 79, 110, 141. Then all the shapes combine correctly in this repo. Implemented in https://github.com/ultralytics/yolov3/commit/86588f15796a47faa2deb572da0bc62d15fe6c9c. Not sure if this is a correct modification according to the original cfg designer.
@glenn-jocher Hello,
In pytorch, do zero padding to same size, then add. for example, in pytorch 0.4:
if residual_channel != shortcut_channel:
padding = torch.autograd.Variable(torch.cuda.FloatTensor(batch_size, residual_channel - shortcut_channel, featuremap_size[0], featuremap_size[1]).fill_(0))
out += torch.cat((shortcut, padding), 1)
else:
out += shortcut
@glenn-jocher
For convenient, I change line 57 to filters=128
instead of filter=64
to make it has consistent filter number. Here are csresnext50c.cfg and csresnext50c.conv.80.
@WongKinYiu @LukeAI @AlexeyAB I trained csresnext50-panet-spp.cfg https://github.com/ultralytics/yolov3/commit/86588f15796a47faa2deb572da0bc62d15fe6c9c against default yolov3-spp.cfg for 27 COCO epochs at 416 (10% of full training), but got worse results at a slower speed. I ran yolov3-spp3.cfg (see https://github.com/ultralytics/yolov3/issues/694) with slightly worse results as well. Commands to reproduce:
git clone https://github.com/ultralytics/yolov3
bash yolov3/data/get_coco_dataset_gdrive.sh
cd yolov3
python3 train.py --epochs 27 --weights '' --cfg yolov3-spp.cfg --name 113
python3 train.py --epochs 27 --weights '' --cfg yolov3-spp3.cfg --name 115
python3 train.py --epochs 27 --weights '' --cfg csresnext50-panet-spp.cfg --name 121
mAP @0.5...0.95 |
mAP @0.5 |
time (hrs) to 27 epochs |
|
---|---|---|---|
yolov3-spp.cfg |
29.7 | 49.5 | 12.7 |
yolov3-spp3.cfg |
29.1 | 49.0 | 13.5 |
csresnext50-panet-spp.cfg https://github.com/ultralytics/yolov3/commit/86588f15796a47faa2deb572da0bc62d15fe6c9c |
25.9 | 44.2 | 28.3 |
csresnext50-panet-spp.cfg zero-pad TODO? |
|||
csresnext50c.cfg TODO? |
If you guys have time and are good with PyTorch please feel free to clone this repo and try the https://github.com/WongKinYiu/CrossStagePartialNetworks/ implementations yourself. I'd really like to exploit some of the research there but I don't have time. We are getting excellent results with our baseline yolov3-spp.cfg from scratch (40.9mAP@0.5...0.95, 60.9mAP@0.5 see https://github.com/ultralytics/yolov3#map), so if the improvements are relative, then they should help here also I assume.
ok, i ll try to install this repo.
so all of ur training do not use imagenet pre-trained model?
@WongKinYiu ok great! No I don't use any pre-trained model for the initial weights. In an earlier test I found that starting from darknet53.conv.74 produced worse mAP after 273 epochs than starting from randomly initialized weights. For quick results (a day or less of training) yes, the imagenet trained weights will help, but for longer training I found they hurt.
To reproduce:
git clone https://github.com/ultralytics/yolov3
bash yolov3/data/get_coco_dataset_gdrive.sh
cd yolov3
python3 train.py --epochs 273 --weights darknet53.conv.74 --cfg yolov3-spp.cfg --name 41
python3 train.py --epochs 273 --weights '' --cfg yolov3-spp.cfg --name 42
mAP @0.5...0.95 |
mAP @0.5 |
|
---|---|---|
results41: 416 multiscale to 273 epochs (darknet53.conv.74 start) | 56.8 | 36.2 |
results42: 416 multiscale to 273 epochs (random start) | 57.5 | 37.1 |
@glenn-jocher
Thanks for your reply. PANet need more training epochs to converge when compare with YOLOv3.
Do your models are trained using single GPU?
@WongKinYiu yes I typically train them on one 2080Ti or V100, which usually do about 50 epochs per day with the default settings (5 days to train COCO). See https://github.com/ultralytics/yolov3#speed for training speeds. Multi-GPU can also be used.
To get the best mAPs though --multi-scale
must be used, which adds about 50% more training time (7-8 days on 1 GPU). This is why I usually test changes on 27 epochs.
@glenn-jocher
Thanks for your reply. PANet need more training epochs to converge when compare with YOLOv3.
Should I try csresnext50c.cfg?
UPDATE: I put it in, but there are new layers again :)
python3 train.py --epochs 27 --weights '' --cfg csresnext50c.cfg --name 122
Warning: Unrecognized Layer Type: avgpool
Warning: Unrecognized Layer Type: softmax
@glenn-jocher
No, if train from scratch, i think u will get similar results. panet has additional path than fpn, so it need more epochs.
oh, it is becuz csresnext50c.cfg is for imagenet classifier.
@glenn-jocher
No, if train from scratch, i think u will get similar results. panet has additional path than fpn, so it need more epochs.
oh, it is becuz csresnext50c.cfg is for imagenet classifier.
Oh, haha, ok I'll leave csresnext50c.cfg alone then.
@glenn-jocher
start training...
do u use python3 train.py --epochs 273 --weights '' --cfg yolov3-spp.cfg --name 42
to get 40.9 AP?
@WongKinYiu the exact training command to get to 40.9 AP with one GPU is:
python3 train.py --weights '' --epochs 273 --batch 16 --accumulate 4 --multi --pre
If you use multi-GPU though you will have more memory available, so you can use a larger --batch --accumulate combination to get to 64 like 32x2, or even 64x1:
python3 train.py --weights '' --epochs 273 --batch 32 --accumulate 2 --multi --pre
yolov3-spp.cfg
is the default cfg, so you don't need to supply it above (but you can). The --pre
argument performs one epoch of biasing the yolo output neurons before training starts. See https://github.com/ultralytics/yolov3/issues/460
my gpu ram is not enough even though i set --batch 16 --accumulate 4 --multi --pre
.
i will borrow other gpu for training.
@glenn-jocher
I trained csresnext50-panet-spp.cfg 86588f1 against default yolov3-spp.cfg for 27 COCO epochs at 416 (10% of full training), but got worse results at a slower speed.
This is weird, did you measure speed on GPU? And what FPS/ms did you get for SPP vs CSP?
Have you tried converting an already trained on Darknet model CSPResNeXt50-PANet-SPP (cfg / weights) to ultralytics (pytorch), and did you get better mAP and better speed?
Or does this inconsistency interfere with this conversion? https://github.com/ultralytics/yolov3/issues/698#issuecomment-563466452
@AlexeyAB Hello,
I think slow speed is talking about training speed. training of group convolution is slower than training of conventional convolution.
@AlexeyAB @WongKinYiu if I run test.py on the two trained models, this applies inference (and NMS) on the 5000 images in 5k.txt. This takes 138 seconds with yolov3-spp.cfg on a P4 GPU, and 139 seconds with csresnext50-panet-spp.cfg. Ah interesting, so the inference speed is nearly identical, but the training speed takes twice as long.
So is the CSPResNeXt50-PANet-SPP operational? And does it provide better results? I am looking more into it right now. And reading the article.
my gpu ram is not enough even though i set
--batch 16 --accumulate 4 --multi --pre
. i will borrow other gpu for training.
@WongKinYiu I forgot to mention, you should install Nvidia Apex for mixed precision training with this repo. It increases speed substantially and reduces memory requirements substantially. Once installed correctly you should see this:
@glenn-jocher
Yes, I have installed apex.
Now I training with --multi
with scale 320~608.
@glenn-jocher Hello,
I would like to know why --pre
need a little bit more gpu memory than without using it.
@WongKinYiu ahhh this is interesting, I had not realized that. There is a memory leak when invoking train.py repeatedly, which is very obvious when running hyperparameter evolution as train.py is called repeatedly in a foor loop https://github.com/ultralytics/yolov3/issues/392#issuecomment-565475680, but I did not realize --pre also causes this. This makes sense though, as it is calling train.py once to train the output biases for one epoch, then calling it again for actual training. How much extra memory is this using?
Is this complementary to Gaussian YOLO, can they both be used togethor?
I independently found good improvements ~+3mAP with Gaussian-Yolo and also cspresnext50-pan-spp vs. yolov3-spp - but I got pretty bad results when I tried combining them (-10mAP) - this may be because: (1) I made a mistake (2) The features that are useful to gaussian-yolo are quite different to the features that are useful for yolo so training a network with a gaussian-yolo head from pretrained weights from a non-gaussian head gives poor results (3) I need to tune the hyper-parameters more (I tried 3 different learning rates but no dice) (4) for some subtle reason these features just don't play well together - seems unlikely to me though.
@WongKinYiu have you tried Gaussian with cspresnext-pan-spp? Do you have any thoughts or results?
I think you need more iterations for warmup when combine cspresnext50-pan-spp with gaussian-yolo (I have no gpus to test it currently).
In my experiments, when combining cspresnext50-pan-spp with gaussian-yolo, the precision drops and recall improves. And the strange thing is that the loss become lager after 200k epochs.
@glenn-jocher @AlexeyAB
i do some optimization of hyper-parameter, including iou_thresh, ciou, .... the new results training by darknet(AlexeyAB) are as follows:
model | size | AP | AP50 | AP75 |
---|---|---|---|---|
CSPResNeXt50-PANet-SPP | 512x512 | 42.4 | 64.4 | 45.9 |
CSPResNeXt50-PANet-SPP | 608x608 | 43.2 | 65.4 | 47.0 |
@WongKinYiu ah those are very very good!!
@glenn-jocher Hello,
@WongKinYiu @AlexeyAB I tried to run detections with the trained weights and csresnext50-panet-spp-original-optimal.cfg as below, but I get the same issue regarding padding as before https://github.com/ultralytics/yolov3/issues/698#issuecomment-563521134. I tried to pad these shortcut layers with zeros at dimension 1, but then another error appears which seems to do with the groupings. Do you have any idea what this might be?
Namespace(cfg='cfg/csresnext50-panet-spp-original-optimal.cfg', classes=None, conf_thres=0.3, device='', fourcc='mp4v', half=False, img_size=416, iou_thres=0.5, names='data/coco.names', output='output', save_txt=False, source='data/samples', view_img=False, weights='csresnext50-panet-spp-original-optimal_final.weights')
Using CPU
image 1/2 data/samples/bus.jpg: 8 torch.Size([1, 128, 104, 80]) torch.Size([1, 64, 104, 80])
Traceback (most recent call last):
File "/Users/glennjocher/PycharmProjects/yolov3/detect.py", line 176, in <module>
detect()
File "/Users/glennjocher/PycharmProjects/yolov3/detect.py", line 83, in detect
pred = model(img)[0]
File "/Users/glennjocher/opt/anaconda3/envs/pn1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/Users/glennjocher/PycharmProjects/yolov3/models.py", line 248, in forward
x = module(x)
File "/Users/glennjocher/opt/anaconda3/envs/pn1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/Users/glennjocher/opt/anaconda3/envs/pn1/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/Users/glennjocher/opt/anaconda3/envs/pn1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/Users/glennjocher/opt/anaconda3/envs/pn1/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
return self.conv2d_forward(input, self.weight)
File "/Users/glennjocher/opt/anaconda3/envs/pn1/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 128 64 1 1, expected input[1, 128, 104, 80] to have 64 channels, but got 128 channels instead
@glenn-jocher Hello,
I think it is because of that the output shape of shorcut layer in ultralytics
is same as from
layer, and the output shape of shorcut layer in darknet
is same as -1
layer.
I already fixed the zero padding problem in ultralytics
, i will share you the code after finish my breakfast.
and some of cspne(x)t
-based model training by ultralytics
will finish next week.
but maybe the group convolution of ultralytics
and darknet
are different.
i have tried to convert cspdensenet
to .pt, and it works.
however, when i convert cspresnext
to .pt, it detects nothing.
@WongKinYiu the error is being produced in the first conv2d layer after the first shortcut layer. For a batch-size 1 image of size 416x320 the shapes look like this going into the error:
0 convolutional torch.Size([1, 3, 416, 320])
1 maxpool torch.Size([1, 64, 208, 160])
2 convolutional torch.Size([1, 64, 104, 80])
3 route torch.Size([1, 128, 104, 80])
4 convolutional torch.Size([1, 64, 104, 80])
5 convolutional torch.Size([1, 64, 104, 80])
6 convolutional torch.Size([1, 128, 104, 80])
7 convolutional torch.Size([1, 128, 104, 80])
8 shortcut torch.Size([1, 128, 104, 80])
9 convolutional torch.Size([1, 128, 104, 80])
update: I used this code to temporarily patch the shape issue in models.py L261:
elif mtype == 'shortcut':
b = layer_outputs[int(mdef['from'])]
if b.shape == x.shape:
x = x + b
else:
pad = max(x.shape[i] - b.shape[i] for i in range(len(x.shape))) // 2
x = x + F.pad(b, pad=[0, 0, 0, 0, pad, pad])
print(i, x.shape, b.shape)
@glenn-jocher
yes, becuz the output_filters
tell next convolution the input size has 64 channel in this case.
https://github.com/ultralytics/yolov3/blob/master/models.py#L64
@glenn-jocher
yes, becuz the
output_filters
tell next convolution the input size has 64 channel in this case. https://github.com/ultralytics/yolov3/blob/master/models.py#L64
Ah, yes, I understand now. Ok, this is going to need a bit of cleanup before csresnext50-panet-spp-original-optimal.cfg
runs correctly, but I think I can do this in the next couple days. One important question though is, when I am shortcutting a size [1, 64, 104, 80] to a size [1, 128, 104, 80] as in the example, do I pad dimension 1 with 64 zeros at the end, 64 zeros at the beginning, or 32 zeros before and after (reflect pad 32) to bring the smaller size up to [1, 128, 104, 80]?
@glenn-jocher
in darknet implementation, the equivalent is padding at the end. https://github.com/AlexeyAB/darknet/blob/master/src/blas.c#L83-L94
@glenn-jocher
your code is more clean than mine, just put my code here for your reference.
change filters from from
layer to -1
layer.
elif mdef['type'] == 'shortcut': # nn.Sequential() placeholder for 'shortcut' layer
filters = output_filters[i]
layer = int(mdef['from'])
routs.extend([i + layer if layer < 0 else layer])
currently i just use 2x2 pooling when w
and h
are different, it should be
https://github.com/AlexeyAB/darknet/blob/master/src/blas.c#L73-L74
i have not implemented the code for the case which channel of from
layer is larger than the -1
layer.
elif mtype == 'shortcut':
s1 = x.size()[1]
s2 = layer_outputs[int(mdef['from'])].size()[1]
s3 = x.size()[2]
s4 = layer_outputs[int(mdef['from'])].size()[2]
if s3 == s4:
ad = layer_outputs[int(mdef['from'])]
else:
# only 2 by 2 currently
maxpool = nn.MaxPool2d(kernel_size=2, stride=2, padding=int((2 - 1) // 2))
ad = maxpool(layer_outputs[int(mdef['from'])])
if s1==s2:
x = x + layer_outputs[int(mdef['from'])]
elif s1 > s2:
padding = torch.autograd.Variable(torch.cuda.FloatTensor(ad.size()[0], s1-s2, ad.size()[2], ad.size()[3]).fill_(0))
ot = torch.cat((ad, padding), 1)
x = x + ot
else:
# not yet implement.
pass
@WongKinYiu would you kindly consider sharing the trained weights for the hyper-parameter optimised config? I'd love to try it out! Can post my oblation results here.
@LukeAI Hello,
The cfg/weight already put on the github. https://github.com/WongKinYiu/CrossStagePartialNetworks just download and try it.
@LukeAI Hello,
The cfg/weight already put on the github. https://github.com/WongKinYiu/CrossStagePartialNetworks just download and try it.
Hi. What about csresnext50-elastic.cfg? How i can see, there is no num of classes, filters, anchors inside. Can you make cfg file for yolov3? Thanks!
@Spectra456 Hello,
csresnext50-elastic.cfg is for imagenet.
All of other yolov3 models can be generated by following above steps.
@hwijune did it works with standard yolov3 model? I tried, but i'm getting error, when i try to use pruned model
= torch.cat([layer_outputs[i] for i in layers], 1) RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 94 and 190 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71
@hwijune did it works with standard yolov3 model? I tried, but i'm getting error, when i try to use pruned model
= torch.cat([layer_outputs[i] for i in layers], 1) RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 94 and 190 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71
It works normally.
https://github.com/erikguo/yolov3 <== this repo I checked the -3 mAP in size 608 ( 0.7 overall_ratio)
previous Yolov3 running test.
https://github.com/WongKinYiu/CrossStagePartialNetworks/blob/master/cfg/csresnet50-panet-spp.cfg << I will pruning test of cfg without group param.
@WongKinYiu What is the current situation for now? Can this repo use CSP backbone? @glenn-jocher I am looking for PyTorch implementation because I don't want to port debugging/visualization code to CPP. Like GradCam, like these: https://github.com/utkuozbulak/pytorch-cnn-visualizations
pytorch results are posted on https://github.com/WongKinYiu/CrossStagePartialNetworks/tree/pytorch
pytorch results are posted on https://github.com/WongKinYiu/CrossStagePartialNetworks/tree/pytorch
Thank you for your hard and great work! I have a question that why the CSPResNeXt50-PANet-SPP in this project can only get 59.5mAP@0.5 but the alexeyAB's version can get 64.4mAP? Maybe still some small flaws in this project? @WongKinYiu
64.4 is got by CSPResNeXt50-PANet-SPP-optimal, plz check https://github.com/WongKinYiu/CrossStagePartialNetworks
CSPResNeXt50-PANet-SPP an CSPResNeXt50-PANet-SPP-optimal are same models with different hyper-parameter. We have not implemented all of optimizations using PyTorch.
So what if if I convert weights that trained using alexey's darknet and using pytorch YOLO for forward pass?
The group convolution of darknet and pytorch seems different.
Does this repo. support CSPResNeXt50-PANet-SPP? (https://github.com/WongKinYiu/CrossStagePartialNetworks/)
AlexeyABs support: https://github.com/AlexeyAB/darknet/issues/4406
My tests have found it to be a clear winner over yolov3-spp in terms of mAP and speed.