yolov3-tiny_xnor.cfg weight ?

decombas commented 6 years ago

Do you have the weights for this cfg file or do we need to retrain it ?

AlexeyAB commented 6 years ago

@decombas You need to train it using such cfg-file as usual on CPU or GPU:

For detection, may be you should use lower threshold than usually, for example, try to use flag -thresh 0.1

Currently XNOR-net Detection is accelerated 4x times (20-25 FPS) on CPU only if: your CPU supports AVX2, and if you enabled AVX=1 in the Makefile (on Windows it is enabled by default).

https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2

CPUs with AVX2 Intel Haswell processor, Q2 2013 Haswell E processor, Q3 2014 Broadwell processor, Q4 2014 Broadwell E processor, Q3 2016 Skylake processor, Q3 2015 Kaby Lake processor, Q3 2016(ULV mobile)/Q1 2017(desktop/mobile) Skylake-X processor, Q2 2017 Coffee Lake processor, Q4 2017 Cannon Lake processor, expected in 2018 Cascade Lake processor, expected in 2018 Ice Lake processor, expected in 2018 AMD Excavator processor and newer, Q2 2015 Zen processor, Q1 2017

Just trained model https://github.com/AlexeyAB/darknet/blob/master/cfg/yolov3-tiny_xnor.cfg on GPU 10 000 iterations for 6 custom classes by using command: darknet.exe detector train data/obj.data yolov3-tiny_xnor.cfg yolov3-tiny.conv.15

Pre-trained weights-file yolov3-tiny.conv.15 was got by using file https://pjreddie.com/media/files/yolov3-tiny.weights and command: darknet.exe partial cfg/yolov3-tiny.cfg yolov3-tiny.weights yolov3-tiny.conv.15 15

Result mAP 54.65%: darknet_no_gpu.exe detector map data/obj.data yolov3-tiny_xnor.cfg backup/yolov3-tiny_xnor_10000.weights

 mean average precision (mAP) = 0.546526, or 54.65 %

for -thresh 0.25:

 for thresh = 0.25, precision = 0.81, recall = 0.25, F1-score = 0.38
 for thresh = 0.25, TP = 2107, FP = 501, FN = 6471, average IoU = 58.93 %

for -thresh 0.15:

 for thresh = 0.15, precision = 0.70, recall = 0.38, F1-score = 0.49
 for thresh = 0.15, TP = 3264, FP = 1423, FN = 5314, average IoU = 50.05 %

for -thresh 0.08 - higher F1-score:

 for thresh = 0.08, precision = 0.54, recall = 0.51, F1-score = 0.53
 for thresh = 0.08, TP = 4411, FP = 3688, FN = 4167, average IoU = 38.77 %

Avg loss chart - final average loss 0.58: chart_xnor_yolov3-tiny

Result of detection:

Detection on CPU (Core i7 6500K) - ~20 FPS: darknet_no_gpu.exe detector demo data/obj.data yolov3-tiny_xnor.cfg backup/yolov3-tiny_xnor_10000.weights test.mp4 -thresh 0.15

PeterQuinn925 commented 6 years ago

The performance increase on CPU is impressive. I would like to try this with my custom data. Looks like I just have to add xnor=1 to my custom CFG for all the convolutional layers except the first and last.

I assume I can train on GPU and detect on CPU with this cfg and the new code. Do I need to rebuild my GPU training exe in order to use this?

Thanks. I look forward to trying this out.

AlexeyAB commented 6 years ago

@PeterQuinn925 You can use old darknet.exe for training XNOR on GPU.

You should add xnor=1 parameter for each [convolutional]-layer:
- except 1st-layer: https://github.com/AlexeyAB/darknet/blob/5b6be00d4b1ffd671c20c4c72d2239c924eaa3d4/cfg/yolov3-tiny_xnor.cfg#L25
- and except layers with activation=linear (which are before each [yolo] or [region] layers).
  - https://github.com/AlexeyAB/darknet/blob/5b6be00d4b1ffd671c20c4c72d2239c924eaa3d4/cfg/yolov3-tiny_xnor.cfg#L136
  - https://github.com/AlexeyAB/darknet/blob/5b6be00d4b1ffd671c20c4c72d2239c924eaa3d4/cfg/yolov3-tiny_xnor.cfg#L182

Also, to increase mAP - you can remove xnor=1 from layers:
- [convolutional]-layer that is before [convolutional]-layer with activation=linear
  - https://github.com/AlexeyAB/darknet/blob/5b6be00d4b1ffd671c20c4c72d2239c924eaa3d4/cfg/yolov3-tiny_xnor.cfg#L123
  - https://github.com/AlexeyAB/darknet/blob/5b6be00d4b1ffd671c20c4c72d2239c924eaa3d4/cfg/yolov3-tiny_xnor.cfg#L169
- [convolutional]-layer with stride=2 if you use full yolov3.cfg:

PeterQuinn925 commented 6 years ago

On my Windows box with xnor=1 in the cfg file, I'm getting new CUDA errors during training. This is with an older version of the code. (April or so)

Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005 Resizing 768 try to allocate workspace = 8388608 * sizeof(float), CUDA allocate done! Loaded: 0.000000 seconds CUDA Error: unspecified launch failure CUDA Error: unspecified launch failure: No error Sometimes it runs for a bit before failing

On my Linux box and the old weights trained without xnor=1 and with the new cfg that contains xnor=1 it runs fine from the command line, but I'm getting a floating point exception when I call it from Python.

AlexeyAB commented 6 years ago

@PeterQuinn925 Try to train yolo-v2-tiny by using this cfg-file: https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/tiny-yolo_xnor.cfg

PeterQuinn925 commented 6 years ago

I made some minor changes because I have 4 classes and I get the same result as before. It runs for a few iterations and then CUDA error.

I tried it with an untouched copy of your _xnor.cfg using the voc dataset (I know, it's the wrong number of classes) and it does the same thing. Runs for 20 iterations and then chokes.

My edited version of Yolov3-tiny.cfg with classes=4 and no xnor trains with no problems.

AlexeyAB commented 6 years ago

@PeterQuinn925 Try to update your code from GitHub and recompile. I added info how to train yolov3-tiny-xnor: https://github.com/AlexeyAB/darknet/issues/1472#issuecomment-415480653

Also what GPU do you use? And what param do you use in the Makefile?

PeterQuinn925 commented 6 years ago

I have a Nvidia GeForce GTX 960M with 8GB. Makefile param? I'm using VS on windows to compile, so I don't have a makefile. I think these are the options that VS is using: /MP /GS /GL /W3 /Gy /Zc:wchar_t /I"C:\Users\peter\OpenCV\build\include" /I"....\3rdparty\include" /I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\include" /Zi /Gm- /O2 /sdl /Fd"x64\Release\vc140.pdb" /Zc:inline /fp:precise /D "CUDNN" /D "OPENCV" /D "_TIMESPEC_DEFINED" /D "_CRT_SECURE_NO_WARNINGS" /D "_CRT_RAND_S" /D "GPU" /D "WIN32" /D "NDEBUG" /D "_CONSOLE" /D "_LIB" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope /Gd /Oi /MD /Fa"x64\Release\" /EHsc /nologo /Fo"x64\Release\" /Fp"x64\Release\darknet.pch"

Latest code doesn't seem to crash anymore on GPU and is running fine with xnor in the cfg. It recommends subdivision=64. Is that right?

When it gets done training I'll try it on my linux machine with no GPU.

AlexeyAB commented 6 years ago

@PeterQuinn925 Try to use batch=64 subdivisions=8 Also as I can see GeForce GTX 960M has Size (GB) 2 or 4. So try to use subdivisions=4 or 8.

When it gets done training I'll try it on my linux machine with no GPU.

And during detection look at these messages, does your CPU support FMA & AVX2 or not.

PeterQuinn925 commented 6 years ago

Got it. Set subdivision=8 and it runs without running out of CUDA memory.

On my Linux box, if I set AVX=1 in the makefile, I get "illegal instruction (core dumped)" when I run darknet. It's funny though. It's a Intel Core i-3-2350M. Intel's page says it has AVX.

AlexeyAB commented 6 years ago

@PeterQuinn925 You should have AVX2 (not AVX).

You can see that Intel Core i-3-2350M doesn't have AVX2:

Also Intel Core i-3-2350M is a Sandy Bridge (Q4 2011) architecture, but AVX2 is supported only since Haswell (Q2 2013) architecture. I added a list of supported CPU architectures here: https://github.com/AlexeyAB/darknet/issues/1472#issuecomment-415480653

PeterQuinn925 commented 6 years ago

Yeah, I saw your list, that's why I checked Intel's site. Oh well, this one won't help with my hardware. Thanks for all your help.

What's funny, though is when I run it via the command line with my weights and cfg, it runs ok. But when I run it via a python script it gives a floating point exception. If I remove the xnor lines from the cfg I don't get that error via Python. I'm not bothered by this, since I can't take advantage of AVX2, but thought you might like to know.

AlexeyAB commented 5 years ago

It looks like INT1 (bit 1) operations will be supported on Tensor Cores for XNOR-net with performance ~2 Peta Ops on GeForce RTX 2080 Ti (TU102 chip).

I.e. 7 x GPUs (GeForce RTX 2080 Ti) will process XNOR neural network with the same speed as Top-10 super computer of the world - Cori - Cray XC40 with 14 Peta Flops-DP could process Float-64-bit neural network: https://www.top500.org/lists/2018/06/

http://on-demand.gputechconf.com/gtc-il/2018/pdf/sil8140-optimizing-cuda-applications-for-the-volta-turing-gpu-architecture.pdf

AlexeyAB commented 5 years ago

tensor

linquanxu commented 5 years ago

@AlexeyAB hi，when I run with "./darknet detector train train-xnor/voc.data train-xnor/yolov3-tiny_xnor.cfg -gpus 0" ,An error has occurred. environment： ubuntu 16.04 cuda 9.0 cudnn 7.4.1

however, instead of yolov3-voc.cfg,it works well. how to fix it ,thanks.

AlexeyAB commented 5 years ago

@linquanxu

Can you show output of commands nvidia-smi and nvcc --version ?
What parameters do you use in the Makefile?
Did you change anything in the yolov3-tiny_xnor.cfg? Can you attach your yolov3-tiny_xnor.cfg ?

linquanxu commented 5 years ago

@AlexeyAB hi, the output of commands nvidia-smi and nvcc --version:

Makefile:

GPU=1
CUDNN=1
OPENCV=1
OPENMP=0
DEBUG=0
ARCH= -gencode arch=compute_30,code=sm_30 \
      -gencode arch=compute_35,code=sm_35 \
      -gencode arch=compute_50,code=[sm_50,compute_50] \
      -gencode arch=compute_52,code=[sm_52,compute_52]
#      -gencode arch=compute_20,code=[sm_20,sm_21] \ This one is deprecated?

# This is what I use, uncomment if you know your arch and want to specify
# ARCH= -gencode arch=compute_52,code=compute_52

VPATH=./src/:./examples
SLIB=libdarknet.so
ALIB=libdarknet.a
EXEC=darknet
OBJDIR=./obj/

CC=gcc
NVCC=nvcc 
AR=ar
ARFLAGS=rcs
OPTS=-Ofast
LDFLAGS= -lm -pthread 
COMMON= -Iinclude/ -Isrc/
CFLAGS=-Wall -Wno-unused-result -Wno-unknown-pragmas -Wfatal-errors -fPIC

ifeq ($(OPENMP), 1) 
CFLAGS+= -fopenmp
endif

ifeq ($(DEBUG), 1) 
OPTS=-O0 -g
endif

CFLAGS+=$(OPTS)

ifeq ($(OPENCV), 1) 
COMMON+= -DOPENCV
CFLAGS+= -DOPENCV
LDFLAGS+= `pkg-config --libs opencv` 
COMMON+= `pkg-config --cflags opencv` 
endif

ifeq ($(GPU), 1) 
COMMON+= -DGPU -I/usr/local/cuda/include/
CFLAGS+= -DGPU
LDFLAGS+= -L/usr/local/cuda/lib64 -lcuda -lcudart -lcublas -lcurand
endif

ifeq ($(CUDNN), 1) 
COMMON+= -DCUDNN 
CFLAGS+= -DCUDNN
LDFLAGS+= -lcudnn
endif

OBJ=gemm.o utils.o cuda.o deconvolutional_layer.o convolutional_layer.o list.o image.o activations.o im2col.o col2im.o blas.o crop_layer.o dropout_layer.o maxpool_layer.o softmax_layer.o data.o matrix.o network.o connected_layer.o cost_layer.o parser.o option_list.o detection_layer.o route_layer.o upsample_layer.o box.o normalization_layer.o avgpool_layer.o layer.o local_layer.o shortcut_layer.o logistic_layer.o activation_layer.o rnn_layer.o gru_layer.o crnn_layer.o demo.o batchnorm_layer.o region_layer.o reorg_layer.o tree.o  lstm_layer.o l2norm_layer.o yolo_layer.o
EXECOBJA=captcha.o lsd.o super.o art.o tag.o cifar.o go.o rnn.o segmenter.o regressor.o classifier.o coco.o yolo.o detector.o nightmare.o darknet.o
ifeq ($(GPU), 1) 
LDFLAGS+= -lstdc++ 
OBJ+=convolutional_kernels.o deconvolutional_kernels.o activation_kernels.o im2col_kernels.o col2im_kernels.o blas_kernels.o crop_layer_kernels.o dropout_layer_kernels.o maxpool_layer_kernels.o avgpool_layer_kernels.o
endif

EXECOBJ = $(addprefix $(OBJDIR), $(EXECOBJA))
OBJS = $(addprefix $(OBJDIR), $(OBJ))
DEPS = $(wildcard src/*.h) Makefile include/darknet.h

all: obj backup results $(SLIB) $(ALIB) $(EXEC)
#all: obj  results $(SLIB) $(ALIB) $(EXEC)

$(EXEC): $(EXECOBJ) $(ALIB)
    $(CC) $(COMMON) $(CFLAGS) $^ -o $@ $(LDFLAGS) $(ALIB)

$(ALIB): $(OBJS)
    $(AR) $(ARFLAGS) $@ $^

$(SLIB): $(OBJS)
    $(CC) $(CFLAGS) -shared $^ -o $@ $(LDFLAGS)

$(OBJDIR)%.o: %.c $(DEPS)
    $(CC) $(COMMON) $(CFLAGS) -c $< -o $@

$(OBJDIR)%.o: %.cu $(DEPS)
    $(NVCC) $(ARCH) $(COMMON) --compiler-options "$(CFLAGS)" -c $< -o $@

obj:
    mkdir -p obj
backup:
    mkdir -p backup
results:
    mkdir -p results

.PHONY: clean

clean:
    rm -rf $(OBJS) $(SLIB) $(ALIB) $(EXEC) $(EXECOBJ) $(OBJDIR)/*

yolov3-tiny_xnor.cfg:

[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=64
subdivisions=2
width=416
height=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.001
burn_in=1000
max_batches = 500200
policy=steps
steps=400000,450000
scales=.1,.1

[convolutional]
batch_normalize=1
filters=16
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
xnor=1
bin_output=1
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
xnor=1
bin_output=1
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
xnor=1
bin_output=1
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
xnor=1
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
xnor=1
bin_output=1
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=1

[convolutional]
xnor=1
bin_output=1
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

###########

[convolutional]
xnor=1
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=255
activation=linear

[yolo]
mask = 3,4,5
anchors = 10,14,  23,27,  37,58,  81,82,  135,169,  344,319
classes=80
num=6
jitter=.3
ignore_thresh = .7
truth_thresh = 1
random=1

[route]
layers = -4

[convolutional]
xnor=1
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 8

[convolutional]
xnor=1
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=255
activation=linear

[yolo]
mask = 0,1,2
anchors = 10,14,  23,27,  37,58,  81,82,  135,169,  344,319
classes=80
num=6
jitter=.3
ignore_thresh = .7
truth_thresh = 1
random=1

AlexeyAB commented 5 years ago

@linquanxu Hi,

Try to use subdivisions=64 in your cfg-file
If it doesn't help, then try to use this repository https://github.com/AlexeyAB/darknet instead of original

linquanxu commented 5 years ago

This repository https://github.com/AlexeyAB/darknet works well. thanks @AlexeyAB

joaomiguelvieira commented 5 years ago

Hi @AlexeyAB

I was also trying to train yolov3-tiny_xnor.cfg, and I get the very same error than @linquanxu.

I have tried all the stuff you advised him to do, but nothing seems to work, and at the end I always get

When trying to train with the Pascal VOC Data, I immediatly get the error, however, when I try to train another dataset that I came up with by my own, it completes some iterations and then the same error appears.

The dataset I am using is the Street View House Numbers. It has only 10 classes. I have tried to train yolov3.cfg and yolov3-tiny.cfg and it works just fine. In the end I get a .weights file that I can use for detection.

My environment is the following:

CentOS 7.4
Cuda 9.1.85
nvidia-smi
nvcc --version
The parameters on my Makefile are:

The configuration file I am using is yolov3-tiny-svhn_xnor.cfg:


[net]
# Testing
# batch=1
# subdivisions=1
# Training
batch=64
subdivisions=4
width=416
height=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.001 burn_in=1000 max_batches = 500200 policy=steps steps=400000,450000 scales=.1,.1

[convolutional] batch_normalize=1 filters=16 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] xnor=1 bin_output=1 batch_normalize=1 filters=32 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] xnor=1 bin_output=1 batch_normalize=1 filters=64 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] xnor=1 bin_output=1 batch_normalize=1 filters=128 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] xnor=1 batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=2

[convolutional] xnor=1 bin_output=1 batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky

[maxpool] size=2 stride=1

[convolutional] xnor=1 bin_output=1 batch_normalize=1 filters=1024 size=3 stride=1 pad=1 activation=leaky

###########

[convolutional] xnor=1 batch_normalize=1 filters=256 size=1 stride=1 pad=1 activation=leaky

[convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky

[convolutional] size=1 stride=1 pad=1 filters=45 activation=linear

[yolo] mask = 3,4,5 anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 classes=10 num=6 jitter=.3 ignore_thresh = .7 truth_thresh = 1 random=1

[route] layers = -4

[convolutional] xnor=1 batch_normalize=1 filters=128 size=1 stride=1 pad=1 activation=leaky

[upsample] stride=2

[route] layers = -1, 8

[convolutional] xnor=1 batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky

[convolutional] size=1 stride=1 pad=1 filters=45 activation=linear

[yolo] mask = 0,1,2 anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319 classes=10 num=6 jitter=.3 ignore_thresh = .7 truth_thresh = 1 random=1

joaomiguelvieira commented 5 years ago

@AlexeyAB, this is what I get when using valgrind. It seems to me that the problem occurs when trying to binarize the images (?)

==10747== Memcheck, a memory error detector
==10747== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==10747== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==10747== Command: ./darknet detector train data/svhn/svhn.data cfg/non-functional/yolov3-tiny-svhn_xnor.cfg.nf1
==10747== 
yolov3-tiny-svhn_xnor
layer     filters    size              input                output
   0 conv     16  3 x 3 / 1   416 x 416 x   3   ->   416 x 416 x  16 0.150 BF
   1 max          2 x 2 / 2   416 x 416 x  16   ->   208 x 208 x  16 0.003 BF
   2 convXB   32  3 x 3 / 1   208 x 208 x  16   ->   208 x 208 x  32 0.399 BF
   3 max          2 x 2 / 2   208 x 208 x  32   ->   104 x 104 x  32 0.001 BF
   4 convXB   64  3 x 3 / 1   104 x 104 x  32   ->   104 x 104 x  64 0.399 BF
   5 max          2 x 2 / 2   104 x 104 x  64   ->    52 x  52 x  64 0.001 BF
   6 convXB  128  3 x 3 / 1    52 x  52 x  64   ->    52 x  52 x 128 0.399 BF
   7 max          2 x 2 / 2    52 x  52 x 128   ->    26 x  26 x 128 0.000 BF
   8 convX   256  3 x 3 / 1    26 x  26 x 128   ->    26 x  26 x 256 0.399 BF
   9 max          2 x 2 / 2    26 x  26 x 256   ->    13 x  13 x 256 0.000 BF
  10 convXB  512  3 x 3 / 1    13 x  13 x 256   ->    13 x  13 x 512 0.399 BF
  11 max          2 x 2 / 1    13 x  13 x 512   ->    13 x  13 x 512 0.000 BF
  12 convXB 1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024 1.595 BF
  13 convX   256  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 256 0.089 BF
  14 conv    512  3 x 3 / 1    13 x  13 x 256   ->    13 x  13 x 512 0.399 BF
  15 conv     45  1 x 1 / 1    13 x  13 x 512   ->    13 x  13 x  45 0.008 BF
  16 yolo
  17 route  13
  18 convX   128  1 x 1 / 1    13 x  13 x 256   ->    13 x  13 x 128 0.011 BF
  19 upsample            2x    13 x  13 x 128   ->    26 x  26 x 128
  20 route  19 8
  21 convX   256  3 x 3 / 1    26 x  26 x 384   ->    26 x  26 x 256 1.196 BF
  22 conv     45  1 x 1 / 1    26 x  26 x 256   ->    26 x  26 x  45 0.016 BF
  23 yolo
Total BFLOPS 5.462 
Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005
Resizing
608 x 608 
==10747== Thread 3:
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406B7A: find_replace (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406D12: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406B7A: find_replace (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406D2B: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406B7A: find_replace (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406D44: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406B7A: find_replace (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406D5D: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406B7A: find_replace (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406D76: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406C8C: find_replace_extension (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406DB0: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406C8C: find_replace_extension (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406DC9: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406C8C: find_replace_extension (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406DE2: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406C8C: find_replace_extension (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406DFB: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406C8C: find_replace_extension (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406E14: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406C8C: find_replace_extension (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406E2D: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406C8C: find_replace_extension (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406E46: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406C8C: find_replace_extension (in /opt/image_detection/darknet/darknet)
==10747==    by 0x406E5F: replace_image_to_label (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
==10747== Source and destination overlap in strcpy(0x87825c70, 0x87825c70)
==10747==    at 0x4C2CB42: strcpy (vg_replace_strmem.c:510)
==10747==    by 0x406C8C: find_replace_extension (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43B2C7: fill_truth_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43D179: load_data_detection (in /opt/image_detection/darknet/darknet)
==10747==    by 0x43E10F: load_thread (in /opt/image_detection/darknet/darknet)
==10747==    by 0x565DE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==10747==    by 0x596A34C: clone (in /usr/lib64/libc-2.17.so)
==10747== 
Loaded: 13.179834 seconds
==10747== Thread 1:
==10747== Invalid write of size 8
==10747==    at 0x4093E6: binarize_cpu (in /opt/image_detection/darknet/darknet)
==10747==    by 0x40B822: forward_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4423C2: forward_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442850: train_network_datum (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442B89: train_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45E7AE: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747==  Address 0x4563e040 is 0 bytes after a block of size 44,302,336 alloc'd
==10747==    at 0x4C2B955: calloc (vg_replace_malloc.c:711)
==10747==    by 0x40A2FD: make_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x446F44: parse_convolutional (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4496DA: parse_network_cfg_custom (in /opt/image_detection/darknet/darknet)
==10747==    by 0x44A50C: parse_network_cfg (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45F512: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747== 
==10747== Invalid write of size 8
==10747==    at 0x4093EA: binarize_cpu (in /opt/image_detection/darknet/darknet)
==10747==    by 0x40B822: forward_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4423C2: forward_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442850: train_network_datum (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442B89: train_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45E7AE: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747==  Address 0x4563e048 is 8 bytes after a block of size 44,302,336 alloc'd
==10747==    at 0x4C2B955: calloc (vg_replace_malloc.c:711)
==10747==    by 0x40A2FD: make_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x446F44: parse_convolutional (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4496DA: parse_network_cfg_custom (in /opt/image_detection/darknet/darknet)
==10747==    by 0x44A50C: parse_network_cfg (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45F512: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747== 
==10747== Invalid write of size 4
==10747==    at 0x436529: im2col_cpu (in /opt/image_detection/darknet/darknet)
==10747==    by 0x40B90A: forward_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4423C2: forward_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442850: train_network_datum (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442B89: train_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45E7AE: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747==  Address 0x120319040 is 0 bytes after a block of size 39,923,712 alloc'd
==10747==    at 0x4C2B955: calloc (vg_replace_malloc.c:711)
==10747==    by 0x4430B3: resize_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45F14B: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747== 
==10747== Invalid read of size 8
==10747==    at 0x403931: gemm_nn (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4063A5: gemm_cpu (in /opt/image_detection/darknet/darknet)
==10747==    by 0x40B948: forward_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4423C2: forward_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442850: train_network_datum (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442B89: train_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45E7AE: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747==  Address 0x120319040 is 0 bytes after a block of size 39,923,712 alloc'd
==10747==    at 0x4C2B955: calloc (vg_replace_malloc.c:711)
==10747==    by 0x4430B3: resize_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45F14B: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747== 
==10747== Invalid read of size 8
==10747==    at 0x403939: gemm_nn (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4063A5: gemm_cpu (in /opt/image_detection/darknet/darknet)
==10747==    by 0x40B948: forward_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4423C2: forward_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442850: train_network_datum (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442B89: train_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45E7AE: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747==  Address 0x120319048 is 8 bytes after a block of size 39,923,712 alloc'd
==10747==    at 0x4C2B955: calloc (vg_replace_malloc.c:711)
==10747==    by 0x4430B3: resize_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45F14B: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747== 
==10747== Invalid read of size 4
==10747==    at 0x4363C2: im2col_get_pixel (in /opt/image_detection/darknet/darknet)
==10747==    by 0x436528: im2col_cpu (in /opt/image_detection/darknet/darknet)
==10747==    by 0x40B90A: forward_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4423C2: forward_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442850: train_network_datum (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442B89: train_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45E7AE: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747==  Address 0x4563e040 is 0 bytes after a block of size 44,302,336 alloc'd
==10747==    at 0x4C2B955: calloc (vg_replace_malloc.c:711)
==10747==    by 0x40A2FD: make_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x446F44: parse_convolutional (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4496DA: parse_network_cfg_custom (in /opt/image_detection/darknet/darknet)
==10747==    by 0x44A50C: parse_network_cfg (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45F512: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747== 
==10747== 
==10747== Process terminating with default action of signal 11 (SIGSEGV)
==10747==  Access not within mapped region at address 0x57E06000
==10747==    at 0x4093E6: binarize_cpu (in /opt/image_detection/darknet/darknet)
==10747==    by 0x40B822: forward_convolutional_layer (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4423C2: forward_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442850: train_network_datum (in /opt/image_detection/darknet/darknet)
==10747==    by 0x442B89: train_network (in /opt/image_detection/darknet/darknet)
==10747==    by 0x45E7AE: train_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x4610D2: run_detector (in /opt/image_detection/darknet/darknet)
==10747==    by 0x402431: main (in /opt/image_detection/darknet/darknet)
==10747==  If you believe this happened as a result of a stack
==10747==  overflow in your program's main thread (unlikely but
==10747==  possible), you can try to increase the size of the
==10747==  main thread stack using the --main-stacksize= flag.
==10747==  The main thread stack size used in this run was 8388608.
==10747== 
==10747== HEAP SUMMARY:
==10747==     in use at exit: 4,884,340,166 bytes in 67,347 blocks
==10747==   total heap usage: 76,023 allocs, 8,676 frees, 7,064,323,663 bytes allocated
==10747== 
==10747== LEAK SUMMARY:
==10747==    definitely lost: 0 bytes in 0 blocks
==10747==    indirectly lost: 0 bytes in 0 blocks
==10747==      possibly lost: 576 bytes in 1 blocks
==10747==    still reachable: 4,884,339,590 bytes in 67,346 blocks
==10747==         suppressed: 0 bytes in 0 blocks
==10747== Rerun with --leak-check=full to see details of leaked memory
==10747== 
==10747== For counts of detected and suppressed errors, rerun with: -v
==10747== ERROR SUMMARY: 848481 errors from 20 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

AlexeyAB commented 5 years ago

@joaomiguelvieira

Try to set GPU=1 again and do

make clean
make

This is very strange, that you set GPU=1 in the Makefile, but your code uses CPU (GEMM/im2col/binarize) functions.

The parameters on my Makefile are:

==10747== by 0x4063A5: gemm_cpu (in /opt/image_detection/darknet/darknet) ... ==10747== by 0x436528: im2col_cpu (in /opt/image_detection/darknet/darknet) ... ==10747== at 0x4093E6: binarize_cpu (in /opt/image_detection/darknet/darknet)

joaomiguelvieira commented 5 years ago

Hi @AlexeyAB,

In the report I gave you from valgrind, I am not using the GPU at all. I wanted to run the code in the CPU to see where it crashes. On the GPU I have no other information than CUDA Error Prev: an illegal memory access was encountered.

In general, it does not work regardless of the options I use in the Makefile.

AlexeyAB commented 5 years ago

@joaomiguelvieira

I added fix. Try to download the latest version of Darknet.

Set GPU=1 DEBUG=1 in the Makefile, do make. And run training again - then show fullscreen screenshot of output.
Also show content of files: /data/obj.data bad.list bad_label.list
Show output of command: ./darknet detector calc_anchors data/svhn/svhn.data -num_of_clusters 6 -width 416 -height 416

joaomiguelvieira commented 5 years ago

Hi again @AlexeyAB, this is the output I get (It only fails after 40th the iteration):

./darknet detector train data/svhn/svhn.data cfg/yolov3-tiny-svhn_xnor.cfg 
yolov3-tiny-svhn_xnor
layer     filters    size              input                output
   0 conv     16  3 x 3 / 1   416 x 416 x   3   ->   416 x 416 x  16 0.150 BF
   1 max          2 x 2 / 2   416 x 416 x  16   ->   208 x 208 x  16 0.003 BF
   2 convXB   32  3 x 3 / 1   208 x 208 x  16   ->   208 x 208 x  32 0.399 BF
   3 max          2 x 2 / 2   208 x 208 x  32   ->   104 x 104 x  32 0.001 BF
   4 convXB   64  3 x 3 / 1   104 x 104 x  32   ->   104 x 104 x  64 0.399 BF
   5 max          2 x 2 / 2   104 x 104 x  64   ->    52 x  52 x  64 0.001 BF
   6 convXB  128  3 x 3 / 1    52 x  52 x  64   ->    52 x  52 x 128 0.399 BF
   7 max          2 x 2 / 2    52 x  52 x 128   ->    26 x  26 x 128 0.000 BF
   8 convX   256  3 x 3 / 1    26 x  26 x 128   ->    26 x  26 x 256 0.399 BF
   9 max          2 x 2 / 2    26 x  26 x 256   ->    13 x  13 x 256 0.000 BF
  10 convXB  512  3 x 3 / 1    13 x  13 x 256   ->    13 x  13 x 512 0.399 BF
  11 max          2 x 2 / 1    13 x  13 x 512   ->    13 x  13 x 512 0.000 BF
  12 convXB 1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024 1.595 BF
  13 convX   256  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 256 0.089 BF
  14 conv    512  3 x 3 / 1    13 x  13 x 256   ->    13 x  13 x 512 0.399 BF
  15 conv     45  1 x 1 / 1    13 x  13 x 512   ->    13 x  13 x  45 0.008 BF
  16 yolo
  17 route  13
  18 convX   128  1 x 1 / 1    13 x  13 x 256   ->    13 x  13 x 128 0.011 BF
  19 upsample            2x    13 x  13 x 128   ->    26 x  26 x 128
  20 route  19 8
  21 convX   256  3 x 3 / 1    26 x  26 x 384   ->    26 x  26 x 256 1.196 BF
  22 conv     45  1 x 1 / 1    26 x  26 x 256   ->    26 x  26 x  45 0.016 BF
  23 yolo
Total BFLOPS 5.462 
 Allocate additional workspace_size = 24.94 MB 
Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005
Resizing
608 x 608 
 try to allocate additional workspace_size = 39.92 MB 
 CUDA allocate done! 
Loaded: 0.000052 seconds
Region 16 Avg IOU: 0.327850, Class: 0.506288, Obj: 0.406473, No Obj: 0.434516, .5R: 0.090909, .75R: 0.030303,  count: 33
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.477452, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.290760, Class: 0.482134, Obj: 0.402713, No Obj: 0.435891, .5R: 0.064516, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.131745, Class: 0.488313, Obj: 0.623213, No Obj: 0.476384, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.272013, Class: 0.487319, Obj: 0.401335, No Obj: 0.436146, .5R: 0.090909, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.545606, Class: 0.389284, Obj: 0.504803, No Obj: 0.476239, .5R: 1.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.305570, Class: 0.506993, Obj: 0.392071, No Obj: 0.436937, .5R: 0.108108, .75R: 0.000000,  count: 37
Region 23 Avg IOU: 0.274067, Class: 0.469209, Obj: 0.345783, No Obj: 0.476754, .5R: 0.000000, .75R: 0.000000,  count: 3

 1: 646.512207, 646.512207 avg loss, 0.000000 rate, 3.038291 seconds, 64 images
Loaded: 0.000060 seconds
Region 16 Avg IOU: 0.301409, Class: 0.488505, Obj: 0.361043, No Obj: 0.436678, .5R: 0.107143, .75R: 0.035714,  count: 28
Region 23 Avg IOU: 0.096922, Class: 0.540398, Obj: 0.436085, No Obj: 0.476223, .5R: 0.000000, .75R: 0.000000,  count: 2
Region 16 Avg IOU: 0.268781, Class: 0.521444, Obj: 0.416384, No Obj: 0.435762, .5R: 0.030303, .75R: 0.000000,  count: 33
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.476750, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.274787, Class: 0.524794, Obj: 0.391410, No Obj: 0.435175, .5R: 0.055556, .75R: 0.000000,  count: 36
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.478321, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.316860, Class: 0.567761, Obj: 0.422675, No Obj: 0.435490, .5R: 0.135135, .75R: 0.000000,  count: 37
Region 23 Avg IOU: 0.607602, Class: 0.450851, Obj: 0.488305, No Obj: 0.474424, .5R: 0.500000, .75R: 0.000000,  count: 2

 2: 644.349426, 646.295898 avg loss, 0.000000 rate, 2.920537 seconds, 128 images
Loaded: 0.000061 seconds
Region 16 Avg IOU: 0.347225, Class: 0.506350, Obj: 0.378959, No Obj: 0.435601, .5R: 0.137931, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.366719, Class: 0.621691, Obj: 0.519996, No Obj: 0.476320, .5R: 0.333333, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.314948, Class: 0.485703, Obj: 0.395134, No Obj: 0.435616, .5R: 0.193548, .75R: 0.000000,  count: 31
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.476611, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.299116, Class: 0.515657, Obj: 0.440438, No Obj: 0.436374, .5R: 0.088235, .75R: 0.000000,  count: 34
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.476142, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.353246, Class: 0.578336, Obj: 0.381800, No Obj: 0.435997, .5R: 0.171429, .75R: 0.028571,  count: 35
Region 23 Avg IOU: 0.327859, Class: 0.564773, Obj: 0.473527, No Obj: 0.477166, .5R: 0.333333, .75R: 0.000000,  count: 3

 3: 645.219177, 646.188232 avg loss, 0.000000 rate, 2.880263 seconds, 192 images
Loaded: 0.000061 seconds
Region 16 Avg IOU: 0.297042, Class: 0.486483, Obj: 0.411926, No Obj: 0.435652, .5R: 0.135135, .75R: 0.000000,  count: 37
Region 23 Avg IOU: 0.487654, Class: 0.461317, Obj: 0.450584, No Obj: 0.476359, .5R: 0.500000, .75R: 0.000000,  count: 2
Region 16 Avg IOU: 0.285648, Class: 0.580103, Obj: 0.390270, No Obj: 0.436266, .5R: 0.085714, .75R: 0.000000,  count: 35
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.477676, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.325419, Class: 0.522186, Obj: 0.378662, No Obj: 0.435278, .5R: 0.081081, .75R: 0.000000,  count: 37
Region 23 Avg IOU: 0.126455, Class: 0.506123, Obj: 0.520876, No Obj: 0.474708, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.352137, Class: 0.529410, Obj: 0.400703, No Obj: 0.436484, .5R: 0.233333, .75R: 0.000000,  count: 30
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.477896, .5R: -nan, .75R: -nan,  count: 0

 4: 646.785645, 646.247986 avg loss, 0.000000 rate, 2.945124 seconds, 256 images
Loaded: 0.000064 seconds
Region 16 Avg IOU: 0.277896, Class: 0.489376, Obj: 0.386438, No Obj: 0.436191, .5R: 0.032258, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.198547, Class: 0.596458, Obj: 0.375616, No Obj: 0.475205, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.276645, Class: 0.481059, Obj: 0.412607, No Obj: 0.436381, .5R: 0.138889, .75R: 0.027778,  count: 36
Region 23 Avg IOU: 0.365134, Class: 0.717622, Obj: 0.480134, No Obj: 0.476511, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.277690, Class: 0.488070, Obj: 0.375524, No Obj: 0.435967, .5R: 0.093750, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.470383, Class: 0.577309, Obj: 0.368052, No Obj: 0.476007, .5R: 0.333333, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.359469, Class: 0.519959, Obj: 0.401171, No Obj: 0.435358, .5R: 0.179487, .75R: 0.000000,  count: 39
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.476401, .5R: -nan, .75R: -nan,  count: 0

 5: 644.531250, 646.076294 avg loss, 0.000000 rate, 2.971751 seconds, 320 images
Loaded: 0.000062 seconds
Region 16 Avg IOU: 0.344291, Class: 0.510871, Obj: 0.398764, No Obj: 0.436162, .5R: 0.181818, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.140461, Class: 0.596022, Obj: 0.677039, No Obj: 0.476695, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.308231, Class: 0.508206, Obj: 0.401276, No Obj: 0.435858, .5R: 0.062500, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.222854, Class: 0.795604, Obj: 0.346030, No Obj: 0.478388, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.366904, Class: 0.530525, Obj: 0.407083, No Obj: 0.435891, .5R: 0.263158, .75R: 0.026316,  count: 38
Region 23 Avg IOU: 0.337338, Class: 0.588814, Obj: 0.521945, No Obj: 0.476648, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.324480, Class: 0.488660, Obj: 0.370155, No Obj: 0.436157, .5R: 0.142857, .75R: 0.000000,  count: 35
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.477920, .5R: -nan, .75R: -nan,  count: 0

 6: 647.070557, 646.175720 avg loss, 0.000000 rate, 3.018654 seconds, 384 images
Loaded: 0.000060 seconds
Region 16 Avg IOU: 0.289655, Class: 0.540087, Obj: 0.439548, No Obj: 0.435812, .5R: 0.121212, .75R: 0.060606,  count: 33
Region 23 Avg IOU: 0.403168, Class: 0.428480, Obj: 0.403405, No Obj: 0.476732, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.284847, Class: 0.458454, Obj: 0.398289, No Obj: 0.436108, .5R: 0.031250, .75R: 0.000000,  count: 32
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.476579, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.350425, Class: 0.548126, Obj: 0.430361, No Obj: 0.435836, .5R: 0.187500, .75R: 0.062500,  count: 32
Region 23 Avg IOU: 0.100170, Class: 0.662777, Obj: 0.366580, No Obj: 0.477224, .5R: 0.000000, .75R: 0.000000,  count: 2
Region 16 Avg IOU: 0.334457, Class: 0.477384, Obj: 0.424189, No Obj: 0.435728, .5R: 0.085714, .75R: 0.000000,  count: 35
Region 23 Avg IOU: 0.176985, Class: 0.412941, Obj: 0.344387, No Obj: 0.475604, .5R: 0.000000, .75R: 0.000000,  count: 2

 7: 643.750732, 645.933228 avg loss, 0.000000 rate, 3.002682 seconds, 448 images
Loaded: 0.000075 seconds
Region 16 Avg IOU: 0.321100, Class: 0.501749, Obj: 0.380542, No Obj: 0.436124, .5R: 0.151515, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.066769, Class: 0.668623, Obj: 0.302284, No Obj: 0.477283, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.303517, Class: 0.542974, Obj: 0.406557, No Obj: 0.435370, .5R: 0.057143, .75R: 0.028571,  count: 35
Region 23 Avg IOU: 0.420482, Class: 0.662136, Obj: 0.406715, No Obj: 0.476936, .5R: 0.333333, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.315097, Class: 0.518230, Obj: 0.396555, No Obj: 0.436326, .5R: 0.147059, .75R: 0.000000,  count: 34
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.477649, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.260131, Class: 0.481716, Obj: 0.362289, No Obj: 0.436252, .5R: 0.031250, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.142607, Class: 0.551648, Obj: 0.479959, No Obj: 0.475028, .5R: 0.000000, .75R: 0.000000,  count: 6

 8: 647.932373, 646.133118 avg loss, 0.000000 rate, 3.042231 seconds, 512 images
Loaded: 0.000062 seconds
Region 16 Avg IOU: 0.320930, Class: 0.498491, Obj: 0.424221, No Obj: 0.434759, .5R: 0.090909, .75R: 0.000000,  count: 33
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.477365, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.295289, Class: 0.531655, Obj: 0.389459, No Obj: 0.435896, .5R: 0.026316, .75R: 0.000000,  count: 38
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.474972, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.293154, Class: 0.515302, Obj: 0.410176, No Obj: 0.436356, .5R: 0.054054, .75R: 0.000000,  count: 37
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.476372, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.250313, Class: 0.482085, Obj: 0.426679, No Obj: 0.436328, .5R: 0.000000, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.280460, Class: 0.533774, Obj: 0.300709, No Obj: 0.477687, .5R: 0.000000, .75R: 0.000000,  count: 3

 9: 644.452271, 645.965027 avg loss, 0.000000 rate, 3.010366 seconds, 576 images
Loaded: 0.000063 seconds
Region 16 Avg IOU: 0.259865, Class: 0.473068, Obj: 0.442863, No Obj: 0.436092, .5R: 0.100000, .75R: 0.000000,  count: 30
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.475860, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.267784, Class: 0.498488, Obj: 0.407814, No Obj: 0.435818, .5R: 0.031250, .75R: 0.000000,  count: 32
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.475959, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.316572, Class: 0.531769, Obj: 0.390164, No Obj: 0.435919, .5R: 0.078947, .75R: 0.000000,  count: 38
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.477257, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.324103, Class: 0.500054, Obj: 0.413661, No Obj: 0.436263, .5R: 0.156250, .75R: 0.031250,  count: 32
Region 23 Avg IOU: 0.137634, Class: 0.512378, Obj: 0.462570, No Obj: 0.476215, .5R: 0.000000, .75R: 0.000000,  count: 5

 10: 644.833130, 645.851807 avg loss, 0.000000 rate, 2.829789 seconds, 640 images
Resizing
384 x 384 
 try to allocate additional workspace_size = 24.94 MB 
 CUDA allocate done! 
Loaded: 0.000045 seconds
Region 16 Avg IOU: 0.315603, Class: 0.509949, Obj: 0.390317, No Obj: 0.435831, .5R: 0.093750, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.268472, Class: 0.541832, Obj: 0.453498, No Obj: 0.476213, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 16 Avg IOU: 0.317146, Class: 0.522364, Obj: 0.426087, No Obj: 0.435627, .5R: 0.064516, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.232492, Class: 0.499983, Obj: 0.445320, No Obj: 0.475948, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.401169, Class: 0.477398, Obj: 0.434082, No Obj: 0.435304, .5R: 0.307692, .75R: 0.038462,  count: 26
Region 23 Avg IOU: 0.296676, Class: 0.503652, Obj: 0.360109, No Obj: 0.476113, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.360108, Class: 0.474918, Obj: 0.412703, No Obj: 0.435944, .5R: 0.208333, .75R: 0.083333,  count: 24
Region 23 Avg IOU: 0.274151, Class: 0.565575, Obj: 0.416448, No Obj: 0.476196, .5R: 0.111111, .75R: 0.000000,  count: 9

 11: 262.573822, 607.523987 avg loss, 0.000000 rate, 1.224467 seconds, 704 images
Loaded: 0.000068 seconds
Region 16 Avg IOU: 0.284115, Class: 0.526285, Obj: 0.409193, No Obj: 0.435750, .5R: 0.153846, .75R: 0.000000,  count: 26
Region 23 Avg IOU: 0.194781, Class: 0.557387, Obj: 0.349827, No Obj: 0.475890, .5R: 0.000000, .75R: 0.000000,  count: 7
Region 16 Avg IOU: 0.323897, Class: 0.487704, Obj: 0.443133, No Obj: 0.435714, .5R: 0.142857, .75R: 0.047619,  count: 21
Region 23 Avg IOU: 0.241240, Class: 0.532919, Obj: 0.440069, No Obj: 0.475692, .5R: 0.000000, .75R: 0.000000,  count: 18
Region 16 Avg IOU: 0.341880, Class: 0.506373, Obj: 0.413742, No Obj: 0.435822, .5R: 0.133333, .75R: 0.033333,  count: 30
Region 23 Avg IOU: 0.274418, Class: 0.594952, Obj: 0.353779, No Obj: 0.475998, .5R: 0.111111, .75R: 0.000000,  count: 9
Region 16 Avg IOU: 0.307084, Class: 0.482341, Obj: 0.409790, No Obj: 0.435380, .5R: 0.068966, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.172488, Class: 0.451670, Obj: 0.434636, No Obj: 0.476159, .5R: 0.000000, .75R: 0.000000,  count: 10

 12: 265.171295, 573.288696 avg loss, 0.000000 rate, 1.521413 seconds, 768 images
Loaded: 0.000075 seconds
Region 16 Avg IOU: 0.285718, Class: 0.503648, Obj: 0.350194, No Obj: 0.435803, .5R: 0.041667, .75R: 0.041667,  count: 24
Region 23 Avg IOU: 0.205498, Class: 0.487803, Obj: 0.421501, No Obj: 0.476377, .5R: 0.000000, .75R: 0.000000,  count: 9
Region 16 Avg IOU: 0.337206, Class: 0.501511, Obj: 0.380650, No Obj: 0.435781, .5R: 0.160000, .75R: 0.040000,  count: 25
Region 23 Avg IOU: 0.173826, Class: 0.560081, Obj: 0.452932, No Obj: 0.475756, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 16 Avg IOU: 0.305783, Class: 0.475438, Obj: 0.449320, No Obj: 0.435312, .5R: 0.173913, .75R: 0.000000,  count: 23
Region 23 Avg IOU: 0.255048, Class: 0.513574, Obj: 0.361701, No Obj: 0.475810, .5R: 0.100000, .75R: 0.000000,  count: 10
Region 16 Avg IOU: 0.346770, Class: 0.488819, Obj: 0.399526, No Obj: 0.435235, .5R: 0.096774, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.283409, Class: 0.625526, Obj: 0.448169, No Obj: 0.475383, .5R: 0.000000, .75R: 0.000000,  count: 4

 13: 263.302856, 542.290100 avg loss, 0.000000 rate, 1.460588 seconds, 832 images
Loaded: 0.000072 seconds
Region 16 Avg IOU: 0.306376, Class: 0.533638, Obj: 0.408482, No Obj: 0.435832, .5R: 0.130435, .75R: 0.000000,  count: 23
Region 23 Avg IOU: 0.194222, Class: 0.528552, Obj: 0.434298, No Obj: 0.476189, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.336088, Class: 0.545721, Obj: 0.383270, No Obj: 0.435868, .5R: 0.172414, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.282305, Class: 0.544102, Obj: 0.390298, No Obj: 0.476115, .5R: 0.000000, .75R: 0.000000,  count: 12
Region 16 Avg IOU: 0.342665, Class: 0.475039, Obj: 0.386240, No Obj: 0.436003, .5R: 0.142857, .75R: 0.047619,  count: 21
Region 23 Avg IOU: 0.188953, Class: 0.568877, Obj: 0.449078, No Obj: 0.476014, .5R: 0.000000, .75R: 0.000000,  count: 15
Region 16 Avg IOU: 0.256449, Class: 0.525412, Obj: 0.405196, No Obj: 0.435737, .5R: 0.062500, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.250656, Class: 0.547459, Obj: 0.424628, No Obj: 0.475900, .5R: 0.111111, .75R: 0.000000,  count: 9

 14: 265.173126, 514.578430 avg loss, 0.000000 rate, 1.365117 seconds, 896 images
Loaded: 0.000072 seconds
Region 16 Avg IOU: 0.297855, Class: 0.476356, Obj: 0.453412, No Obj: 0.435954, .5R: 0.129032, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.277347, Class: 0.403720, Obj: 0.410386, No Obj: 0.476472, .5R: 0.125000, .75R: 0.125000,  count: 8
Region 16 Avg IOU: 0.275646, Class: 0.461133, Obj: 0.467673, No Obj: 0.436008, .5R: 0.000000, .75R: 0.000000,  count: 30
Region 23 Avg IOU: 0.336456, Class: 0.648652, Obj: 0.444415, No Obj: 0.476479, .5R: 0.333333, .75R: 0.000000,  count: 9
Region 16 Avg IOU: 0.345256, Class: 0.548461, Obj: 0.402743, No Obj: 0.435681, .5R: 0.185185, .75R: 0.037037,  count: 27
Region 23 Avg IOU: 0.266084, Class: 0.535433, Obj: 0.358179, No Obj: 0.475804, .5R: 0.100000, .75R: 0.000000,  count: 10
Region 16 Avg IOU: 0.285392, Class: 0.513468, Obj: 0.398157, No Obj: 0.436158, .5R: 0.130435, .75R: 0.000000,  count: 23
Region 23 Avg IOU: 0.211166, Class: 0.569515, Obj: 0.348217, No Obj: 0.475933, .5R: 0.100000, .75R: 0.000000,  count: 10

 15: 264.555634, 489.576141 avg loss, 0.000000 rate, 1.396277 seconds, 960 images
Loaded: 0.000083 seconds
Region 16 Avg IOU: 0.357968, Class: 0.489274, Obj: 0.386297, No Obj: 0.435397, .5R: 0.151515, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.238639, Class: 0.575213, Obj: 0.429758, No Obj: 0.476163, .5R: 0.000000, .75R: 0.000000,  count: 4
Region 16 Avg IOU: 0.298092, Class: 0.522045, Obj: 0.438208, No Obj: 0.435835, .5R: 0.062500, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.352258, Class: 0.433973, Obj: 0.496150, No Obj: 0.476125, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.345333, Class: 0.521379, Obj: 0.401503, No Obj: 0.435782, .5R: 0.115385, .75R: 0.000000,  count: 26
Region 23 Avg IOU: 0.385438, Class: 0.571849, Obj: 0.420753, No Obj: 0.475817, .5R: 0.230769, .75R: 0.000000,  count: 13
Region 16 Avg IOU: 0.327984, Class: 0.515296, Obj: 0.376324, No Obj: 0.435743, .5R: 0.115385, .75R: 0.000000,  count: 26
Region 23 Avg IOU: 0.388767, Class: 0.557882, Obj: 0.355113, No Obj: 0.476403, .5R: 0.181818, .75R: 0.090909,  count: 11

 16: 262.751801, 466.893707 avg loss, 0.000000 rate, 1.359950 seconds, 1024 images
Loaded: 0.000079 seconds
Region 16 Avg IOU: 0.377786, Class: 0.503406, Obj: 0.421384, No Obj: 0.435362, .5R: 0.238095, .75R: 0.000000,  count: 21
Region 23 Avg IOU: 0.248225, Class: 0.535285, Obj: 0.370734, No Obj: 0.476085, .5R: 0.000000, .75R: 0.000000,  count: 12
Region 16 Avg IOU: 0.299131, Class: 0.501336, Obj: 0.392456, No Obj: 0.435684, .5R: 0.076923, .75R: 0.000000,  count: 26
Region 23 Avg IOU: 0.320417, Class: 0.545687, Obj: 0.400074, No Obj: 0.476727, .5R: 0.250000, .75R: 0.000000,  count: 12
Region 16 Avg IOU: 0.302098, Class: 0.487670, Obj: 0.396930, No Obj: 0.436390, .5R: 0.074074, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.220255, Class: 0.587409, Obj: 0.522484, No Obj: 0.476698, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 16 Avg IOU: 0.382118, Class: 0.547766, Obj: 0.388816, No Obj: 0.436128, .5R: 0.290323, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.298015, Class: 0.547376, Obj: 0.366150, No Obj: 0.476225, .5R: 0.285714, .75R: 0.000000,  count: 7

 17: 263.367004, 446.541046 avg loss, 0.000000 rate, 1.435596 seconds, 1088 images
Loaded: 0.000075 seconds
Region 16 Avg IOU: 0.362670, Class: 0.513942, Obj: 0.426803, No Obj: 0.435948, .5R: 0.250000, .75R: 0.000000,  count: 24
Region 23 Avg IOU: 0.226295, Class: 0.522170, Obj: 0.357236, No Obj: 0.476226, .5R: 0.100000, .75R: 0.000000,  count: 10
Region 16 Avg IOU: 0.338896, Class: 0.518549, Obj: 0.409378, No Obj: 0.436657, .5R: 0.205882, .75R: 0.000000,  count: 34
Region 23 Avg IOU: 0.236513, Class: 0.557908, Obj: 0.428482, No Obj: 0.476954, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.333251, Class: 0.489062, Obj: 0.430005, No Obj: 0.435609, .5R: 0.111111, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.284222, Class: 0.490024, Obj: 0.372377, No Obj: 0.475910, .5R: 0.000000, .75R: 0.000000,  count: 7
Region 16 Avg IOU: 0.343335, Class: 0.468039, Obj: 0.372492, No Obj: 0.436026, .5R: 0.172414, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.317139, Class: 0.510988, Obj: 0.344973, No Obj: 0.476482, .5R: 0.222222, .75R: 0.000000,  count: 9

 18: 264.091827, 428.296112 avg loss, 0.000000 rate, 1.395934 seconds, 1152 images
Loaded: 0.000085 seconds
Region 16 Avg IOU: 0.253857, Class: 0.497227, Obj: 0.398320, No Obj: 0.435130, .5R: 0.083333, .75R: 0.000000,  count: 24
Region 23 Avg IOU: 0.218374, Class: 0.512152, Obj: 0.421295, No Obj: 0.477420, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 16 Avg IOU: 0.325746, Class: 0.530271, Obj: 0.397882, No Obj: 0.435940, .5R: 0.208333, .75R: 0.000000,  count: 24
Region 23 Avg IOU: 0.256075, Class: 0.515809, Obj: 0.455406, No Obj: 0.476658, .5R: 0.166667, .75R: 0.000000,  count: 12
Region 16 Avg IOU: 0.291180, Class: 0.525453, Obj: 0.416716, No Obj: 0.435408, .5R: 0.076923, .75R: 0.000000,  count: 26
Region 23 Avg IOU: 0.216557, Class: 0.547811, Obj: 0.335077, No Obj: 0.476735, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.310194, Class: 0.495745, Obj: 0.403733, No Obj: 0.435959, .5R: 0.034483, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.375184, Class: 0.523187, Obj: 0.426553, No Obj: 0.476583, .5R: 0.428571, .75R: 0.000000,  count: 7

 19: 264.654297, 411.931946 avg loss, 0.000000 rate, 1.305750 seconds, 1216 images
Loaded: 0.000074 seconds
Region 16 Avg IOU: 0.363566, Class: 0.486108, Obj: 0.406973, No Obj: 0.435616, .5R: 0.259259, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.327857, Class: 0.671221, Obj: 0.406071, No Obj: 0.475988, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.259278, Class: 0.510071, Obj: 0.438089, No Obj: 0.436289, .5R: 0.071429, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.208759, Class: 0.491982, Obj: 0.427885, No Obj: 0.476180, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.328131, Class: 0.508564, Obj: 0.404389, No Obj: 0.436186, .5R: 0.142857, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.165441, Class: 0.623006, Obj: 0.473028, No Obj: 0.476301, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.328051, Class: 0.521075, Obj: 0.412092, No Obj: 0.436018, .5R: 0.064516, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.200149, Class: 0.596025, Obj: 0.298933, No Obj: 0.475942, .5R: 0.200000, .75R: 0.000000,  count: 5

 20: 262.824310, 397.021179 avg loss, 0.000000 rate, 1.377255 seconds, 1280 images
Resizing
416 x 416 
 try to allocate additional workspace_size = 24.94 MB 
 CUDA allocate done! 
Loaded: 0.000047 seconds
Region 16 Avg IOU: 0.381127, Class: 0.498384, Obj: 0.395618, No Obj: 0.435899, .5R: 0.250000, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.142513, Class: 0.583922, Obj: 0.399872, No Obj: 0.476039, .5R: 0.000000, .75R: 0.000000,  count: 2
Region 16 Avg IOU: 0.354933, Class: 0.506217, Obj: 0.402670, No Obj: 0.435947, .5R: 0.185185, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.220075, Class: 0.619733, Obj: 0.428045, No Obj: 0.476742, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.308159, Class: 0.474001, Obj: 0.464340, No Obj: 0.435998, .5R: 0.107143, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.188830, Class: 0.500187, Obj: 0.474352, No Obj: 0.477021, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.274432, Class: 0.500592, Obj: 0.405798, No Obj: 0.435972, .5R: 0.111111, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.316845, Class: 0.555914, Obj: 0.358924, No Obj: 0.476692, .5R: 0.125000, .75R: 0.000000,  count: 8

 21: 305.944702, 387.913544 avg loss, 0.000000 rate, 1.709487 seconds, 1344 images
Loaded: 0.000063 seconds
Region 16 Avg IOU: 0.320062, Class: 0.501193, Obj: 0.428364, No Obj: 0.435295, .5R: 0.066667, .75R: 0.000000,  count: 30
Region 23 Avg IOU: 0.194003, Class: 0.670961, Obj: 0.394868, No Obj: 0.477243, .5R: 0.142857, .75R: 0.000000,  count: 7
Region 16 Avg IOU: 0.372327, Class: 0.534572, Obj: 0.414905, No Obj: 0.435860, .5R: 0.137931, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.242769, Class: 0.424352, Obj: 0.447711, No Obj: 0.475788, .5R: 0.000000, .75R: 0.000000,  count: 2
Region 16 Avg IOU: 0.403285, Class: 0.503024, Obj: 0.420424, No Obj: 0.435006, .5R: 0.269231, .75R: 0.000000,  count: 26
Region 23 Avg IOU: 0.336546, Class: 0.571228, Obj: 0.391613, No Obj: 0.476824, .5R: 0.200000, .75R: 0.200000,  count: 5
Region 16 Avg IOU: 0.316684, Class: 0.479569, Obj: 0.423987, No Obj: 0.435709, .5R: 0.133333, .75R: 0.000000,  count: 30
Region 23 Avg IOU: 0.279744, Class: 0.479900, Obj: 0.462842, No Obj: 0.476417, .5R: 0.200000, .75R: 0.000000,  count: 5

 22: 306.782013, 379.800385 avg loss, 0.000000 rate, 1.593568 seconds, 1408 images
Loaded: 0.000095 seconds
Region 16 Avg IOU: 0.310712, Class: 0.524959, Obj: 0.406363, No Obj: 0.435986, .5R: 0.117647, .75R: 0.000000,  count: 34
Region 23 Avg IOU: 0.406036, Class: 0.474579, Obj: 0.351653, No Obj: 0.477228, .5R: 0.333333, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.327237, Class: 0.511212, Obj: 0.379945, No Obj: 0.435967, .5R: 0.206897, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.181102, Class: 0.565634, Obj: 0.523898, No Obj: 0.476282, .5R: 0.000000, .75R: 0.000000,  count: 4
Region 16 Avg IOU: 0.312253, Class: 0.489797, Obj: 0.393628, No Obj: 0.436203, .5R: 0.086957, .75R: 0.000000,  count: 23
Region 23 Avg IOU: 0.368659, Class: 0.561829, Obj: 0.402418, No Obj: 0.477662, .5R: 0.300000, .75R: 0.000000,  count: 10
Region 16 Avg IOU: 0.354387, Class: 0.506219, Obj: 0.415228, No Obj: 0.435910, .5R: 0.153846, .75R: 0.000000,  count: 26
Region 23 Avg IOU: 0.329371, Class: 0.550671, Obj: 0.349617, No Obj: 0.476244, .5R: 0.250000, .75R: 0.000000,  count: 8

 23: 307.923035, 372.612640 avg loss, 0.000000 rate, 1.452081 seconds, 1472 images
Loaded: 0.000094 seconds
Region 16 Avg IOU: 0.312017, Class: 0.495189, Obj: 0.429939, No Obj: 0.435733, .5R: 0.096774, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.296813, Class: 0.542978, Obj: 0.400778, No Obj: 0.476022, .5R: 0.111111, .75R: 0.000000,  count: 9
Region 16 Avg IOU: 0.323961, Class: 0.523513, Obj: 0.432064, No Obj: 0.435609, .5R: 0.083333, .75R: 0.000000,  count: 24
Region 23 Avg IOU: 0.205133, Class: 0.555578, Obj: 0.399532, No Obj: 0.476436, .5R: 0.181818, .75R: 0.000000,  count: 11
Region 16 Avg IOU: 0.366179, Class: 0.517344, Obj: 0.429053, No Obj: 0.435854, .5R: 0.218750, .75R: 0.031250,  count: 32
Region 23 Avg IOU: 0.191508, Class: 0.477979, Obj: 0.397984, No Obj: 0.476045, .5R: 0.000000, .75R: 0.000000,  count: 7
Region 16 Avg IOU: 0.332232, Class: 0.469373, Obj: 0.419846, No Obj: 0.435678, .5R: 0.214286, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.265483, Class: 0.515871, Obj: 0.384884, No Obj: 0.475978, .5R: 0.153846, .75R: 0.000000,  count: 13

 24: 308.750488, 366.226440 avg loss, 0.000000 rate, 1.409168 seconds, 1536 images
Loaded: 0.000055 seconds
Region 16 Avg IOU: 0.322171, Class: 0.480785, Obj: 0.414380, No Obj: 0.436175, .5R: 0.066667, .75R: 0.000000,  count: 30
Region 23 Avg IOU: 0.240970, Class: 0.574398, Obj: 0.371058, No Obj: 0.475915, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.316189, Class: 0.501465, Obj: 0.383215, No Obj: 0.435846, .5R: 0.240000, .75R: 0.000000,  count: 25
Region 23 Avg IOU: 0.257573, Class: 0.495212, Obj: 0.382969, No Obj: 0.476853, .5R: 0.000000, .75R: 0.000000,  count: 10
Region 16 Avg IOU: 0.353893, Class: 0.500234, Obj: 0.447267, No Obj: 0.436038, .5R: 0.269231, .75R: 0.000000,  count: 26
Region 23 Avg IOU: 0.284821, Class: 0.502118, Obj: 0.331193, No Obj: 0.475732, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 16 Avg IOU: 0.321699, Class: 0.482678, Obj: 0.393122, No Obj: 0.435646, .5R: 0.187500, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.151647, Class: 0.409233, Obj: 0.352628, No Obj: 0.476949, .5R: 0.000000, .75R: 0.000000,  count: 5

 25: 308.975372, 360.501343 avg loss, 0.000000 rate, 1.522829 seconds, 1600 images
Loaded: 0.000075 seconds
Region 16 Avg IOU: 0.332095, Class: 0.485779, Obj: 0.431058, No Obj: 0.435962, .5R: 0.030303, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.222367, Class: 0.471329, Obj: 0.414784, No Obj: 0.475807, .5R: 0.000000, .75R: 0.000000,  count: 4
Region 16 Avg IOU: 0.329479, Class: 0.478512, Obj: 0.417224, No Obj: 0.436401, .5R: 0.171429, .75R: 0.057143,  count: 35
Region 23 Avg IOU: 0.235052, Class: 0.573864, Obj: 0.456813, No Obj: 0.477448, .5R: 0.000000, .75R: 0.000000,  count: 5
Region 16 Avg IOU: 0.288020, Class: 0.492340, Obj: 0.418729, No Obj: 0.435600, .5R: 0.111111, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.184874, Class: 0.534623, Obj: 0.393775, No Obj: 0.476678, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.342189, Class: 0.487293, Obj: 0.433677, No Obj: 0.436125, .5R: 0.160000, .75R: 0.000000,  count: 25
Region 23 Avg IOU: 0.229122, Class: 0.599644, Obj: 0.419399, No Obj: 0.475071, .5R: 0.000000, .75R: 0.000000,  count: 6

 26: 308.835388, 355.334747 avg loss, 0.000000 rate, 1.538354 seconds, 1664 images
Loaded: 0.000083 seconds
Region 16 Avg IOU: 0.313362, Class: 0.506747, Obj: 0.357369, No Obj: 0.435796, .5R: 0.093750, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.170800, Class: 0.653871, Obj: 0.433032, No Obj: 0.477155, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.275993, Class: 0.509395, Obj: 0.479864, No Obj: 0.436088, .5R: 0.045455, .75R: 0.000000,  count: 22
Region 23 Avg IOU: 0.230047, Class: 0.615456, Obj: 0.374141, No Obj: 0.476381, .5R: 0.000000, .75R: 0.000000,  count: 10
Region 16 Avg IOU: 0.339342, Class: 0.498816, Obj: 0.435686, No Obj: 0.436189, .5R: 0.148148, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.294690, Class: 0.609666, Obj: 0.454571, No Obj: 0.476173, .5R: 0.142857, .75R: 0.000000,  count: 7
Region 16 Avg IOU: 0.313249, Class: 0.539520, Obj: 0.419922, No Obj: 0.436306, .5R: 0.074074, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.279373, Class: 0.567037, Obj: 0.378921, No Obj: 0.475962, .5R: 0.000000, .75R: 0.000000,  count: 5

 27: 307.062073, 350.507477 avg loss, 0.000000 rate, 1.655512 seconds, 1728 images
Loaded: 0.000075 seconds
Region 16 Avg IOU: 0.324614, Class: 0.508480, Obj: 0.401940, No Obj: 0.435665, .5R: 0.083333, .75R: 0.000000,  count: 24
Region 23 Avg IOU: 0.293698, Class: 0.532384, Obj: 0.432657, No Obj: 0.475732, .5R: 0.090909, .75R: 0.000000,  count: 11
Region 16 Avg IOU: 0.338169, Class: 0.537973, Obj: 0.410488, No Obj: 0.435716, .5R: 0.193548, .75R: 0.064516,  count: 31
Region 23 Avg IOU: 0.154844, Class: 0.477938, Obj: 0.361943, No Obj: 0.475181, .5R: 0.000000, .75R: 0.000000,  count: 7
Region 16 Avg IOU: 0.307549, Class: 0.535696, Obj: 0.453226, No Obj: 0.435766, .5R: 0.096774, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.204899, Class: 0.475122, Obj: 0.377508, No Obj: 0.476570, .5R: 0.166667, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.255245, Class: 0.523569, Obj: 0.435356, No Obj: 0.435868, .5R: 0.000000, .75R: 0.000000,  count: 21
Region 23 Avg IOU: 0.216220, Class: 0.635920, Obj: 0.400718, No Obj: 0.475291, .5R: 0.083333, .75R: 0.000000,  count: 12

 28: 308.162231, 346.272949 avg loss, 0.000000 rate, 1.609449 seconds, 1792 images
Loaded: 0.000081 seconds
Region 16 Avg IOU: 0.336767, Class: 0.483346, Obj: 0.383984, No Obj: 0.436161, .5R: 0.178571, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.374231, Class: 0.542676, Obj: 0.433913, No Obj: 0.475977, .5R: 0.400000, .75R: 0.000000,  count: 5
Region 16 Avg IOU: 0.338351, Class: 0.517480, Obj: 0.411752, No Obj: 0.435716, .5R: 0.218750, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.544327, Class: 0.484103, Obj: 0.536708, No Obj: 0.475041, .5R: 0.666667, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.296580, Class: 0.494293, Obj: 0.405675, No Obj: 0.435853, .5R: 0.142857, .75R: 0.000000,  count: 21
Region 23 Avg IOU: 0.339712, Class: 0.613700, Obj: 0.410557, No Obj: 0.476935, .5R: 0.181818, .75R: 0.000000,  count: 11
Region 16 Avg IOU: 0.315226, Class: 0.483579, Obj: 0.416105, No Obj: 0.435721, .5R: 0.107143, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.256965, Class: 0.573962, Obj: 0.418599, No Obj: 0.476843, .5R: 0.100000, .75R: 0.000000,  count: 10

 29: 307.054413, 342.351105 avg loss, 0.000000 rate, 1.513933 seconds, 1856 images
Loaded: 0.000071 seconds
Region 16 Avg IOU: 0.334439, Class: 0.499144, Obj: 0.412688, No Obj: 0.435896, .5R: 0.157895, .75R: 0.000000,  count: 38
Region 23 Avg IOU: 0.191617, Class: 0.494596, Obj: 0.467894, No Obj: 0.477147, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.314207, Class: 0.519991, Obj: 0.410280, No Obj: 0.435600, .5R: 0.090909, .75R: 0.030303,  count: 33
Region 23 Avg IOU: 0.311701, Class: 0.584788, Obj: 0.435242, No Obj: 0.476397, .5R: 0.000000, .75R: 0.000000,  count: 5
Region 16 Avg IOU: 0.250543, Class: 0.474819, Obj: 0.395517, No Obj: 0.435854, .5R: 0.000000, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.219978, Class: 0.521424, Obj: 0.464242, No Obj: 0.476506, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.288136, Class: 0.499161, Obj: 0.382380, No Obj: 0.435533, .5R: 0.068966, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.166926, Class: 0.563962, Obj: 0.302259, No Obj: 0.476504, .5R: 0.000000, .75R: 0.000000,  count: 4

 30: 309.019165, 339.017914 avg loss, 0.000000 rate, 1.609415 seconds, 1920 images
Resizing
448 x 448 
 try to allocate additional workspace_size = 24.94 MB 
 CUDA allocate done! 
Loaded: 0.000047 seconds
Region 16 Avg IOU: 0.292144, Class: 0.516982, Obj: 0.422206, No Obj: 0.435545, .5R: 0.090909, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.316587, Class: 0.447116, Obj: 0.445946, No Obj: 0.475837, .5R: 0.250000, .75R: 0.000000,  count: 4
Region 16 Avg IOU: 0.313608, Class: 0.502883, Obj: 0.406357, No Obj: 0.435515, .5R: 0.166667, .75R: 0.027778,  count: 36
Region 23 Avg IOU: 0.379297, Class: 0.457302, Obj: 0.475013, No Obj: 0.475811, .5R: 0.333333, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.309189, Class: 0.470387, Obj: 0.427961, No Obj: 0.436039, .5R: 0.222222, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.234952, Class: 0.501513, Obj: 0.383902, No Obj: 0.476549, .5R: 0.111111, .75R: 0.000000,  count: 9
Region 16 Avg IOU: 0.344019, Class: 0.547289, Obj: 0.409871, No Obj: 0.435390, .5R: 0.161290, .75R: 0.032258,  count: 31
Region 23 Avg IOU: 0.156132, Class: 0.591578, Obj: 0.344989, No Obj: 0.474847, .5R: 0.000000, .75R: 0.000000,  count: 6

 31: 355.728302, 340.688965 avg loss, 0.000000 rate, 1.850821 seconds, 1984 images
Loaded: 0.000060 seconds
Region 16 Avg IOU: 0.312383, Class: 0.543920, Obj: 0.393480, No Obj: 0.436074, .5R: 0.103448, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.168116, Class: 0.564910, Obj: 0.380423, No Obj: 0.475933, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.311491, Class: 0.457071, Obj: 0.418983, No Obj: 0.434720, .5R: 0.096774, .75R: 0.000000,  count: 31
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.476664, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.303632, Class: 0.518550, Obj: 0.368801, No Obj: 0.435995, .5R: 0.125000, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.578072, Class: 0.560082, Obj: 0.336745, No Obj: 0.475924, .5R: 0.666667, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.314689, Class: 0.523892, Obj: 0.404835, No Obj: 0.435669, .5R: 0.103448, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.052949, Class: 0.498760, Obj: 0.281984, No Obj: 0.476206, .5R: 0.000000, .75R: 0.000000,  count: 1

 32: 354.377350, 342.057800 avg loss, 0.000000 rate, 1.677403 seconds, 2048 images
Loaded: 0.000061 seconds
Region 16 Avg IOU: 0.292976, Class: 0.499552, Obj: 0.424709, No Obj: 0.435685, .5R: 0.066667, .75R: 0.000000,  count: 30
Region 23 Avg IOU: 0.206199, Class: 0.600852, Obj: 0.482914, No Obj: 0.476002, .5R: 0.000000, .75R: 0.000000,  count: 4
Region 16 Avg IOU: 0.327491, Class: 0.498001, Obj: 0.417700, No Obj: 0.436160, .5R: 0.235294, .75R: 0.000000,  count: 34
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.477269, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.358760, Class: 0.521174, Obj: 0.403054, No Obj: 0.435689, .5R: 0.200000, .75R: 0.000000,  count: 20
Region 23 Avg IOU: 0.201386, Class: 0.561880, Obj: 0.457196, No Obj: 0.475578, .5R: 0.000000, .75R: 0.000000,  count: 12
Region 16 Avg IOU: 0.342534, Class: 0.481297, Obj: 0.419302, No Obj: 0.435764, .5R: 0.166667, .75R: 0.000000,  count: 24
Region 23 Avg IOU: 0.345734, Class: 0.393653, Obj: 0.311383, No Obj: 0.475839, .5R: 0.000000, .75R: 0.000000,  count: 7

 33: 354.696960, 343.321716 avg loss, 0.000000 rate, 1.791701 seconds, 2112 images
Loaded: 0.000061 seconds
Region 16 Avg IOU: 0.322915, Class: 0.463747, Obj: 0.397470, No Obj: 0.435854, .5R: 0.133333, .75R: 0.000000,  count: 30
Region 23 Avg IOU: 0.256739, Class: 0.504578, Obj: 0.343711, No Obj: 0.476031, .5R: 0.000000, .75R: 0.000000,  count: 9
Region 16 Avg IOU: 0.335879, Class: 0.523252, Obj: 0.408862, No Obj: 0.436304, .5R: 0.142857, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.244905, Class: 0.551795, Obj: 0.433725, No Obj: 0.475761, .5R: 0.000000, .75R: 0.000000,  count: 7
Region 16 Avg IOU: 0.309317, Class: 0.546416, Obj: 0.408610, No Obj: 0.436123, .5R: 0.103448, .75R: 0.034483,  count: 29
Region 23 Avg IOU: 0.205618, Class: 0.528446, Obj: 0.377442, No Obj: 0.475620, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.294868, Class: 0.534462, Obj: 0.388986, No Obj: 0.435775, .5R: 0.107143, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.361969, Class: 0.553613, Obj: 0.325248, No Obj: 0.475977, .5R: 0.142857, .75R: 0.000000,  count: 7

 34: 355.099060, 344.499451 avg loss, 0.000000 rate, 1.855584 seconds, 2176 images
Loaded: 0.000064 seconds
Region 16 Avg IOU: 0.337581, Class: 0.517164, Obj: 0.385596, No Obj: 0.435751, .5R: 0.137931, .75R: 0.034483,  count: 29
Region 23 Avg IOU: 0.262610, Class: 0.614347, Obj: 0.417742, No Obj: 0.476442, .5R: 0.100000, .75R: 0.000000,  count: 10
Region 16 Avg IOU: 0.327769, Class: 0.521583, Obj: 0.388762, No Obj: 0.435972, .5R: 0.100000, .75R: 0.000000,  count: 30
Region 23 Avg IOU: 0.133915, Class: 0.572435, Obj: 0.461359, No Obj: 0.474923, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.374730, Class: 0.500077, Obj: 0.409231, No Obj: 0.436064, .5R: 0.200000, .75R: 0.040000,  count: 25
Region 23 Avg IOU: 0.374587, Class: 0.503936, Obj: 0.378835, No Obj: 0.475740, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 16 Avg IOU: 0.309078, Class: 0.489697, Obj: 0.439845, No Obj: 0.436364, .5R: 0.062500, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.333501, Class: 0.553741, Obj: 0.344505, No Obj: 0.475930, .5R: 0.111111, .75R: 0.000000,  count: 9

 35: 354.506409, 345.500153 avg loss, 0.000000 rate, 1.669898 seconds, 2240 images
Loaded: 0.000062 seconds
Region 16 Avg IOU: 0.324399, Class: 0.510815, Obj: 0.393005, No Obj: 0.435702, .5R: 0.111111, .75R: 0.037037,  count: 27
Region 23 Avg IOU: 0.294283, Class: 0.533149, Obj: 0.304497, No Obj: 0.475788, .5R: 0.000000, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.321173, Class: 0.507775, Obj: 0.421717, No Obj: 0.435931, .5R: 0.111111, .75R: 0.000000,  count: 36
Region 23 Avg IOU: 0.168433, Class: 0.550922, Obj: 0.443215, No Obj: 0.476145, .5R: 0.000000, .75R: 0.000000,  count: 2
Region 16 Avg IOU: 0.314076, Class: 0.501272, Obj: 0.439376, No Obj: 0.435927, .5R: 0.166667, .75R: 0.000000,  count: 30
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.475824, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.324816, Class: 0.436050, Obj: 0.378748, No Obj: 0.435734, .5R: 0.193548, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.145926, Class: 0.563734, Obj: 0.223075, No Obj: 0.475878, .5R: 0.000000, .75R: 0.000000,  count: 2

 36: 354.262177, 346.376343 avg loss, 0.000000 rate, 1.838970 seconds, 2304 images
Loaded: 0.000063 seconds
Region 16 Avg IOU: 0.269330, Class: 0.503162, Obj: 0.413392, No Obj: 0.435518, .5R: 0.000000, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.369289, Class: 0.511927, Obj: 0.419090, No Obj: 0.474887, .5R: 0.166667, .75R: 0.000000,  count: 6
Region 16 Avg IOU: 0.314137, Class: 0.512675, Obj: 0.433413, No Obj: 0.435749, .5R: 0.187500, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.231137, Class: 0.429118, Obj: 0.436629, No Obj: 0.475582, .5R: 0.000000, .75R: 0.000000,  count: 4
Region 16 Avg IOU: 0.355547, Class: 0.487339, Obj: 0.359871, No Obj: 0.436246, .5R: 0.142857, .75R: 0.035714,  count: 28
Region 23 Avg IOU: 0.089365, Class: 0.599525, Obj: 0.378553, No Obj: 0.476119, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.303385, Class: 0.470994, Obj: 0.426358, No Obj: 0.436015, .5R: 0.035714, .75R: 0.000000,  count: 28
Region 23 Avg IOU: 0.303495, Class: 0.536251, Obj: 0.418086, No Obj: 0.474588, .5R: 0.000000, .75R: 0.000000,  count: 10

 37: 354.733032, 347.212006 avg loss, 0.000000 rate, 1.638657 seconds, 2368 images
Loaded: 0.000061 seconds
Region 16 Avg IOU: 0.328915, Class: 0.538399, Obj: 0.407687, No Obj: 0.435687, .5R: 0.147059, .75R: 0.000000,  count: 34
Region 23 Avg IOU: 0.294107, Class: 0.403174, Obj: 0.306742, No Obj: 0.476177, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.298635, Class: 0.472952, Obj: 0.407453, No Obj: 0.435071, .5R: 0.000000, .75R: 0.000000,  count: 29
Region 23 Avg IOU: 0.317804, Class: 0.569716, Obj: 0.383555, No Obj: 0.475800, .5R: 0.000000, .75R: 0.000000,  count: 4
Region 16 Avg IOU: 0.349534, Class: 0.525193, Obj: 0.385702, No Obj: 0.436074, .5R: 0.222222, .75R: 0.000000,  count: 27
Region 23 Avg IOU: 0.283129, Class: 0.582511, Obj: 0.464626, No Obj: 0.475451, .5R: 0.142857, .75R: 0.000000,  count: 7
Region 16 Avg IOU: 0.278425, Class: 0.491931, Obj: 0.394486, No Obj: 0.436097, .5R: 0.032258, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.301381, Class: 0.627559, Obj: 0.387017, No Obj: 0.475120, .5R: 0.166667, .75R: 0.000000,  count: 6

 38: 354.225372, 347.913330 avg loss, 0.000000 rate, 1.848529 seconds, 2432 images
Loaded: 0.000063 seconds
Region 16 Avg IOU: 0.341374, Class: 0.512212, Obj: 0.394430, No Obj: 0.436252, .5R: 0.103448, .75R: 0.034483,  count: 29
Region 23 Avg IOU: 0.343017, Class: 0.633973, Obj: 0.389243, No Obj: 0.475278, .5R: 0.250000, .75R: 0.000000,  count: 4
Region 16 Avg IOU: 0.352816, Class: 0.509881, Obj: 0.381053, No Obj: 0.435508, .5R: 0.242424, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.170081, Class: 0.458321, Obj: 0.444045, No Obj: 0.476458, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 16 Avg IOU: 0.357795, Class: 0.505314, Obj: 0.404028, No Obj: 0.435710, .5R: 0.192308, .75R: 0.000000,  count: 26
Region 23 Avg IOU: 0.337828, Class: 0.506057, Obj: 0.399511, No Obj: 0.474735, .5R: 0.142857, .75R: 0.000000,  count: 7
Region 16 Avg IOU: 0.323091, Class: 0.535872, Obj: 0.385729, No Obj: 0.435745, .5R: 0.062500, .75R: 0.000000,  count: 32
Region 23 Avg IOU: 0.068011, Class: 0.548809, Obj: 0.417860, No Obj: 0.475215, .5R: 0.000000, .75R: 0.000000,  count: 3

 39: 354.909546, 348.612946 avg loss, 0.000000 rate, 1.785213 seconds, 2496 images
Loaded: 0.000065 seconds
Region 16 Avg IOU: 0.356603, Class: 0.463786, Obj: 0.388511, No Obj: 0.435862, .5R: 0.212121, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.054114, Class: 0.579468, Obj: 0.409063, No Obj: 0.475429, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.270250, Class: 0.454317, Obj: 0.418334, No Obj: 0.435225, .5R: 0.060606, .75R: 0.000000,  count: 33
Region 23 Avg IOU: 0.421349, Class: 0.519394, Obj: 0.468668, No Obj: 0.475517, .5R: 0.400000, .75R: 0.000000,  count: 5
Region 16 Avg IOU: 0.306895, Class: 0.548131, Obj: 0.402699, No Obj: 0.435893, .5R: 0.032258, .75R: 0.000000,  count: 31
Region 23 Avg IOU: 0.148044, Class: 0.415407, Obj: 0.370292, No Obj: 0.474640, .5R: 0.000000, .75R: 0.000000,  count: 3
Region 16 Avg IOU: 0.324074, Class: 0.527141, Obj: 0.396324, No Obj: 0.436503, .5R: 0.172414, .75R: 0.034483,  count: 29
Region 23 Avg IOU: 0.228145, Class: 0.573107, Obj: 0.392422, No Obj: 0.475941, .5R: 0.000000, .75R: 0.000000,  count: 8

 40: 354.438049, 349.195465 avg loss, 0.000000 rate, 1.812226 seconds, 2560 images
Resizing
544 x 544 
 try to allocate additional workspace_size = 31.96 MB 
 CUDA allocate done! 
Loaded: 0.000051 seconds
CUDA Error: file: ./src/im2col_kernels.cu : () : line: 90 : build time: Feb  4 2019 - 21:58:20 
CUDA Error: an illegal memory access was encountered
darknet: ./src/cuda.c:35: check_error: Assertion `0' failed.
Aborted (core dumped)

The content of the file data/svhn/svhn.data is the following:

classes = 10
train   = data/svhn/train/train.txt
valid   = data/svhn/test/test.txt
names   = data/svhn/svhn.names
backup  = backup/

There were no files bad.list or bad_label.list being generated.

The output of the command ./darknet detector calc_anchors data/svhn/svhn.data -num_of_clusters 6 -width 416 -height 416 is the following:

./darknet detector calc_anchors data/svhn/svhn.data -num_of_clusters 6 -width 416 -height 416

 num_of_clusters = 6, width = 416, height = 416 
 read labels from 33402 images 
 loaded      image: 33402    box: 73257
 all loaded. 

 calculating k-means++ ...

 iterations = 72 

 avg IoU = 81.45 % 

Saving anchors to the file: anchors.txt 
anchors =  26,125,  37,201,  44,292,  57,234,  65,311,  90,327

AlexeyAB commented 5 years ago

@joaomiguelvieira Thanks!

Until I fix it, you can try to install cuDNN-library and make Darknet with CUDNN=1 in the Makefile, it should work well and much faster.

joaomiguelvieira commented 5 years ago

@AlexeyAB thanks for your help. I will do it then!

AlexeyAB / darknet

yolov3-tiny_xnor.cfg weight ? #1472