wangyuan111
commented
3 years ago Closed wangyuan111 closed 3 years ago
wangyuan111
commented
3 years ago wangyuan111
commented
3 years ago 在dataloader中可视化标签 没有发现什么问题
Joker316701882
commented
3 years ago @wangyuan111 Could you provide your training command. It's better to provide one sample image from your dataset.
wangyuan111
commented
3 years ago raining command and config
python tools/train.py -f exps/example/yolox_voc/title_coco.py -d 1 -b 32 --fp16 -o -c yolox_s.pth.tarCOCO_CLASSES = (
'edge_anomaly',
'angular_anomaly',
'white_point_defect',
'light_block_defect',
'dark_block_defect',
'halo_defect',
)
//序号从0到5class Exp(MyExp):
def __init__(self):
super(Exp, self).__init__()
self.num_classes = 6
self.depth = 0.33
self.width = 0.50
self.eval_interval = 2
self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]
# ---------------- dataloader config ---------------- #
# set worker to 4 for shorter dataloader init time
self.data_num_workers = 0
self.input_size = (640, 640)
self.random_size = (14, 26)
self.train_ann = "train2017.json"
self.val_ann = "val2017.json"
# --------------- transform config ----------------- #
self.degrees = 10.0
self.translate = 0.1
self.scale = (0.1, 2)
self.mscale = (0.8, 1.6)
self.shear = 2.0
self.perspective = 0.0
self.enable_mixup = True
# -------------- training config --------------------- #
self.warmup_epochs = 5
self.max_epoch = 300
self.warmup_lr = 0
self.basic_lr_per_img = 0.01 / 64.0
self.scheduler = "yoloxwarmcos"
self.no_aug_epochs = 15
self.min_lr_ratio = 0.05
self.ema = True
self.weight_decay = 5e-4
self.momentum = 0.9
self.print_interval = 1
self.eval_interval = 1
self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]
# ----------------- testing config ------------------ #
self.test_size = (640, 640)
self.test_conf = 0.2
self.nmsthre = 0.65
def get_data_loader(self, batch_size, is_distributed, no_aug=False):
from yolox.data import (
COCODataset,
TrainTransform,
YoloBatchSampler,
DataLoader,
InfiniteSampler,
MosaicDetection,
)
dataset = COCODataset(
data_dir='datasets/COCO/train2017/',
json_file="train2017.json",
name="train2017",
img_size=self.input_size,
preproc=TrainTransform(
rgb_means=(0.485, 0.456, 0.406),
std=(0.229, 0.224, 0.225),
max_labels=50,
),
)
dataset = MosaicDetection(
dataset,
mosaic=not no_aug,
img_size=self.input_size,
preproc=TrainTransform(
rgb_means=(0.485, 0.456, 0.406),
std=(0.229, 0.224, 0.225),
max_labels=120,
),
degrees=self.degrees,
translate=self.translate,
scale=self.scale,
shear=self.shear,
perspective=self.perspective,
enable_mixup=self.enable_mixup,
)
self.dataset = dataset
if is_distributed:
batch_size = batch_size // dist.get_world_size()
sampler = InfiniteSampler(
len(self.dataset), seed=self.seed if self.seed else 0
)
else:
sampler = torch.utils.data.RandomSampler(self.dataset)
batch_sampler = YoloBatchSampler(
sampler=sampler,
batch_size=batch_size,
drop_last=False,
input_dimension=self.input_size,
mosaic=not no_aug,
)
dataloader_kwargs = {"num_workers": self.data_num_workers, "pin_memory": True}
dataloader_kwargs["batch_sampler"] = batch_sampler
train_loader = DataLoader(self.dataset, **dataloader_kwargs)
return train_loader
def get_eval_loader(self, batch_size, is_distributed, testdev=False):
from yolox.data import COCODataset, ValTransform
valdataset = COCODataset(
data_dir='datasets/COCO/val2017/',
json_file="val2017.json",
name="val2017",
img_size=self.test_size,
preproc=ValTransform(
rgb_means=(0.485, 0.456, 0.406),
std=(0.229, 0.224, 0.225),
),
)
if is_distributed:
batch_size = batch_size // dist.get_world_size()
sampler = torch.utils.data.distributed.DistributedSampler(
valdataset, shuffle=False
)
else:
sampler = torch.utils.data.SequentialSampler(valdataset)
dataloader_kwargs = {
"num_workers": self.data_num_workers,
"pin_memory": True,
"sampler": sampler,
}
dataloader_kwargs["batch_size"] = batch_size
val_loader = torch.utils.data.DataLoader(valdataset, **dataloader_kwargs)
return val_loader
def get_evaluator(self, batch_size, is_distributed, testdev=False):
from yolox.evaluators import COCOEvaluator
val_loader = self.get_eval_loader(batch_size, is_distributed, testdev=testdev)
evaluator = COCOEvaluator(
dataloader=val_loader,
img_size=self.test_size,
confthre=self.test_conf,
nmsthre=self.nmsthre,
num_classes=self.num_classes,
)
return evaluator
In my data set, the serial number of the label category is set to 1-6, there is no 0. In other words, the label category only has
"categories": [{"id": 1, "name": "edge_anomaly"}, {"id": 2, "name": "angular_anomaly"}, {"id": 3, "name": "white_point_defect"}, {"id": 4, "name": "black_defect"}, {"id": 5, "name": "white_block_defect"}, {"id": 6, "name": "dark_block_defect"}]After 300 rounds of training on yolov5, mAP can reach about 70%。
GOATmessi8
commented
3 years ago Closing the mixup aug and setting the mosaic scale from (0,1, 2) to (0.5, 1.5) may be helpful
But your final loss is so weird (too small). I suggest you to eval on your training set to check the case of overfitting.
And could you share your training log file?
wangyuan111
commented
3 years ago @ruinmessi Thank you, a complete training log is as follows： train_log.txt
wangyuan111
commented
3 years ago Closing the mixup aug and setting the mosaic scale from (0,1, 2) to (0.5, 1.5) may be helpful
But your final loss is so weird (too small). I suggest you to eval on your training set to check the case of overfitting.
And could you share your training log file?
Hello, I modified the parameters according to your method, and the mAP was improved, but I found that the score was decreasing during the training. My data set is very large, with more than 10000 pieces. It may not so easy to
over fitting
wangyuan111
commented
3 years ago Closing the mixup aug and setting the mosaic scale from (0,1, 2) to (0.5, 1.5) may be helpful But your final loss is so weird (too small). I suggest you to eval on your training set to check the case of overfitting. And could you share your training log file?
Hello, I modified the parameters according to your method, and the mAP was improved, but I found that the score was decreasing during the training. My data set is very large, with more than 10000 pieces. It may not so easy to over fitting
训练参数如下：
2021-07-30 01:41:34.665 | INFO | yolox.core.trainer:before_train:128 - args: Namespace(batch_size=32, ckpt='yolox_s.pth.tar', devices=1, dist_backend='nccl', dist_url=None, exp_file='exps/example/yolox_voc/title_coco.py', experiment_name='title_coco', fp16=True, local_rank=0, machine_rank=0, name=None, num_machine=1, occumpy=True, opts=[], resume=False, start_epoch=None)
2021-07-30 01:41:34.670 | INFO | yolox.core.trainer:before_train:129 - exp value:
╒══════════════════╤════════════════════════╕
│ keys │ values │
╞══════════════════╪════════════════════════╡
│ seed │ None │
├──────────────────┼────────────────────────┤
│ output_dir │ './data/yolox_output/' │
├──────────────────┼────────────────────────┤
│ print_interval │ 50 │
├──────────────────┼────────────────────────┤
│ eval_interval │ 1 │
├──────────────────┼────────────────────────┤
│ num_classes │ 6 │
├──────────────────┼────────────────────────┤
│ depth │ 0.33 │
├──────────────────┼────────────────────────┤
│ width │ 0.5 │
├──────────────────┼────────────────────────┤
│ data_num_workers │ 0 │
├──────────────────┼────────────────────────┤
│ input_size │ (640, 640) │
├──────────────────┼────────────────────────┤
│ random_size │ (14, 26) │
├──────────────────┼────────────────────────┤
│ train_ann │ 'train2017.json' │
├──────────────────┼────────────────────────┤
│ val_ann │ 'val2017.json' │
├──────────────────┼────────────────────────┤
│ degrees │ 0 │
├──────────────────┼────────────────────────┤
│ translate │ 0.1 │
├──────────────────┼────────────────────────┤
│ scale │ (0.5, 1.5) │
├──────────────────┼────────────────────────┤
│ mscale │ (0.8, 1.6) │
├──────────────────┼────────────────────────┤
│ shear │ 1.0 │
├──────────────────┼────────────────────────┤
│ perspective │ 0.0 │
├──────────────────┼────────────────────────┤
│ enable_mixup │ False │
├──────────────────┼────────────────────────┤
│ warmup_epochs │ 5 │
├──────────────────┼────────────────────────┤
│ max_epoch │ 300 │
├──────────────────┼────────────────────────┤
│ warmup_lr │ 0 │
├──────────────────┼────────────────────────┤
│ basic_lr_per_img │ 9.375e-05 │
├──────────────────┼────────────────────────┤
│ scheduler │ 'yoloxwarmcos' │
├──────────────────┼────────────────────────┤
│ no_aug_epochs │ 15 │
├──────────────────┼────────────────────────┤
│ min_lr_ratio │ 0.05 │
├──────────────────┼────────────────────────┤
│ ema │ True │
├──────────────────┼────────────────────────┤
│ weight_decay │ 0.0005 │
├──────────────────┼────────────────────────┤
│ momentum │ 0.9 │
├──────────────────┼────────────────────────┤
│ exp_name │ 'title_coco' │
├──────────────────┼────────────────────────┤
│ test_size │ (640, 640) │
├──────────────────┼────────────────────────┤
│ test_conf │ 0.2 │
├──────────────────┼────────────────────────┤
│ nmsthre │ 0.65 │
╘══════════════════╧════════════════════════╛
2021-07-30 01:41:36.475 | INFO | yolox.core.trainer:before_train:134 - Model Summary: Params: 8.94M, Gflops: 26.65
2021-07-30 01:41:40.855 | INFO | apex.amp.frontend:initialize:328 - Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.
2021-07-30 01:41:40.857 | INFO | apex.amp.frontend:initialize:329 - Defaults for this optimization level are:
2021-07-30 01:41:40.857 | INFO | apex.amp.frontend:initialize:331 - enabled : True
2021-07-30 01:41:40.857 | INFO | apex.amp.frontend:initialize:331 - opt_level : O1
2021-07-30 01:41:40.858 | INFO | apex.amp.frontend:initialize:331 - cast_model_type : None
2021-07-30 01:41:40.859 | INFO | apex.amp.frontend:initialize:331 - patch_torch_functions : True
2021-07-30 01:41:40.859 | INFO | apex.amp.frontend:initialize:331 - keep_batchnorm_fp32 : None
2021-07-30 01:41:40.860 | INFO | apex.amp.frontend:initialize:331 - master_weights : None
2021-07-30 01:41:40.861 | INFO | apex.amp.frontend:initialize:331 - loss_scale : dynamic
2021-07-30 01:41:40.861 | INFO | apex.amp.frontend:initialize:336 - Processing user overrides (additional kwargs that are not None)...
2021-07-30 01:41:40.862 | INFO | apex.amp.frontend:initialize:354 - After processing overrides, optimization options are:
2021-07-30 01:41:40.863 | INFO | apex.amp.frontend:initialize:356 - enabled : True
2021-07-30 01:41:40.863 | INFO | apex.amp.frontend:initialize:356 - opt_level : O1
2021-07-30 01:41:40.864 | INFO | apex.amp.frontend:initialize:356 - cast_model_type : None
2021-07-30 01:41:40.864 | INFO | apex.amp.frontend:initialize:356 - patch_torch_functions : True
2021-07-30 01:41:40.865 | INFO | apex.amp.frontend:initialize:356 - keep_batchnorm_fp32 : None
2021-07-30 01:41:40.865 | INFO | apex.amp.frontend:initialize:356 - master_weights : None
2021-07-30 01:41:40.866 | INFO | apex.amp.frontend:initialize:356 - loss_scale : dynamic
2021-07-30 01:41:40.888 | INFO | yolox.core.trainer:resume_train:287 - loading checkpoint for fine tuning
2021-07-30 01:41:41.027 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.0.weight in checkpoint is torch.Size([80, 128, 1, 1]), while shape of head.cls_preds.0.weight in model is torch.Size([6, 128, 1, 1]).
2021-07-30 01:41:41.028 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.0.bias in checkpoint is torch.Size([80]), while shape of head.cls_preds.0.bias in model is torch.Size([6]).
2021-07-30 01:41:41.029 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.1.weight in checkpoint is torch.Size([80, 128, 1, 1]), while shape of head.cls_preds.1.weight in model is torch.Size([6, 128, 1, 1]).
2021-07-30 01:41:41.030 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.1.bias in checkpoint is torch.Size([80]), while shape of head.cls_preds.1.bias in model is torch.Size([6]).
2021-07-30 01:41:41.030 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.2.weight in checkpoint is torch.Size([80, 128, 1, 1]), while shape of head.cls_preds.2.weight in model is torch.Size([6, 128, 1, 1]).
2021-07-30 01:41:41.031 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.2.bias in checkpoint is torch.Size([80]), while shape of head.cls_preds.2.bias in model is torch.Size([6]).
2021-07-30 01:41:41.071 | INFO | yolox.data.datasets.coco:__init__:54 - loading annotations into memory...
2021-07-30 01:41:41.200 | INFO | yolox.data.datasets.coco:__init__:54 - Done (t=0.13s)
2021-07-30 01:41:41.200 | INFO | pycocotools.coco:__init__:89 - creating index...
2021-07-30 01:41:41.209 | INFO | pycocotools.coco:__init__:89 - index created!
2021-07-30 01:41:41.210 | INFO | yolox.core.trainer:before_train:153 - init prefetcher, this might take one minute or less...
2021-07-30 01:41:49.129 | INFO | yolox.data.datasets.coco:__init__:54 - loading annotations into memory...
2021-07-30 01:41:49.146 | INFO | yolox.data.datasets.coco:__init__:54 - Done (t=0.02s)
2021-07-30 01:41:49.147 | INFO | pycocotools.coco:__init__:89 - creating index...
2021-07-30 01:41:49.150 | INFO | pycocotools.coco:__init__:89 - index created!
2021-07-30 01:41:49.153 | INFO | yolox.core.trainer:before_train:183 - Training start...
2021-07-30 01:41:49.155 | INFO | yolox.core.trainer:before_train:184 -
YOLOX(
(backbone): YOLOPAFPN(
(backbone): CSPDarknet(
(stem): Focus(
(conv): BaseConv(
(conv): Conv2d(12, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(dark2): Sequential(
(0): BaseConv(
(conv): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(32, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(dark3): Sequential(
(0): BaseConv(
(conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(dark4): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(dark5): Sequential(
(0): BaseConv(
(conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): SPPBottleneck(
(conv1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): MaxPool2d(kernel_size=5, stride=1, padding=2, dilation=1, ceil_mode=False)
(1): MaxPool2d(kernel_size=9, stride=1, padding=4, dilation=1, ceil_mode=False)
(2): MaxPool2d(kernel_size=13, stride=1, padding=6, dilation=1, ceil_mode=False)
)
(conv2): BaseConv(
(conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
)
(upsample): Upsample(scale_factor=2.0, mode=nearest)
(lateral_conv0): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_p4): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
(reduce_conv1): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_p3): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
(bu_conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_n3): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
(bu_conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_n4): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(head): YOLOXHead(
(cls_convs): ModuleList(
(0): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
(reg_convs): ModuleList(
(0): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
(cls_preds): ModuleList(
(0): Conv2d(128, 6, kernel_size=(1, 1), stride=(1, 1))
(1): Conv2d(128, 6, kernel_size=(1, 1), stride=(1, 1))
(2): Conv2d(128, 6, kernel_size=(1, 1), stride=(1, 1))
)
(reg_preds): ModuleList(
(0): Conv2d(128, 4, kernel_size=(1, 1), stride=(1, 1))
(1): Conv2d(128, 4, kernel_size=(1, 1), stride=(1, 1))
(2): Conv2d(128, 4, kernel_size=(1, 1), stride=(1, 1))
)
(obj_preds): ModuleList(
(0): Conv2d(128, 1, kernel_size=(1, 1), stride=(1, 1))
(1): Conv2d(128, 1, kernel_size=(1, 1), stride=(1, 1))
(2): Conv2d(128, 1, kernel_size=(1, 1), stride=(1, 1))
)
(stems): ModuleList(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(2): BaseConv(
(conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(l1_loss): L1Loss()
(bcewithlog_loss): BCEWithLogitsLoss()
(iou_loss): IOUloss()
)
)
2021-07-30 01:41:49.156 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch1
2021-07-30 01:41:54.080 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32768.0
2021-07-30 01:41:57.496 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-07-30 01:42:00.380 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 8192.0
2021-07-30 01:42:06.256 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 4096.0
2021-07-30 01:42:47.656 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 2048.0
2021-07-30 01:44:21.073 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 50/408, mem: 34780Mb, iter_time: 3.038s, data_time: 2.464s, total_loss: 13.6, iou_loss: 3.4, l1_loss: 0.0, conf_loss: 7.0, cls_loss: 3.1, lr: 1.802e-06, size: 480, ETA: 4 days, 7:15:08
2021-07-30 01:47:30.175 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 100/408, mem: 34780Mb, iter_time: 3.782s, data_time: 3.129s, total_loss: 9.7, iou_loss: 2.9, l1_loss: 0.0, conf_loss: 4.3, cls_loss: 2.4, lr: 7.209e-06, size: 736, ETA: 4 days, 19:50:29
2021-07-30 01:49:50.969 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 150/408, mem: 34780Mb, iter_time: 2.816s, data_time: 2.335s, total_loss: 7.6, iou_loss: 3.2, l1_loss: 0.0, conf_loss: 2.9, cls_loss: 1.5, lr: 1.622e-05, size: 800, ETA: 4 days, 13:04:00
2021-07-30 01:52:04.298 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 200/408, mem: 34780Mb, iter_time: 2.666s, data_time: 2.238s, total_loss: 6.6, iou_loss: 2.5, l1_loss: 0.0, conf_loss: 2.5, cls_loss: 1.6, lr: 2.884e-05, size: 512, ETA: 4 days, 8:23:30
2021-07-30 01:54:25.454 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 250/408, mem: 34780Mb, iter_time: 2.823s, data_time: 2.374s, total_loss: 6.3, iou_loss: 2.6, l1_loss: 0.0, conf_loss: 2.3, cls_loss: 1.4, lr: 4.505e-05, size: 448, ETA: 4 days, 6:38:07
2021-07-30 01:57:16.605 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 300/408, mem: 34780Mb, iter_time: 3.423s, data_time: 2.899s, total_loss: 6.6, iou_loss: 2.9, l1_loss: 0.0, conf_loss: 2.7, cls_loss: 1.0, lr: 6.488e-05, size: 768, ETA: 4 days, 8:50:31
2021-07-30 01:59:32.974 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 350/408, mem: 34780Mb, iter_time: 2.727s, data_time: 2.317s, total_loss: 6.4, iou_loss: 2.9, l1_loss: 0.0, conf_loss: 2.6, cls_loss: 0.9, lr: 8.831e-05, size: 448, ETA: 4 days, 7:02:08
2021-07-30 02:02:03.334 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 400/408, mem: 34780Mb, iter_time: 3.007s, data_time: 2.564s, total_loss: 6.0, iou_loss: 3.0, l1_loss: 0.0, conf_loss: 2.2, cls_loss: 0.8, lr: 1.153e-04, size: 704, ETA: 4 days, 6:51:23
2021-07-30 02:02:25.942 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:04:55.544 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:171 - Evaluate in main process...
2021-07-30 02:04:55.591 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - Loading and preparing results...
2021-07-30 02:04:55.601 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - DONE (t=0.01s)
2021-07-30 02:04:55.602 | INFO | pycocotools.coco:loadRes:365 - creating index...
2021-07-30 02:04:55.604 | INFO | pycocotools.coco:loadRes:365 - index created!
2021-07-30 02:04:56.207 | INFO | yolox.core.trainer:evaluate_and_save_model:302 -
Average forward time: 1.23 ms, Average NMS time: 0.70 ms, Average inference time: 1.93 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.124
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.315
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.077
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.081
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.149
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.127
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.196
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.211
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.211
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.149
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.255
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.163
2021-07-30 02:04:56.208 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:04:56.466 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch2
2021-07-30 02:05:11.666 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 50/408, mem: 34780Mb, iter_time: 0.304s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.4, lr: 1.512e-04, size: 640, ETA: 3 days, 20:34:57
2021-07-30 02:05:27.108 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 100/408, mem: 34780Mb, iter_time: 0.308s, data_time: 0.000s, total_loss: 1.2, iou_loss: 0.8, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 1.860e-04, size: 448, ETA: 3 days, 12:27:50
2021-07-30 02:05:42.510 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 150/408, mem: 34780Mb, iter_time: 0.308s, data_time: 0.000s, total_loss: 1.1, iou_loss: 0.7, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 2.245e-04, size: 480, ETA: 3 days, 5:47:49
2021-07-30 02:05:57.777 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 200/408, mem: 34780Mb, iter_time: 0.305s, data_time: 0.000s, total_loss: 1.1, iou_loss: 0.7, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 2.665e-04, size: 736, ETA: 3 days, 0:13:06
2021-07-30 02:06:14.147 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 250/408, mem: 34780Mb, iter_time: 0.327s, data_time: 0.000s, total_loss: 1.2, iou_loss: 0.8, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 3.121e-04, size: 608, ETA: 2 days, 19:32:37
2021-07-30 02:06:29.240 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 300/408, mem: 34780Mb, iter_time: 0.302s, data_time: 0.000s, total_loss: 0.9, iou_loss: 0.6, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 3.613e-04, size: 544, ETA: 2 days, 15:28:04
2021-07-30 02:06:44.828 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 350/408, mem: 34780Mb, iter_time: 0.311s, data_time: 0.000s, total_loss: 0.5, iou_loss: 0.3, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.1, lr: 4.142e-04, size: 832, ETA: 2 days, 11:57:03
2021-07-30 02:07:00.114 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 400/408, mem: 34780Mb, iter_time: 0.305s, data_time: 0.000s, total_loss: 0.7, iou_loss: 0.5, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 4.706e-04, size: 576, ETA: 2 days, 8:51:22
2021-07-30 02:07:02.562 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:09:20.137 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:171 - Evaluate in main process...
2021-07-30 02:09:20.170 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - Loading and preparing results...
2021-07-30 02:09:20.177 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - DONE (t=0.01s)
2021-07-30 02:09:20.178 | INFO | pycocotools.coco:loadRes:365 - creating index...
2021-07-30 02:09:20.180 | INFO | pycocotools.coco:loadRes:365 - index created!
2021-07-30 02:09:20.658 | INFO | yolox.core.trainer:evaluate_and_save_model:302 -
Average forward time: 1.25 ms, Average NMS time: 0.61 ms, Average inference time: 1.86 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.030
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.105
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.007
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.025
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.042
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.017
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.063
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.069
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.069
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.078
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.032
2021-07-30 02:09:20.658 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:09:20.827 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch3
2021-07-30 02:09:36.431 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 50/408, mem: 34780Mb, iter_time: 0.312s, data_time: 0.000s, total_loss: 0.7, iou_loss: 0.5, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 5.406e-04, size: 544, ETA: 2 days, 5:43:33
2021-07-30 02:09:51.673 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 100/408, mem: 34780Mb, iter_time: 0.304s, data_time: 0.000s, total_loss: 0.6, iou_loss: 0.3, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 6.049e-04, size: 544, ETA: 2 days, 3:19:59
2021-07-30 02:10:07.495 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 150/408, mem: 34780Mb, iter_time: 0.316s, data_time: 0.000s, total_loss: 0.6, iou_loss: 0.3, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 6.727e-04, size: 704, ETA: 2 days, 1:12:28
2021-07-30 02:10:22.488 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 200/408, mem: 34780Mb, iter_time: 0.300s, data_time: 0.000s, total_loss: 0.6, iou_loss: 0.3, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 7.441e-04, size: 800, ETA: 1 day, 23:15:50
2021-07-30 02:10:37.957 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 250/408, mem: 34780Mb, iter_time: 0.309s, data_time: 0.000s, total_loss: 0.8, iou_loss: 0.5, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 8.192e-04, size: 704, ETA: 1 day, 21:31:02
2021-07-30 02:10:53.257 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 300/408, mem: 34780Mb, iter_time: 0.306s, data_time: 0.000s, total_loss: 0.9, iou_loss: 0.6, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 8.978e-04, size: 544, ETA: 1 day, 19:55:17
2021-07-30 02:11:08.805 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 350/408, mem: 34780Mb, iter_time: 0.311s, data_time: 0.000s, total_loss: 0.9, iou_loss: 0.6, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 9.801e-04, size: 832, ETA: 1 day, 18:28:09
2021-07-30 02:11:24.098 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 400/408, mem: 34780Mb, iter_time: 0.306s, data_time: 0.000s, total_loss: 0.9, iou_loss: 0.6, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 1.066e-03, size: 512, ETA: 1 day, 17:07:44
2021-07-30 02:11:26.473 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:13:50.841 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:171 - Evaluate in main process...
2021-07-30 02:13:50.873 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - Loading and preparing results...
2021-07-30 02:13:50.880 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - DONE (t=0.01s)
2021-07-30 02:13:50.880 | INFO | pycocotools.coco:loadRes:365 - creating index...
2021-07-30 02:13:50.882 | INFO | pycocotools.coco:loadRes:365 - index created!
2021-07-30 02:13:51.395 | INFO | yolox.core.trainer:evaluate_and_save_model:302 -
Average forward time: 1.22 ms, Average NMS time: 0.64 ms, Average inference time: 1.86 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.021
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.081
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.020
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.027
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.004
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.048
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.052
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.052
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.050
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.047
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.006
2021-07-30 02:13:51.395 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:13:51.571 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch4
2021-07-30 02:14:06.565 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 50/408, mem: 34780Mb, iter_time: 0.300s, data_time: 0.000s, total_loss: 0.8, iou_loss: 0.5, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 1.170e-03, size: 576, ETA: 1 day, 15:41:45
2021-07-30 02:14:21.940 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 100/408, mem: 34780Mb, iter_time: 0.307s, data_time: 0.000s, total_loss: 0.9, iou_loss: 0.6, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 1.264e-03, size: 800, ETA: 1 day, 14:34:16
2021-07-30 02:14:37.617 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 150/408, mem: 34780Mb, iter_time: 0.313s, data_time: 0.000s, total_loss: 1.2, iou_loss: 0.8, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 1.361e-03, size: 832, ETA: 1 day, 13:32:07
2021-07-30 02:14:52.789 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 200/408, mem: 34780Mb, iter_time: 0.303s, data_time: 0.000s, total_loss: 1.3, iou_loss: 0.8, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 1.462e-03, size: 736, ETA: 1 day, 12:33:36
2021-07-30 02:15:08.362 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 250/408, mem: 34780Mb, iter_time: 0.311s, data_time: 0.000s, total_loss: 1.4, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.3, lr: 1.566e-03, size: 544, ETA: 1 day, 11:39:35
2021-07-30 02:15:23.627 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 300/408, mem: 34780Mb, iter_time: 0.305s, data_time: 0.000s, total_loss: 1.4, iou_loss: 0.9, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.3, lr: 1.674e-03, size: 512, ETA: 1 day, 10:48:41
2021-07-30 02:15:39.564 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 350/408, mem: 34780Mb, iter_time: 0.318s, data_time: 0.000s, total_loss: 1.1, iou_loss: 0.7, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 1.786e-03, size: 448, ETA: 1 day, 10:01:53
2021-07-30 02:15:54.403 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 400/408, mem: 34780Mb, iter_time: 0.296s, data_time: 0.000s, total_loss: 1.2, iou_loss: 0.7, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 1.901e-03, size: 480, ETA: 1 day, 9:16:34
2021-07-30 02:15:56.815 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:18:20.748 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:171 - Evaluate in main process...
2021-07-30 02:18:20.812 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - Loading and preparing results...
2021-07-30 02:18:20.820 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - DONE (t=0.01s)
2021-07-30 02:18:20.820 | INFO | pycocotools.coco:loadRes:365 - creating index...
2021-07-30 02:18:20.822 | INFO | pycocotools.coco:loadRes:365 - index created!
2021-07-30 02:18:21.436 | INFO | yolox.core.trainer:evaluate_and_save_model:302 -
Average forward time: 1.21 ms, Average NMS time: 0.64 ms, Average inference time: 1.85 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.015
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.065
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.016
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.021
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.034
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.038
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.038
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.041
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.030
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001
2021-07-30 02:18:21.437 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:18:21.615 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch5
2021-07-30 02:18:36.894 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 50/408, mem: 34780Mb, iter_time: 0.305s, data_time: 0.000s, total_loss: 1.2, iou_loss: 0.8, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 2.039e-03, size: 640, ETA: 1 day, 8:27:56
2021-07-30 02:18:52.429 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 100/408, mem: 34780Mb, iter_time: 0.310s, data_time: 0.000s, total_loss: 1.2, iou_loss: 0.8, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 2.163e-03, size: 768, ETA: 1 day, 7:48:56
2021-07-30 02:19:07.635 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 150/408, mem: 34780Mb, iter_time: 0.304s, data_time: 0.000s, total_loss: 1.3, iou_loss: 0.8, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 2.289e-03, size: 672, ETA: 1 day, 7:11:44
2021-07-30 02:19:22.678 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 200/408, mem: 34780Mb, iter_time: 0.301s, data_time: 0.000s, total_loss: 0.9, iou_loss: 0.6, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 2.419e-03, size: 512, ETA: 1 day, 6:36:22
2021-07-30 02:19:38.476 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 250/408, mem: 34780Mb, iter_time: 0.316s, data_time: 0.000s, total_loss: 0.8, iou_loss: 0.5, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.2, lr: 2.553e-03, size: 480, ETA: 1 day, 6:03:41
2021-07-30 02:19:38.796 | INFO | yolox.core.trainer:after_train:188 - Training of experiment is done and the best AP is 12.43
2021-07-30 02:26:04.609 | INFO | yolox.core.trainer:before_train:128 - args: Namespace(batch_size=32, ckpt='yolox_s.pth.tar', devices=1, dist_backend='nccl', dist_url=None, exp_file='exps/example/yolox_voc/title_coco.py', experiment_name='title_coco', fp16=True, local_rank=0, machine_rank=0, name=None, num_machine=1, occumpy=True, opts=[], resume=False, start_epoch=None)
2021-07-30 02:26:04.615 | INFO | yolox.core.trainer:before_train:129 - exp value:
╒══════════════════╤════════════════════════╕
│ keys │ values │
╞══════════════════╪════════════════════════╡
│ seed │ None │
├──────────────────┼────────────────────────┤
│ output_dir │ './data/yolox_output/' │
├──────────────────┼────────────────────────┤
│ print_interval │ 50 │
├──────────────────┼────────────────────────┤
│ eval_interval │ 1 │
├──────────────────┼────────────────────────┤
│ num_classes │ 6 │
├──────────────────┼────────────────────────┤
│ depth │ 0.33 │
├──────────────────┼────────────────────────┤
│ width │ 0.5 │
├──────────────────┼────────────────────────┤
│ data_num_workers │ 0 │
├──────────────────┼────────────────────────┤
│ input_size │ (640, 640) │
├──────────────────┼────────────────────────┤
│ random_size │ (14, 26) │
├──────────────────┼────────────────────────┤
│ train_ann │ 'train2017.json' │
├──────────────────┼────────────────────────┤
│ val_ann │ 'val2017.json' │
├──────────────────┼────────────────────────┤
│ degrees │ 0 │
├──────────────────┼────────────────────────┤
│ translate │ 0.1 │
├──────────────────┼────────────────────────┤
│ scale │ (0.5, 1.5) │
├──────────────────┼────────────────────────┤
│ mscale │ (0.8, 1.6) │
├──────────────────┼────────────────────────┤
│ shear │ 1.0 │
├──────────────────┼────────────────────────┤
│ perspective │ 0.0 │
├──────────────────┼────────────────────────┤
│ enable_mixup │ False │
├──────────────────┼────────────────────────┤
│ warmup_epochs │ 5 │
├──────────────────┼────────────────────────┤
│ max_epoch │ 300 │
├──────────────────┼────────────────────────┤
│ warmup_lr │ 0 │
├──────────────────┼────────────────────────┤
│ basic_lr_per_img │ 0.00015625 │
├──────────────────┼────────────────────────┤
│ scheduler │ 'yoloxwarmcos' │
├──────────────────┼────────────────────────┤
│ no_aug_epochs │ 15 │
├──────────────────┼────────────────────────┤
│ min_lr_ratio │ 0.05 │
├──────────────────┼────────────────────────┤
│ ema │ True │
├──────────────────┼────────────────────────┤
│ weight_decay │ 0.0005 │
├──────────────────┼────────────────────────┤
│ momentum │ 0.9 │
├──────────────────┼────────────────────────┤
│ exp_name │ 'title_coco' │
├──────────────────┼────────────────────────┤
│ test_size │ (640, 640) │
├──────────────────┼────────────────────────┤
│ test_conf │ 0.2 │
├──────────────────┼────────────────────────┤
│ nmsthre │ 0.65 │
╘══════════════════╧════════════════════════╛
2021-07-30 02:26:06.975 | INFO | yolox.core.trainer:before_train:134 - Model Summary: Params: 8.94M, Gflops: 26.65
2021-07-30 02:26:12.717 | INFO | apex.amp.frontend:initialize:328 - Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.
2021-07-30 02:26:12.719 | INFO | apex.amp.frontend:initialize:329 - Defaults for this optimization level are:
2021-07-30 02:26:12.719 | INFO | apex.amp.frontend:initialize:331 - enabled : True
2021-07-30 02:26:12.719 | INFO | apex.amp.frontend:initialize:331 - opt_level : O1
2021-07-30 02:26:12.720 | INFO | apex.amp.frontend:initialize:331 - cast_model_type : None
2021-07-30 02:26:12.720 | INFO | apex.amp.frontend:initialize:331 - patch_torch_functions : True
2021-07-30 02:26:12.720 | INFO | apex.amp.frontend:initialize:331 - keep_batchnorm_fp32 : None
2021-07-30 02:26:12.720 | INFO | apex.amp.frontend:initialize:331 - master_weights : None
2021-07-30 02:26:12.720 | INFO | apex.amp.frontend:initialize:331 - loss_scale : dynamic
2021-07-30 02:26:12.721 | INFO | apex.amp.frontend:initialize:336 - Processing user overrides (additional kwargs that are not None)...
2021-07-30 02:26:12.721 | INFO | apex.amp.frontend:initialize:354 - After processing overrides, optimization options are:
2021-07-30 02:26:12.722 | INFO | apex.amp.frontend:initialize:356 - enabled : True
2021-07-30 02:26:12.722 | INFO | apex.amp.frontend:initialize:356 - opt_level : O1
2021-07-30 02:26:12.722 | INFO | apex.amp.frontend:initialize:356 - cast_model_type : None
2021-07-30 02:26:12.722 | INFO | apex.amp.frontend:initialize:356 - patch_torch_functions : True
2021-07-30 02:26:12.722 | INFO | apex.amp.frontend:initialize:356 - keep_batchnorm_fp32 : None
2021-07-30 02:26:12.723 | INFO | apex.amp.frontend:initialize:356 - master_weights : None
2021-07-30 02:26:12.723 | INFO | apex.amp.frontend:initialize:356 - loss_scale : dynamic
2021-07-30 02:26:12.734 | INFO | yolox.core.trainer:resume_train:287 - loading checkpoint for fine tuning
2021-07-30 02:26:12.859 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.0.weight in checkpoint is torch.Size([80, 128, 1, 1]), while shape of head.cls_preds.0.weight in model is torch.Size([6, 128, 1, 1]).
2021-07-30 02:26:12.859 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.0.bias in checkpoint is torch.Size([80]), while shape of head.cls_preds.0.bias in model is torch.Size([6]).
2021-07-30 02:26:12.859 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.1.weight in checkpoint is torch.Size([80, 128, 1, 1]), while shape of head.cls_preds.1.weight in model is torch.Size([6, 128, 1, 1]).
2021-07-30 02:26:12.860 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.1.bias in checkpoint is torch.Size([80]), while shape of head.cls_preds.1.bias in model is torch.Size([6]).
2021-07-30 02:26:12.860 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.2.weight in checkpoint is torch.Size([80, 128, 1, 1]), while shape of head.cls_preds.2.weight in model is torch.Size([6, 128, 1, 1]).
2021-07-30 02:26:12.860 | WARNING | yolox.utils.checkpoint:load_ckpt:26 - Shape of head.cls_preds.2.bias in checkpoint is torch.Size([80]), while shape of head.cls_preds.2.bias in model is torch.Size([6]).
2021-07-30 02:26:12.889 | INFO | yolox.data.datasets.coco:__init__:54 - loading annotations into memory...
2021-07-30 02:26:13.038 | INFO | yolox.data.datasets.coco:__init__:54 - Done (t=0.15s)
2021-07-30 02:26:13.039 | INFO | pycocotools.coco:__init__:89 - creating index...
2021-07-30 02:26:13.049 | INFO | pycocotools.coco:__init__:89 - index created!
2021-07-30 02:26:13.050 | INFO | yolox.core.trainer:before_train:153 - init prefetcher, this might take one minute or less...
2021-07-30 02:26:20.933 | INFO | yolox.data.datasets.coco:__init__:54 - loading annotations into memory...
2021-07-30 02:26:20.948 | INFO | yolox.data.datasets.coco:__init__:54 - Done (t=0.01s)
2021-07-30 02:26:20.949 | INFO | pycocotools.coco:__init__:89 - creating index...
2021-07-30 02:26:20.951 | INFO | pycocotools.coco:__init__:89 - index created!
2021-07-30 02:26:20.954 | INFO | yolox.core.trainer:before_train:183 - Training start...
2021-07-30 02:26:20.956 | INFO | yolox.core.trainer:before_train:184 -
YOLOX(
(backbone): YOLOPAFPN(
(backbone): CSPDarknet(
(stem): Focus(
(conv): BaseConv(
(conv): Conv2d(12, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(dark2): Sequential(
(0): BaseConv(
(conv): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(32, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(dark3): Sequential(
(0): BaseConv(
(conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(dark4): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(dark5): Sequential(
(0): BaseConv(
(conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): SPPBottleneck(
(conv1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): MaxPool2d(kernel_size=5, stride=1, padding=2, dilation=1, ceil_mode=False)
(1): MaxPool2d(kernel_size=9, stride=1, padding=4, dilation=1, ceil_mode=False)
(2): MaxPool2d(kernel_size=13, stride=1, padding=6, dilation=1, ceil_mode=False)
)
(conv2): BaseConv(
(conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
)
(upsample): Upsample(scale_factor=2.0, mode=nearest)
(lateral_conv0): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_p4): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
(reduce_conv1): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_p3): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
(bu_conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_n3): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
(bu_conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_n4): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(head): YOLOXHead(
(cls_convs): ModuleList(
(0): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
(reg_convs): ModuleList(
(0): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
(cls_preds): ModuleList(
(0): Conv2d(128, 6, kernel_size=(1, 1), stride=(1, 1))
(1): Conv2d(128, 6, kernel_size=(1, 1), stride=(1, 1))
(2): Conv2d(128, 6, kernel_size=(1, 1), stride=(1, 1))
)
(reg_preds): ModuleList(
(0): Conv2d(128, 4, kernel_size=(1, 1), stride=(1, 1))
(1): Conv2d(128, 4, kernel_size=(1, 1), stride=(1, 1))
(2): Conv2d(128, 4, kernel_size=(1, 1), stride=(1, 1))
)
(obj_preds): ModuleList(
(0): Conv2d(128, 1, kernel_size=(1, 1), stride=(1, 1))
(1): Conv2d(128, 1, kernel_size=(1, 1), stride=(1, 1))
(2): Conv2d(128, 1, kernel_size=(1, 1), stride=(1, 1))
)
(stems): ModuleList(
(0): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(2): BaseConv(
(conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(l1_loss): L1Loss()
(bcewithlog_loss): BCEWithLogitsLoss()
(iou_loss): IOUloss()
)
)
2021-07-30 02:26:20.957 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch1
2021-07-30 02:26:26.408 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32768.0
2021-07-30 02:26:30.366 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-07-30 02:26:33.268 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 8192.0
2021-07-30 02:26:36.010 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 4096.0
2021-07-30 02:26:49.332 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 2048.0
2021-07-30 02:28:13.574 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 1024.0
2021-07-30 02:29:27.181 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 50/408, mem: 34780Mb, iter_time: 3.724s, data_time: 3.064s, total_loss: 11.4, iou_loss: 3.0, l1_loss: 0.0, conf_loss: 5.7, cls_loss: 2.7, lr: 3.004e-06, size: 736, ETA: 5 days, 6:34:13
2021-07-30 02:32:31.117 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 100/408, mem: 34780Mb, iter_time: 3.678s, data_time: 3.044s, total_loss: 8.1, iou_loss: 3.3, l1_loss: 0.0, conf_loss: 3.3, cls_loss: 1.5, lr: 1.201e-05, size: 704, ETA: 5 days, 5:44:27
2021-07-30 02:35:13.320 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 150/408, mem: 34780Mb, iter_time: 3.244s, data_time: 2.756s, total_loss: 7.5, iou_loss: 3.1, l1_loss: 0.0, conf_loss: 2.8, cls_loss: 1.7, lr: 2.703e-05, size: 672, ETA: 5 days, 0:30:37
2021-07-30 02:37:50.507 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 200/408, mem: 34780Mb, iter_time: 3.143s, data_time: 2.628s, total_loss: 7.0, iou_loss: 3.3, l1_loss: 0.0, conf_loss: 2.7, cls_loss: 1.1, lr: 4.806e-05, size: 544, ETA: 4 days, 21:01:16
2021-07-30 02:40:21.688 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 250/408, mem: 34780Mb, iter_time: 3.023s, data_time: 2.536s, total_loss: 6.3, iou_loss: 2.8, l1_loss: 0.0, conf_loss: 2.1, cls_loss: 1.5, lr: 7.509e-05, size: 448, ETA: 4 days, 18:05:42
2021-07-30 02:42:34.043 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 300/408, mem: 34780Mb, iter_time: 2.647s, data_time: 2.211s, total_loss: 6.1, iou_loss: 2.8, l1_loss: 0.0, conf_loss: 2.3, cls_loss: 1.1, lr: 1.081e-04, size: 576, ETA: 4 days, 14:00:07
2021-07-30 02:45:22.951 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 350/408, mem: 34780Mb, iter_time: 3.378s, data_time: 2.896s, total_loss: 6.2, iou_loss: 2.8, l1_loss: 0.0, conf_loss: 2.3, cls_loss: 1.0, lr: 1.472e-04, size: 768, ETA: 4 days, 14:36:29
2021-07-30 02:47:54.685 | INFO | yolox.core.trainer:after_iter:249 - epoch: 1/300, iter: 400/408, mem: 34780Mb, iter_time: 3.034s, data_time: 2.564s, total_loss: 6.7, iou_loss: 3.1, l1_loss: 0.0, conf_loss: 2.3, cls_loss: 1.3, lr: 1.922e-04, size: 544, ETA: 4 days, 13:35:47
2021-07-30 02:48:14.251 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:50:21.513 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:171 - Evaluate in main process...
2021-07-30 02:50:21.577 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - Loading and preparing results...
2021-07-30 02:50:21.590 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - DONE (t=0.01s)
2021-07-30 02:50:21.591 | INFO | pycocotools.coco:loadRes:365 - creating index...
2021-07-30 02:50:21.593 | INFO | pycocotools.coco:loadRes:365 - index created!
2021-07-30 02:50:22.148 | INFO | yolox.core.trainer:evaluate_and_save_model:302 -
Average forward time: 1.37 ms, Average NMS time: 0.76 ms, Average inference time: 2.13 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.125
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.315
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.080
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.075
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.157
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.147
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.192
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.212
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.212
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.143
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.255
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.195
2021-07-30 02:50:22.149 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:50:22.411 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch2
2021-07-30 02:50:35.583 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 50/408, mem: 34780Mb, iter_time: 0.263s, data_time: 0.000s, total_loss: 2.1, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.3, cls_loss: 0.4, lr: 2.520e-04, size: 480, ETA: 4 days, 2:05:29
2021-07-30 02:50:48.941 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 100/408, mem: 34780Mb, iter_time: 0.267s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.4, lr: 3.101e-04, size: 736, ETA: 3 days, 17:17:22
2021-07-30 02:51:02.116 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 150/408, mem: 34780Mb, iter_time: 0.263s, data_time: 0.000s, total_loss: 1.4, iou_loss: 0.9, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.3, lr: 3.741e-04, size: 736, ETA: 3 days, 10:03:12
2021-07-30 02:51:15.342 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 200/408, mem: 34780Mb, iter_time: 0.264s, data_time: 0.000s, total_loss: 1.8, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.4, lr: 4.441e-04, size: 448, ETA: 3 days, 4:00:35
2021-07-30 02:51:29.094 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 250/408, mem: 34780Mb, iter_time: 0.275s, data_time: 0.000s, total_loss: 1.3, iou_loss: 0.8, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 5.202e-04, size: 736, ETA: 2 days, 22:54:39
2021-07-30 02:51:42.203 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 300/408, mem: 34780Mb, iter_time: 0.262s, data_time: 0.000s, total_loss: 1.9, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.4, lr: 6.022e-04, size: 544, ETA: 2 days, 18:30:04
2021-07-30 02:51:55.727 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 350/408, mem: 34780Mb, iter_time: 0.270s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 6.903e-04, size: 512, ETA: 2 days, 14:41:28
2021-07-30 02:52:08.976 | INFO | yolox.core.trainer:after_iter:249 - epoch: 2/300, iter: 400/408, mem: 34780Mb, iter_time: 0.265s, data_time: 0.000s, total_loss: 2.3, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.4, cls_loss: 0.4, lr: 7.844e-04, size: 672, ETA: 2 days, 11:20:26
2021-07-30 02:52:11.123 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:54:35.554 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:171 - Evaluate in main process...
2021-07-30 02:54:35.600 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - Loading and preparing results...
2021-07-30 02:54:35.608 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - DONE (t=0.01s)
2021-07-30 02:54:35.609 | INFO | pycocotools.coco:loadRes:365 - creating index...
2021-07-30 02:54:35.611 | INFO | pycocotools.coco:loadRes:365 - index created!
2021-07-30 02:54:36.181 | INFO | yolox.core.trainer:evaluate_and_save_model:302 -
Average forward time: 1.21 ms, Average NMS time: 0.69 ms, Average inference time: 1.90 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.051
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.152
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.024
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.031
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.081
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.044
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.074
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.083
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.083
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.061
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.111
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.047
2021-07-30 02:54:36.182 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:54:36.351 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch3
2021-07-30 02:54:49.829 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 50/408, mem: 34780Mb, iter_time: 0.269s, data_time: 0.000s, total_loss: 1.7, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.4, lr: 9.010e-04, size: 544, ETA: 2 days, 7:56:53
2021-07-30 02:55:02.990 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 100/408, mem: 34780Mb, iter_time: 0.263s, data_time: 0.000s, total_loss: 1.7, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.4, lr: 1.008e-03, size: 480, ETA: 2 days, 5:21:23
2021-07-30 02:55:16.422 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 150/408, mem: 34780Mb, iter_time: 0.268s, data_time: 0.000s, total_loss: 1.9, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.4, lr: 1.121e-03, size: 576, ETA: 2 days, 3:02:33
2021-07-30 02:55:29.610 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 200/408, mem: 34780Mb, iter_time: 0.263s, data_time: 0.000s, total_loss: 1.6, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.4, lr: 1.240e-03, size: 576, ETA: 2 days, 0:56:51
2021-07-30 02:55:43.004 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 250/408, mem: 34780Mb, iter_time: 0.268s, data_time: 0.000s, total_loss: 1.4, iou_loss: 0.9, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.3, lr: 1.365e-03, size: 448, ETA: 1 day, 23:03:20
2021-07-30 02:55:56.723 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 300/408, mem: 34780Mb, iter_time: 0.274s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 1.496e-03, size: 640, ETA: 1 day, 21:20:32
2021-07-30 02:56:09.944 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 350/408, mem: 34780Mb, iter_time: 0.264s, data_time: 0.000s, total_loss: 1.6, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.4, lr: 1.633e-03, size: 672, ETA: 1 day, 19:45:41
2021-07-30 02:56:23.153 | INFO | yolox.core.trainer:after_iter:249 - epoch: 3/300, iter: 400/408, mem: 34780Mb, iter_time: 0.264s, data_time: 0.000s, total_loss: 1.9, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.4, lr: 1.777e-03, size: 800, ETA: 1 day, 18:18:36
2021-07-30 02:56:25.319 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:58:47.554 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:171 - Evaluate in main process...
2021-07-30 02:58:47.599 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - Loading and preparing results...
2021-07-30 02:58:47.610 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - DONE (t=0.01s)
2021-07-30 02:58:47.610 | INFO | pycocotools.coco:loadRes:365 - creating index...
2021-07-30 02:58:47.612 | INFO | pycocotools.coco:loadRes:365 - index created!
2021-07-30 02:58:48.176 | INFO | yolox.core.trainer:evaluate_and_save_model:302 -
Average forward time: 1.30 ms, Average NMS time: 0.66 ms, Average inference time: 1.96 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.032
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.121
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.006
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.024
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.047
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.009
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.050
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.058
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.058
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.051
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.066
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.009
2021-07-30 02:58:48.177 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 02:58:48.365 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch4
2021-07-30 02:59:01.325 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 50/408, mem: 34780Mb, iter_time: 0.259s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 1.950e-03, size: 672, ETA: 1 day, 16:45:48
2021-07-30 02:59:14.808 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 100/408, mem: 34780Mb, iter_time: 0.269s, data_time: 0.000s, total_loss: 1.2, iou_loss: 0.7, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.3, lr: 2.106e-03, size: 512, ETA: 1 day, 15:33:00
2021-07-30 02:59:27.748 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 150/408, mem: 34780Mb, iter_time: 0.258s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 2.268e-03, size: 704, ETA: 1 day, 14:24:40
2021-07-30 02:59:40.808 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 200/408, mem: 34780Mb, iter_time: 0.261s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 2.436e-03, size: 800, ETA: 1 day, 13:21:18
2021-07-30 02:59:54.240 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 250/408, mem: 34780Mb, iter_time: 0.268s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.4, lr: 2.610e-03, size: 576, ETA: 1 day, 12:22:43
2021-07-30 03:00:07.417 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 300/408, mem: 34780Mb, iter_time: 0.263s, data_time: 0.000s, total_loss: 1.8, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.4, lr: 2.790e-03, size: 576, ETA: 1 day, 11:27:38
2021-07-30 03:00:20.557 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 350/408, mem: 34780Mb, iter_time: 0.262s, data_time: 0.000s, total_loss: 1.4, iou_loss: 0.9, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 2.977e-03, size: 608, ETA: 1 day, 10:35:59
2021-07-30 03:00:33.496 | INFO | yolox.core.trainer:after_iter:249 - epoch: 4/300, iter: 400/408, mem: 34780Mb, iter_time: 0.258s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.4, lr: 3.169e-03, size: 640, ETA: 1 day, 9:47:16
2021-07-30 03:00:35.538 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 03:02:58.843 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:171 - Evaluate in main process...
2021-07-30 03:02:58.906 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - Loading and preparing results...
2021-07-30 03:02:59.040 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - DONE (t=0.13s)
2021-07-30 03:02:59.041 | INFO | pycocotools.coco:loadRes:365 - creating index...
2021-07-30 03:02:59.044 | INFO | pycocotools.coco:loadRes:365 - index created!
2021-07-30 03:02:59.544 | INFO | yolox.core.trainer:evaluate_and_save_model:302 -
Average forward time: 1.23 ms, Average NMS time: 0.67 ms, Average inference time: 1.90 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.028
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.106
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.004
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.031
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.002
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.046
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.057
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.057
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.054
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.045
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.002
2021-07-30 03:02:59.545 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 03:02:59.750 | INFO | yolox.core.trainer:before_epoch:192 - ---> start train epoch5
2021-07-30 03:03:13.585 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 50/408, mem: 34780Mb, iter_time: 0.276s, data_time: 0.000s, total_loss: 1.6, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.4, lr: 3.399e-03, size: 480, ETA: 1 day, 8:55:23
2021-07-30 03:03:26.718 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 100/408, mem: 34780Mb, iter_time: 0.262s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.3, lr: 3.604e-03, size: 800, ETA: 1 day, 8:12:48
2021-07-30 03:03:40.222 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 150/408, mem: 34780Mb, iter_time: 0.270s, data_time: 0.000s, total_loss: 1.3, iou_loss: 0.9, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 3.815e-03, size: 512, ETA: 1 day, 7:33:00
2021-07-30 03:03:53.385 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 200/408, mem: 34780Mb, iter_time: 0.263s, data_time: 0.000s, total_loss: 1.4, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 4.032e-03, size: 672, ETA: 1 day, 6:54:59
2021-07-30 03:04:06.230 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 250/408, mem: 34780Mb, iter_time: 0.257s, data_time: 0.000s, total_loss: 1.3, iou_loss: 0.9, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 4.255e-03, size: 736, ETA: 1 day, 6:18:39
2021-07-30 03:04:19.488 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 300/408, mem: 34780Mb, iter_time: 0.265s, data_time: 0.000s, total_loss: 1.5, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.3, lr: 4.485e-03, size: 480, ETA: 1 day, 5:44:37
2021-07-30 03:04:32.388 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 350/408, mem: 34780Mb, iter_time: 0.258s, data_time: 0.000s, total_loss: 1.8, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.2, cls_loss: 0.4, lr: 4.720e-03, size: 640, ETA: 1 day, 5:11:55
2021-07-30 03:04:45.303 | INFO | yolox.core.trainer:after_iter:249 - epoch: 5/300, iter: 400/408, mem: 34780Mb, iter_time: 0.258s, data_time: 0.000s, total_loss: 1.7, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.1, cls_loss: 0.4, lr: 4.961e-03, size: 640, ETA: 1 day, 4:40:50
2021-07-30 03:04:47.620 | INFO | yolox.core.trainer:save_ckpt:311 - Save weights to ./data/yolox_output/title_coco
2021-07-30 03:07:12.742 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:171 - Evaluate in main process...
2021-07-30 03:07:12.817 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - Loading and preparing results...
2021-07-30 03:07:12.832 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:204 - DONE (t=0.01s)
2021-07-30 03:07:12.833 | INFO | pycocotools.coco:loadRes:365 - creating index...
2021-07-30 03:07:12.836 | INFO | pycocotools.coco:loadRes:365 - index created!
2021-07-30 03:07:13.362 | INFO | yolox.core.trainer:evaluate_and_save_model:302 -
Average forward time: 1.24 ms, Average NMS time: 0.68 ms, Average inference time: 1.93 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.022
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.093
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.020
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.019
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.002
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.039
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.051
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.051
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.050
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.032
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.002Hi, your problem seems hard to locate, but we have noticed something wrong with our all_reduc_norm at small models. So could you comment this line https://github.com/Megvii-BaseDetection/YOLOX/blob/main/yolox/core/trainer.py#L214 and try it again for several epochs? Thx!
GOATmessi8
commented
3 years ago Could you try a tiny model? For now, we retrain and check the tiny model, everything looks fine. But for yolox-s model, we also have some weird issues like unstable loss and performance and are looking for the problem.
Joker316701882
commented
3 years ago @wangyuan111 We have located the bug in mixup. Please pull our latest branch and try it again!
wangyuan111
commented
3 years ago Hi, your problem seems hard to locate, but we have noticed something wrong with our all_reduc_norm at small models. So could you comment this line https://github.com/Megvii-BaseDetection/YOLOX/blob/main/yolox/core/trainer.py#L214 and try it again for several epochs? Thx!
I tried to delete this line, but it didn't work @ruinmessi
wangyuan111
commented
3 years ago @wangyuan111 We have located the bug in mixup. Please pull our latest branch and try it again!
Thank you. I'll try @Joker316701882 @ruinmessi
wangyuan111
commented
3 years ago I have pulled the latest code and started training again. mAP seems to be back to normal. Thank you @Joker316701882 @ruinmessi
BatmanofZuhandArrgh
commented
3 years ago Closing the mixup aug and setting the mosaic scale from (0,1, 2) to (0.5, 1.5) may be helpful
But your final loss is so weird (too small). I suggest you to eval on your training set to check the case of overfitting.
And could you share your training log file?
Hi, to improve detecting small objects, should I not do mixup aug at all? Is it because it would make small objects even less detailed? Also, what's the reasoning behind closing the mosaic scale like that? @ruinmessi