search

CoCoPIE-Group / XGen-Report

The repository for reporting issues about CoCoPIE XGen.
0 stars 0 forks source link

The Error when train the Yolov8 with built-in configuration #38

Open hsung2 opened 10 months ago

hsung2 commented 10 months ago

Dear author,

I keep getting this error after I did train once for Yolov8.

Here is the xgen_train.log:

2023-10-24T01:08:24.499131+0000 - DEBUG - xgen_scripts.main:93 - XGen is running in /root/output/YOLOv8_CoCo2017/20231024010820, xgen_logger_setup in xgen_scripts.py
2023-10-24T01:08:25.196550+0000 - INFO - core.training:374 - Your current workplace is /root/output/YOLOv8_CoCo2017/20231024010820
2023-10-24T01:08:25.197096+0000 - INFO - core.training:390 - A new search is started!
2023-10-24T01:08:25.198375+0000 - DEBUG - task_gen.get_default_scenarios_plan:76 - default_plan: {'1': '1', '2': '2', '3': '1'}
xgen-config-path:  /root/output/YOLOv8_CoCo2017/20231024010820/xgen_config.json
xgen-workplace:  /root/output/YOLOv8_CoCo2017/20231024010820
xgen-resume:  False
xgen-mode:  scaling
xgen-pretrained-model-path:  /root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/xgen.pt
detail args:  
{
    'origin': {
        'common_train_epochs': 10,
        'root_path': './Xgen/',
        'pretrain_model_weights_path': None,
        'train_data_path': '/data/object-detection-yolov6/coco',
        'train_label_path': None,
        'eval_data_path': '/data/object-detection-yolov6/coco',
        'eval_label_path': None,
        'learning_rate': 0.01,
        'batch_size': 16,
        'data': '/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/coco.yaml',
        'conf_file': 'yolov8m.yaml',
        'weights': None,
        'device': None,
        'imgsz': 640,
        'width_multiple': 0.5,
        'depth_multiple': 0.33,
        'scaling_factor': 1,
        'workers': 16,
        'noplots': True,
        'num_classes': 80,
        'device_num': 1
    },
    'general': {'user_id': 'test', 'work_place': '/root/output/YOLOv8_CoCo2017/20231024010820', 'random_seed': 3407, 'enable_ddp': False, 'CUDA_VISIBLE_DEVICES': '0', 'tran_scripts_path': None},
    'prune': {
        'sp_store_weights': None,
        'sp_lars': False,
        'sp_lars_trust_coef': 0.001,
        'sp_backbone': False,
        'sp_retrain': False,
        'sp_admm': False,
        'sp_admm_multi': False,
        'sp_retrain_multi': False,
        'sp_config_file': None,
        'sp_subset_progressive': False,
        'sp_admm_fixed_params': False,
        'sp_no_harden': False,
        'nv_sparse': False,
        'sp_load_prune_params': None,
        'sp_store_prune_params': None,
        'generate_rand_seq_gap_yaml': False,
        'sp_admm_update_epoch': 5,
        'sp_admm_update_batch': None,
        'sp_admm_rho': 0.001,
        'sparsity_type': 'block_punched',
        'sp_admm_lr': 0.01,
        'admm_debug': False,
        'sp_global_weight_sparsity': False,
        'sp_prune_threshold': -1.0,
        'sp_block_irregular_sparsity': '(0,0)',
        'sp_block_permute_multiplier': 2,
        'sp_admm_block': '(8,4)',
        'sp_admm_buckets_num': 16,
        'sp_admm_elem_per_row': 1,
        'sp_admm_tile': None,
        'sp_admm_select_number': 4,
        'sp_admm_pattern_row_sub': 1,
        'sp_admm_pattern_col_sub': 4,
        'sp_admm_data_format': None,
        'sp_admm_do_not_permute_conv': False,
        'sp_gs_output_v': None,
        'sp_gs_output_ptr': None,
        'sp_load_frozen_weights': None,
        'retrain_mask_pattern': 'weight',
        'sp_update_init_method': 'weight',
        'sp_mask_update_freq': 10,
        'retrain_mask_sparsity': -1.0,
        'retrain_mask_seed': None,
        'sp_prune_before_retrain': False,
        'output_compressed_format': False,
        'sp_grad_update': False,
        'sp_grad_decay': 0.98,
        'sp_grad_restore_threshold': -1,
        'sp_global_magnitude': False,
        'sp_pre_defined_mask_dir': None,
        'sp_prune_ratios': 0
    },
    'quantization': {
        'qt_aimet': False,
        'qat': True,
        'fold_layers': True,
        'cross_layer_equalization': False,
        'bias_correction': True,
        'rounding_mode': 'nearest',
        'num_quant_samples': 1000,
        'num_bias_correct_samples': 1000,
        'weight_bw': 8,
        'act_bw': 8,
        'quant_scheme': 'tf_enhanced',
        'layers_to_ignore': [],
        'auto_add_bias': True,
        'perform_only_empirical_bias_corr': True
    },
    'pas': {'pas_ratio': 0, 'pas': False, 'limit_loss_weights': 5.0, 'use_limit_loss': False, 'pas_debug': False, 'pas_rebuild': False, 'pas_finetune_epoch': 200, 'pas_pretrained_weight_path': None, 'pas_ignore': ['neck', 'detect', 'cv'], 'pas_searching_ratio': [0.1, 0.2, 0.3]},
    'task': {'specific_scenarios': 'BasicScaling', 'pretrained_model_path': '/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/xgen.pt', 'state': {'stage': 0, 'cycles': 0}, 'max_searching': 10, 'args_2': {'cycles': 10}},
    'user_requirements': {
        'power': None,
        'accuracy': 0.35,
        'accuracy_reverse_yn': 0,
        'model_size': None,
        'memory_size': None,
        'latency': 75.0,
        'margin': 7.5,
        'primary_type': 'latency',
        'primary_range': '<',
        'secondary_type': 'accuracy',
        'secondary_range': '>',
        'searching_variable': 'scaling_factor',
        'searching_range': [0.2, 1],
        'searching_step_size': 0.05,
        'searching_pas_variable': 'pas',
        'express_path': True,
        'target_type': 'latency',
        'searching_granularity': None,
        'using_default_dataset': True,
        'user_model': 'YOLOv8',
        'using_express_path': True,
        'express_mode': 0,
        'use_distillation': False,
        'use_default_distillation_model': True,
        'is_training': True
    },
    'train': {'common_save_best_yn': 1, 'trained_yn': False, 'larger_better': True},
    'compiler': {
        'input_shape': '(1,3,640,640)',
        'opset_version': 11,
        'devices': [
            {
                'task_queue_size': 0,
                'device': {
                    'uuid': 'R5CRC1NFW2E',
                    'device_type': 'android',
                    'connection_status': 'available',
                    'task_status': 'idle',
                    'info': {'uuid': 'R5CRC1NFW2E', 'cpu': 'SM8350', 'gpu': 'Qualcomm, Adreno (TM) 660', 'memory': '5.24 GB', 'battery': '100', 'brand': 'samsung', 'model': 'SM-G990U1', 'os_type': 'android'}
                },
                'agent_id': 'agent-localhost'
            }
        ],
        'ios_devices': []
    },
    'distillation': {
        'distillation_method': 'classic_distillation',
        'enable_ddp': False,
        'enable_dp': False,
        'input_shape': None,
        'original_loss_weights': 0.1,
        'tag_loss_weights': 0.9,
        'tag_loss': 'kl',
        'tag_temperature': 4,
        'tag_loss_combination_method': 'avg',
        'feature_loss_weights': 0.9,
        'feature_default_temperature': 1,
        'advance_feature_mapping': {},
        'regularization_loss_weights': 1,
        'regularization_loss_types': [['tag_discriminator', 1]],
        'discriminator_lr': 0.0001
    }
}
Current search total stages:  3
           Current search stages info           
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Stage       ┃ Max search cycles              ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 1           │ 1                              │
│ 2           │ 10                             │
│ 3           │ 1                              │
└─────────────┴────────────────────────────────┘
2023-10-24T01:08:25.236168+0000 - INFO - core.training:427 - Current Session ID: session-xgen-32a44e70
2023-10-24T01:08:25.246143+0000 - DEBUG - task_gen.task_gen:48 - job_list: [['{"origin": {"common_train_epochs": 0, "root_path": "./Xgen/", "pretrain_model_weights_path": "/root/Projects/.checkpoints/yolov8/yolov8s_xgen.pt", "train_data_path": "/data/object-detection-yolov6/coco", "train_label_path": null, "eval_data_path": "/data/object-detection-yolov6/coco", "eval_label_path": null, "learning_rate": 0.01, "batch_size": 16, "data": "/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/coco.yaml", "conf_file": "yolov8s.yaml", "weights": null, "device": null, "imgsz": 640, "width_multiple": 0.5, "depth_multiple": 0.33, "scaling_factor": 1, "workers": 16, "noplots": true, "num_classes": 80, "device_num": 1}, "general": {"user_id": "test", "work_place": "/root/output/YOLOv8_CoCo2017/20231024010820", "random_seed": 3407, "enable_ddp": false, "CUDA_VISIBLE_DEVICES": "0", "tran_scripts_path": null}, "prune": {"sp_store_weights": null, "sp_lars": false, "sp_lars_trust_coef": 0.001, "sp_backbone": false, "sp_retrain": false, "sp_admm": false, "sp_admm_multi": false, "sp_retrain_multi": false, "sp_config_file": null, "sp_subset_progressive": false, "sp_admm_fixed_params": false, "sp_no_harden": false, "nv_sparse": false, "sp_load_prune_params": null, "sp_store_prune_params": null, "generate_rand_seq_gap_yaml": false, "sp_admm_update_epoch": 5, "sp_admm_update_batch": null, "sp_admm_rho": 0.001, "sparsity_type": "block_punched", "sp_admm_lr": 0.01, "admm_debug": false, "sp_global_weight_sparsity": false, "sp_prune_threshold": -1.0, "sp_block_irregular_sparsity": "(0,0)", "sp_block_permute_multiplier": 2, "sp_admm_block": "(8,4)", "sp_admm_buckets_num": 16, "sp_admm_elem_per_row": 1, "sp_admm_tile": null, "sp_admm_select_number": 4, "sp_admm_pattern_row_sub": 1, "sp_admm_pattern_col_sub": 4, "sp_admm_data_format": null, "sp_admm_do_not_permute_conv": false, "sp_gs_output_v": null, "sp_gs_output_ptr": null, "sp_load_frozen_weights": null, "retrain_mask_pattern": "weight", "sp_update_init_method": "weight", "sp_mask_update_freq": 10, "retrain_mask_sparsity": -1.0, "retrain_mask_seed": null, "sp_prune_before_retrain": false, "output_compressed_format": false, "sp_grad_update": false, "sp_grad_decay": 0.98, "sp_grad_restore_threshold": -1, "sp_global_magnitude": false, "sp_pre_defined_mask_dir": null, "sp_prune_ratios": 0}, "quantization": {"qt_aimet": false, "qat": true, "fold_layers": true, "cross_layer_equalization": false, "bias_correction": true, "rounding_mode": "nearest", "num_quant_samples": 1000, "num_bias_correct_samples": 1000, "weight_bw": 8, "act_bw": 8, "quant_scheme": "tf_enhanced", "layers_to_ignore": [], "auto_add_bias": true, "perform_only_empirical_bias_corr": true}, "pas": {"pas_ratio": 0, "pas": false, "limit_loss_weights": 5.0, "use_limit_loss": false, "pas_debug": false, "pas_rebuild": false, "pas_finetune_epoch": 200, "pas_pretrained_weight_path": null, "pas_ignore": ["neck", "detect", "cv"], "pas_searching_ratio": [0.1, 0.2, 0.3]}, "task": {"specific_scenarios": "BasicScaling", "pretrained_model_path": "/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/xgen.pt", "state": {"stage": 1, "cycles": 0}, "max_searching": 10, "args_2": {"cycles": 10}, "args_1": {"cycles": 1}}, "user_requirements": {"power": null, "accuracy": 0.35, "accuracy_reverse_yn": 0, "model_size": null, "memory_size": null, "latency": 75.0, "margin": 7.5, "primary_type": "latency", "primary_range": "<", "secondary_type": "accuracy", "secondary_range": ">", "searching_variable": "scaling_factor", "searching_range": [0.2, 1], "searching_step_size": 0.05, "searching_pas_variable": "pas", "express_path": true, "target_type": "latency", "searching_granularity": null, "using_default_dataset": true, "user_model": "YOLOv8", "using_express_path": true, "express_mode": 0, "use_distillation": false, "use_default_distillation_model": true, "is_training": true}, "train": {"common_save_best_yn": 1, "trained_yn": true, "larger_better": true, "uuid": "9ae04efe-469c-42"}, "compiler": {"input_shape": "(1,3,640,640)", "opset_version": 11, "devices": [{"task_queue_size": 0, "device": {"uuid": "R5CRC1NFW2E", "device_type": "android", "connection_status": "available", "task_status": "idle", "info": {"uuid": "R5CRC1NFW2E", "cpu": "SM8350", "gpu": "Qualcomm, Adreno (TM) 660", "memory": "5.24 GB", "battery": "100", "brand": "samsung", "model": "SM-G990U1", "os_type": "android"}}, "agent_id": "agent-localhost"}], "ios_devices": []}, "distillation": {"distillation_method": "classic_distillation", "enable_ddp": false, "enable_dp": false, "input_shape": null, "original_loss_weights": 0.1, "tag_loss_weights": 0.9, "tag_loss": "kl", "tag_temperature": 4, "tag_loss_combination_method": "avg", "feature_loss_weights": 0.9, "feature_default_temperature": 1, "advance_feature_mapping": {}, "regularization_loss_weights": 1, "regularization_loss_types": [["tag_discriminator", 1]], "discriminator_lr": 0.0001}}'], ['{"origin": {"common_train_epochs": 0, "root_path": "./Xgen/", "pretrain_model_weights_path": "/root/Projects/.checkpoints/yolov8/yolov8n_xgen.pt", "train_data_path": "/data/object-detection-yolov6/coco", "train_label_path": null, "eval_data_path": "/data/object-detection-yolov6/coco", "eval_label_path": null, "learning_rate": 0.01, "batch_size": 16, "data": "/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/coco.yaml", "conf_file": "yolov8n.yaml", "weights": null, "device": null, "imgsz": 640, "width_multiple": 0.5, "depth_multiple": 0.33, "scaling_factor": 1, "workers": 16, "noplots": true, "num_classes": 80, "device_num": 1}, "general": {"user_id": "test", "work_place": "/root/output/YOLOv8_CoCo2017/20231024010820", "random_seed": 3407, "enable_ddp": false, "CUDA_VISIBLE_DEVICES": "0", "tran_scripts_path": null}, "prune": {"sp_store_weights": null, "sp_lars": false, "sp_lars_trust_coef": 0.001, "sp_backbone": false, "sp_retrain": false, "sp_admm": false, "sp_admm_multi": false, "sp_retrain_multi": false, "sp_config_file": null, "sp_subset_progressive": false, "sp_admm_fixed_params": false, "sp_no_harden": false, "nv_sparse": false, "sp_load_prune_params": null, "sp_store_prune_params": null, "generate_rand_seq_gap_yaml": false, "sp_admm_update_epoch": 5, "sp_admm_update_batch": null, "sp_admm_rho": 0.001, "sparsity_type": "block_punched", "sp_admm_lr": 0.01, "admm_debug": false, "sp_global_weight_sparsity": false, "sp_prune_threshold": -1.0, "sp_block_irregular_sparsity": "(0,0)", "sp_block_permute_multiplier": 2, "sp_admm_block": "(8,4)", "sp_admm_buckets_num": 16, "sp_admm_elem_per_row": 1, "sp_admm_tile": null, "sp_admm_select_number": 4, "sp_admm_pattern_row_sub": 1, "sp_admm_pattern_col_sub": 4, "sp_admm_data_format": null, "sp_admm_do_not_permute_conv": false, "sp_gs_output_v": null, "sp_gs_output_ptr": null, "sp_load_frozen_weights": null, "retrain_mask_pattern": "weight", "sp_update_init_method": "weight", "sp_mask_update_freq": 10, "retrain_mask_sparsity": -1.0, "retrain_mask_seed": null, "sp_prune_before_retrain": false, "output_compressed_format": false, "sp_grad_update": false, "sp_grad_decay": 0.98, "sp_grad_restore_threshold": -1, "sp_global_magnitude": false, "sp_pre_defined_mask_dir": null, "sp_prune_ratios": 0}, "quantization": {"qt_aimet": false, "qat": true, "fold_layers": true, "cross_layer_equalization": false, "bias_correction": true, "rounding_mode": "nearest", "num_quant_samples": 1000, "num_bias_correct_samples": 1000, "weight_bw": 8, "act_bw": 8, "quant_scheme": "tf_enhanced", "layers_to_ignore": [], "auto_add_bias": true, "perform_only_empirical_bias_corr": true}, "pas": {"pas_ratio": 0, "pas": false, "limit_loss_weights": 5.0, "use_limit_loss": false, "pas_debug": false, "pas_rebuild": false, "pas_finetune_epoch": 200, "pas_pretrained_weight_path": null, "pas_ignore": ["neck", "detect", "cv"], "pas_searching_ratio": [0.1, 0.2, 0.3]}, "task": {"specific_scenarios": "BasicScaling", "pretrained_model_path": "/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/xgen.pt", "state": {"stage": 1, "cycles": 0}, "max_searching": 10, "args_2": {"cycles": 10}, "args_1": {"cycles": 1}}, "user_requirements": {"power": null, "accuracy": 0.35, "accuracy_reverse_yn": 0, "model_size": null, "memory_size": null, "latency": 75.0, "margin": 7.5, "primary_type": "latency", "primary_range": "<", "secondary_type": "accuracy", "secondary_range": ">", "searching_variable": "scaling_factor", "searching_range": [0.2, 1], "searching_step_size": 0.05, "searching_pas_variable": "pas", "express_path": true, "target_type": "latency", "searching_granularity": null, "using_default_dataset": true, "user_model": "YOLOv8", "using_express_path": true, "express_mode": 0, "use_distillation": false, "use_default_distillation_model": true, "is_training": true}, "train": {"common_save_best_yn": 1, "trained_yn": true, "larger_better": true, "uuid": "00a9c858-6655-4e"}, "compiler": {"input_shape": "(1,3,640,640)", "opset_version": 11, "devices": [{"task_queue_size": 0, "device": {"uuid": "R5CRC1NFW2E", "device_type": "android", "connection_status": "available", "task_status": "idle", "info": {"uuid": "R5CRC1NFW2E", "cpu": "SM8350", "gpu": "Qualcomm, Adreno (TM) 660", "memory": "5.24 GB", "battery": "100", "brand": "samsung", "model": "SM-G990U1", "os_type": "android"}}, "agent_id": "agent-localhost"}], "ios_devices": []}, "distillation": {"distillation_method": "classic_distillation", "enable_ddp": false, "enable_dp": false, "input_shape": null, "original_loss_weights": 0.1, "tag_loss_weights": 0.9, "tag_loss": "kl", "tag_temperature": 4, "tag_loss_combination_method": "avg", "feature_loss_weights": 0.9, "feature_default_temperature": 1, "advance_feature_mapping": {}, "regularization_loss_weights": 1, "regularization_loss_types": [["tag_discriminator", 1]], "discriminator_lr": 0.0001}}'], ['{"origin": {"common_train_epochs": 0, "root_path": "./Xgen/", "pretrain_model_weights_path": "/root/Projects/.checkpoints/yolov8/yolov8m_xgen.pt", "train_data_path": "/data/object-detection-yolov6/coco", "train_label_path": null, "eval_data_path": "/data/object-detection-yolov6/coco", "eval_label_path": null, "learning_rate": 0.01, "batch_size": 16, "data": "/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/coco.yaml", "conf_file": "yolov8m.yaml", "weights": null, "device": null, "imgsz": 640, "width_multiple": 0.5, "depth_multiple": 0.33, "scaling_factor": 1, "workers": 16, "noplots": true, "num_classes": 80, "device_num": 1}, "general": {"user_id": "test", "work_place": "/root/output/YOLOv8_CoCo2017/20231024010820", "random_seed": 3407, "enable_ddp": false, "CUDA_VISIBLE_DEVICES": "0", "tran_scripts_path": null}, "prune": {"sp_store_weights": null, "sp_lars": false, "sp_lars_trust_coef": 0.001, "sp_backbone": false, "sp_retrain": false, "sp_admm": false, "sp_admm_multi": false, "sp_retrain_multi": false, "sp_config_file": null, "sp_subset_progressive": false, "sp_admm_fixed_params": false, "sp_no_harden": false, "nv_sparse": false, "sp_load_prune_params": null, "sp_store_prune_params": null, "generate_rand_seq_gap_yaml": false, "sp_admm_update_epoch": 5, "sp_admm_update_batch": null, "sp_admm_rho": 0.001, "sparsity_type": "block_punched", "sp_admm_lr": 0.01, "admm_debug": false, "sp_global_weight_sparsity": false, "sp_prune_threshold": -1.0, "sp_block_irregular_sparsity": "(0,0)", "sp_block_permute_multiplier": 2, "sp_admm_block": "(8,4)", "sp_admm_buckets_num": 16, "sp_admm_elem_per_row": 1, "sp_admm_tile": null, "sp_admm_select_number": 4, "sp_admm_pattern_row_sub": 1, "sp_admm_pattern_col_sub": 4, "sp_admm_data_format": null, "sp_admm_do_not_permute_conv": false, "sp_gs_output_v": null, "sp_gs_output_ptr": null, "sp_load_frozen_weights": null, "retrain_mask_pattern": "weight", "sp_update_init_method": "weight", "sp_mask_update_freq": 10, "retrain_mask_sparsity": -1.0, "retrain_mask_seed": null, "sp_prune_before_retrain": false, "output_compressed_format": false, "sp_grad_update": false, "sp_grad_decay": 0.98, "sp_grad_restore_threshold": -1, "sp_global_magnitude": false, "sp_pre_defined_mask_dir": null, "sp_prune_ratios": 0}, "quantization": {"qt_aimet": false, "qat": true, "fold_layers": true, "cross_layer_equalization": false, "bias_correction": true, "rounding_mode": "nearest", "num_quant_samples": 1000, "num_bias_correct_samples": 1000, "weight_bw": 8, "act_bw": 8, "quant_scheme": "tf_enhanced", "layers_to_ignore": [], "auto_add_bias": true, "perform_only_empirical_bias_corr": true}, "pas": {"pas_ratio": 0, "pas": false, "limit_loss_weights": 5.0, "use_limit_loss": false, "pas_debug": false, "pas_rebuild": false, "pas_finetune_epoch": 200, "pas_pretrained_weight_path": null, "pas_ignore": ["neck", "detect", "cv"], "pas_searching_ratio": [0.1, 0.2, 0.3]}, "task": {"specific_scenarios": "BasicScaling", "pretrained_model_path": "/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/xgen.pt", "state": {"stage": 1, "cycles": 0}, "max_searching": 10, "args_2": {"cycles": 10}, "args_1": {"cycles": 1}}, "user_requirements": {"power": null, "accuracy": 0.35, "accuracy_reverse_yn": 0, "model_size": null, "memory_size": null, "latency": 75.0, "margin": 7.5, "primary_type": "latency", "primary_range": "<", "secondary_type": "accuracy", "secondary_range": ">", "searching_variable": "scaling_factor", "searching_range": [0.2, 1], "searching_step_size": 0.05, "searching_pas_variable": "pas", "express_path": true, "target_type": "latency", "searching_granularity": null, "using_default_dataset": true, "user_model": "YOLOv8", "using_express_path": true, "express_mode": 0, "use_distillation": false, "use_default_distillation_model": true, "is_training": true}, "train": {"common_save_best_yn": 1, "trained_yn": true, "larger_better": true, "uuid": "96c7ab72-553a-4c"}, "compiler": {"input_shape": "(1,3,640,640)", "opset_version": 11, "devices": [{"task_queue_size": 0, "device": {"uuid": "R5CRC1NFW2E", "device_type": "android", "connection_status": "available", "task_status": "idle", "info": {"uuid": "R5CRC1NFW2E", "cpu": "SM8350", "gpu": "Qualcomm, Adreno (TM) 660", "memory": "5.24 GB", "battery": "100", "brand": "samsung", "model": "SM-G990U1", "os_type": "android"}}, "agent_id": "agent-localhost"}], "ios_devices": []}, "distillation": {"distillation_method": "classic_distillation", "enable_ddp": false, "enable_dp": false, "input_shape": null, "original_loss_weights": 0.1, "tag_loss_weights": 0.9, "tag_loss": "kl", "tag_temperature": 4, "tag_loss_combination_method": "avg", "feature_loss_weights": 0.9, "feature_default_temperature": 1, "advance_feature_mapping": {}, "regularization_loss_weights": 1, "regularization_loss_types": [["tag_discriminator", 1]], "discriminator_lr": 0.0001}}']]
Using express path to find a suitable model...
processing job 1/3
2023-10-24T01:08:25.249064+0000 - INFO - train_module.model_train_main:155 - MKL_THREADING_LAYER=GNU CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0 python train_script_main.py
2023-10-24T01:08:25.249362+0000 - DEBUG - train_module.model_train_main:156 - dp mode
2023-10-24T01:08:25.249477+0000 - DEBUG - sys.run_cmd_with_logger:25 - Running command: MKL_THREADING_LAYER=GNU CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0 python train_script_main.py
2023-10-24T01:08:29.192012+0000 - INFO - sys.run_cmd_with_logger:32 - 
2023-10-24T01:08:29.192599+0000 - INFO - sys.run_cmd_with_logger:32 -                    from  n    params  module                                       arguments
2023-10-24T01:08:29.212763+0000 - INFO - sys.run_cmd_with_logger:32 -   0                  -1  1       928  ultralytics.nn.modules.conv.Conv             [3, 32, 3, 2]
2023-10-24T01:08:29.213227+0000 - INFO - sys.run_cmd_with_logger:32 -   1                  -1  1     18560  ultralytics.nn.modules.conv.Conv             [32, 64, 3, 2]
2023-10-24T01:08:29.214555+0000 - INFO - sys.run_cmd_with_logger:32 -   2                  -1  1     29056  ultralytics.nn.modules.block.C2f             [64, 64, 1, True]
2023-10-24T01:08:29.215422+0000 - INFO - sys.run_cmd_with_logger:32 -   3                  -1  1     73984  ultralytics.nn.modules.conv.Conv             [64, 128, 3, 2]
2023-10-24T01:08:29.218726+0000 - INFO - sys.run_cmd_with_logger:32 -   4                  -1  2    197632  ultralytics.nn.modules.block.C2f             [128, 128, 2, True]
2023-10-24T01:08:29.221483+0000 - INFO - sys.run_cmd_with_logger:32 -   5                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
2023-10-24T01:08:29.228793+0000 - INFO - sys.run_cmd_with_logger:32 -   6                  -1  2    788480  ultralytics.nn.modules.block.C2f             [256, 256, 2, True]
2023-10-24T01:08:29.237477+0000 - INFO - sys.run_cmd_with_logger:32 -   7                  -1  1   1180672  ultralytics.nn.modules.conv.Conv             [256, 512, 3, 2]
2023-10-24T01:08:29.251787+0000 - INFO - sys.run_cmd_with_logger:32 -   8                  -1  1   1838080  ultralytics.nn.modules.block.C2f             [512, 512, 1, True]
2023-10-24T01:08:29.258220+0000 - INFO - sys.run_cmd_with_logger:32 -   9                  -1  1    656896  ultralytics.nn.modules.block.SPPF            [512, 512, 5]
2023-10-24T01:08:29.258478+0000 - INFO - sys.run_cmd_with_logger:32 -  10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
2023-10-24T01:08:29.260018+0000 - INFO - sys.run_cmd_with_logger:32 -  11             [-1, 6]  1         0  ultralytics.nn.modules.conv.Concat           [1]
2023-10-24T01:08:29.264290+0000 - INFO - sys.run_cmd_with_logger:32 -  12                  -1  1    591360  ultralytics.nn.modules.block.C2f             [768, 256, 1]
2023-10-24T01:08:29.265057+0000 - INFO - sys.run_cmd_with_logger:32 -  13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
2023-10-24T01:08:29.265462+0000 - INFO - sys.run_cmd_with_logger:32 -  14             [-1, 4]  1         0  ultralytics.nn.modules.conv.Concat           [1]
2023-10-24T01:08:29.266906+0000 - INFO - sys.run_cmd_with_logger:32 -  15                  -1  1    148224  ultralytics.nn.modules.block.C2f             [384, 128, 1]
2023-10-24T01:08:29.268544+0000 - INFO - sys.run_cmd_with_logger:32 -  16                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
2023-10-24T01:08:29.269048+0000 - INFO - sys.run_cmd_with_logger:32 -  17            [-1, 12]  1         0  ultralytics.nn.modules.conv.Concat           [1]
2023-10-24T01:08:29.273959+0000 - INFO - sys.run_cmd_with_logger:32 -  18                  -1  1    493056  ultralytics.nn.modules.block.C2f             [384, 256, 1]
2023-10-24T01:08:29.278476+0000 - INFO - sys.run_cmd_with_logger:32 -  19                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
2023-10-24T01:08:29.279169+0000 - INFO - sys.run_cmd_with_logger:32 -  20             [-1, 9]  1         0  ultralytics.nn.modules.conv.Concat           [1]
2023-10-24T01:08:29.293942+0000 - INFO - sys.run_cmd_with_logger:32 -  21                  -1  1   1969152  ultralytics.nn.modules.block.C2f             [768, 512, 1]
2023-10-24T01:08:29.314642+0000 - INFO - sys.run_cmd_with_logger:32 -  22        [15, 18, 21]  1   2147008  ultralytics.nn.modules.head.Detect           [80, [128, 256, 512]]
2023-10-24T01:08:39.940493+0000 - INFO - sys.run_cmd_with_logger:32 - YOLOv8s summary: 225 layers, 11166560 parameters, 11166544 gradients, 28.7 GFLOPs
2023-10-24T01:08:39.941336+0000 - INFO - sys.run_cmd_with_logger:32 - 
2023-10-24T01:08:40.146653+0000 - INFO - sys.run_cmd_with_logger:32 - New https://pypi.org/project/ultralytics/8.0.200 available 😃 Update with 'pip install -U ultralytics'
2023-10-24T01:08:40.175484+0000 - INFO - sys.run_cmd_with_logger:32 - Ultralytics YOLOv8.0.172 🚀 Python-3.7.16 torch-1.9.1+cu111 CUDA:0 (NVIDIA TITAN V, 12064MiB)
2023-10-24T01:08:40.175983+0000 - INFO - sys.run_cmd_with_logger:32 - WARNING ⚠️ Upgrade to torch>=2.0.0 for deterministic training.
2023-10-24T01:08:40.561716+0000 - INFO - sys.run_cmd_with_logger:32 - engine/trainer: task=detect, mode=train, model=yolov8s.yaml, data=/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/coco.yaml, epochs=0, patience=0, batch=16, imgsz=640, save=True, save_period=-1, cache=False, device=0, workers=16, project=None, name=None, exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, freeze=None, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, show_labels=True, show_conf=True, vid_stride=1, stream_buffer=False, line_width=None, visualize=False, augment=False, agnostic_nms=False, classes=None, retina_masks=False, boxes=True, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=None, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, pose=12.0, kobj=1.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, cfg=None, tracker=botsort.yaml, common_train_epochs=0, root_path=./Xgen/, pretrain_model_weights_path=/root/Projects/.checkpoints/yolov8/yolov8s_xgen.pt, train_data_path=/data/object-detection-yolov6/coco, train_label_path=None, eval_data_path=/data/object-detection-yolov6/coco, eval_label_path=None, learning_rate=0.01, batch_size=16, conf_file=yolov8s.yaml, weights=None, width_multiple=0.5, depth_multiple=0.33, scaling_factor=1, noplots=True, num_classes=80, device_num=1, args=Namespace(agnostic_nms=False, amp=True, augment=False, batch=16, batch_size=16, box=7.5, boxes=True, cache=False, cfg=None, classes=None, close_mosaic=10, cls=0.5, common_train_epochs=0, conf=None, conf_file='yolov8s.yaml', copy_paste=0.0, cos_lr=False, data='/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/coco.yaml', degrees=0.0, depth_multiple=0.33, deterministic=True, device=None, device_num=1, dfl=1.5, dnn=False, dropout=0.0, dynamic=False, epochs=0, eval_data_path='/data/object-detection-yolov6/coco', eval_label_path=None, exist_ok=False, fliplr=0.5, flipud=0.0, format='torchscript', fraction=1.0, freeze=None, half=False, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, imgsz=640, int8=False, iou=0.7, keras=False, kobj=1.0, label_smoothing=0.0, learning_rate=0.01, line_width=None, lr0=0.01, lrf=0.01, mask_ratio=4, max_det=300, mixup=0.0, mode='train', model=None, momentum=0.937, mosaic=1.0, name=None, nbs=64, nms=False, noplots=True, num_classes=80, opset=None, optimize=False, optimizer='auto', overlap_mask=True, patience=0, perspective=0.0, plots=True, pose=12.0, pretrain_model_weights_path='/root/Projects/.checkpoints/yolov8/yolov8s_xgen.pt', pretrained=True, profile=False, project=None, rect=False, resume=False, retina_masks=False, root_path='./Xgen/', save=True, save_conf=False, save_crop=False, save_hybrid=False, save_json=False, save_period=-1, save_txt=False, scale=0.5, scaling_factor=1, seed=0, shear=0.0, show=False, show_conf=True, show_labels=True, simplify=False, single_cls=False, source=None, split='val', stream_buffer=False, task='detect', tracker='botsort.yaml', train_data_path='/data/object-detection-yolov6/coco', train_label_path=None, translate=0.1, val=True, verbose=True, vid_stride=1, visualize=False, warmup_bias_lr=0.1, warmup_epochs=3.0, warmup_momentum=0.8, weight_decay=0.0005, weights=None, width_multiple=0.5, workers=16, workspace=4), args_ai={'origin': {'common_train_epochs': 0, 'root_path': './Xgen/', 'pretrain_model_weights_path': '/root/Projects/.checkpoints/yolov8/yolov8s_xgen.pt', 'train_data_path': '/data/object-detection-yolov6/coco', 'train_label_path': None, 'eval_data_path': '/data/object-detection-yolov6/coco', 'eval_label_path': None, 'learning_rate': 0.01, 'batch_size': 16, 'data': '/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/coco.yaml', 'conf_file': 'yolov8s.yaml', 'weights': None, 'device': None, 'imgsz': 640, 'width_multiple': 0.5, 'depth_multiple': 0.33, 'scaling_factor': 1, 'workers': 16, 'noplots': True, 'num_classes': 80, 'device_num': 1}, 'general': {'user_id': 'test', 'work_place': '/root/output/YOLOv8_CoCo2017/20231024010820', 'random_seed': 3407, 'enable_ddp': False, 'CUDA_VISIBLE_DEVICES': '0', 'tran_scripts_path': None}, 'prune': {'sp_store_weights': None, 'sp_lars': False, 'sp_lars_trust_coef': 0.001, 'sp_backbone': False, 'sp_retrain': False, 'sp_admm': False, 'sp_admm_multi': False, 'sp_retrain_multi': False, 'sp_config_file': None, 'sp_subset_progressive': False, 'sp_admm_fixed_params': False, 'sp_no_harden': False, 'nv_sparse': False, 'sp_load_prune_params': None, 'sp_store_prune_params': None, 'generate_rand_seq_gap_yaml': False, 'sp_admm_update_epoch': 5, 'sp_admm_update_batch': None, 'sp_admm_rho': 0.001, 'sparsity_type': 'block_punched', 'sp_admm_lr': 0.01, 'admm_debug': False, 'sp_global_weight_sparsity': False, 'sp_prune_threshold': -1.0, 'sp_block_irregular_sparsity': '(0,0)', 'sp_block_permute_multiplier': 2, 'sp_admm_block': '(8,4)', 'sp_admm_buckets_num': 16, 'sp_admm_elem_per_row': 1, 'sp_admm_tile': None, 'sp_admm_select_number': 4, 'sp_admm_pattern_row_sub': 1, 'sp_admm_pattern_col_sub': 4, 'sp_admm_data_format': None, 'sp_admm_do_not_permute_conv': False, 'sp_gs_output_v': None, 'sp_gs_output_ptr': None, 'sp_load_frozen_weights': None, 'retrain_mask_pattern': 'weight', 'sp_update_init_method': 'weight', 'sp_mask_update_freq': 10, 'retrain_mask_sparsity': -1.0, 'retrain_mask_seed': None, 'sp_prune_before_retrain': False, 'output_compressed_format': False, 'sp_grad_update': False, 'sp_grad_decay': 0.98, 'sp_grad_restore_threshold': -1, 'sp_global_magnitude': False, 'sp_pre_defined_mask_dir': None, 'sp_prune_ratios': 0}, 'quantization': {'qt_aimet': False, 'qat': True, 'fold_layers': True, 'cross_layer_equalization': False, 'bias_correction': True, 'rounding_mode': 'nearest', 'num_quant_samples': 1000, 'num_bias_correct_samples': 1000, 'weight_bw': 8, 'act_bw': 8, 'quant_scheme': 'tf_enhanced', 'layers_to_ignore': [], 'auto_add_bias': True, 'perform_only_empirical_bias_corr': True}, 'pas': {'pas_ratio': 0, 'pas': False, 'limit_loss_weights': 5.0, 'use_limit_loss': False, 'pas_debug': False, 'pas_rebuild': False, 'pas_finetune_epoch': 200, 'pas_pretrained_weight_path': None, 'pas_ignore': ['neck', 'detect', 'cv'], 'pas_searching_ratio': [0.1, 0.2, 0.3]}, 'task': {'specific_scenarios': 'BasicScaling', 'pretrained_model_path': '/root/Projects/object-detection-yolov8/yolov8_xgen/yolov8_config/xgen.pt', 'state': {'stage': 1, 'cycles': 0}, 'max_searching': 10, 'args_2': {'cycles': 10}, 'args_1': {'cycles': 1}}, 'user_requirements': {'power': None, 'accuracy': 0.35, 'accuracy_reverse_yn': 0, 'model_size': None, 'memory_size': None, 'latency': 75.0, 'margin': 7.5, 'primary_type': 'latency', 'primary_range': '<', 'secondary_type': 'accuracy', 'secondary_range': '>', 'searching_variable': 'scaling_factor', 'searching_range': [0.2, 1], 'searching_step_size': 0.05, 'searching_pas_variable': 'pas', 'express_path': True, 'target_type': 'latency', 'searching_granularity': None, 'using_default_dataset': True, 'user_model': 'YOLOv8', 'using_express_path': True, 'express_mode': 0, 'use_distillation': False, 'use_default_distillation_model': True, 'is_training': True}, 'train': {'common_save_best_yn': 1, 'trained_yn': True, 'larger_better': True, 'uuid': '9ae04efe-469c-42'}, 'compiler': {'input_shape': '(1,3,640,640)', 'opset_version': 11, 'devices': [{'task_queue_size': 0, 'device': {'uuid': 'R5CRC1NFW2E', 'device_type': 'android', 'connection_status': 'available', 'task_status': 'idle', 'info': {'uuid': 'R5CRC1NFW2E', 'cpu': 'SM8350', 'gpu': 'Qualcomm, Adreno (TM) 660', 'memory': '5.24 GB', 'battery': '100', 'brand': 'samsung', 'model': 'SM-G990U1', 'os_type': 'android'}}, 'agent_id': 'agent-localhost'}], 'ios_devices': []}}, save_dir=runs/detect/train
2023-10-24T01:08:40.600204+0000 - INFO - sys.run_cmd_with_logger:32 - 
2023-10-24T01:08:40.600468+0000 - INFO - sys.run_cmd_with_logger:32 -                    from  n    params  module                                       arguments
2023-10-24T01:08:40.601073+0000 - INFO - sys.run_cmd_with_logger:32 -   0                  -1  1       928  ultralytics.nn.modules.conv.Conv             [3, 32, 3, 2]
2023-10-24T01:08:40.601264+0000 - INFO - sys.run_cmd_with_logger:32 -   1                  -1  1     18560  ultralytics.nn.modules.conv.Conv             [32, 64, 3, 2]
2023-10-24T01:08:40.603408+0000 - INFO - sys.run_cmd_with_logger:32 -   2                  -1  1     29056  ultralytics.nn.modules.block.C2f             [64, 64, 1, True]
2023-10-24T01:08:40.604276+0000 - INFO - sys.run_cmd_with_logger:32 -   3                  -1  1     73984  ultralytics.nn.modules.conv.Conv             [64, 128, 3, 2]
2023-10-24T01:08:40.606853+0000 - INFO - sys.run_cmd_with_logger:32 -   4                  -1  2    197632  ultralytics.nn.modules.block.C2f             [128, 128, 2, True]
2023-10-24T01:08:40.608873+0000 - INFO - sys.run_cmd_with_logger:32 -   5                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
2023-10-24T01:08:40.614586+0000 - INFO - sys.run_cmd_with_logger:32 -   6                  -1  2    788480  ultralytics.nn.modules.block.C2f             [256, 256, 2, True]
2023-10-24T01:08:40.623840+0000 - INFO - sys.run_cmd_with_logger:32 -   7                  -1  1   1180672  ultralytics.nn.modules.conv.Conv             [256, 512, 3, 2]
2023-10-24T01:08:40.634369+0000 - INFO - sys.run_cmd_with_logger:32 -   8                  -1  1   1838080  ultralytics.nn.modules.block.C2f             [512, 512, 1, True]
2023-10-24T01:08:40.638723+0000 - INFO - sys.run_cmd_with_logger:32 -   9                  -1  1    656896  ultralytics.nn.modules.block.SPPF            [512, 512, 5]
2023-10-24T01:08:40.639013+0000 - INFO - sys.run_cmd_with_logger:32 -  10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
2023-10-24T01:08:40.639503+0000 - INFO - sys.run_cmd_with_logger:32 -  11             [-1, 6]  1         0  ultralytics.nn.modules.conv.Concat           [1]
2023-10-24T01:08:40.643705+0000 - INFO - sys.run_cmd_with_logger:32 -  12                  -1  1    591360  ultralytics.nn.modules.block.C2f             [768, 256, 1]
2023-10-24T01:08:40.644311+0000 - INFO - sys.run_cmd_with_logger:32 -  13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
2023-10-24T01:08:40.644508+0000 - INFO - sys.run_cmd_with_logger:32 -  14             [-1, 4]  1         0  ultralytics.nn.modules.conv.Concat           [1]
2023-10-24T01:08:40.645914+0000 - INFO - sys.run_cmd_with_logger:32 -  15                  -1  1    148224  ultralytics.nn.modules.block.C2f             [384, 128, 1]
2023-10-24T01:08:40.647037+0000 - INFO - sys.run_cmd_with_logger:32 -  16                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
2023-10-24T01:08:40.647294+0000 - INFO - sys.run_cmd_with_logger:32 -  17            [-1, 12]  1         0  ultralytics.nn.modules.conv.Concat           [1]
2023-10-24T01:08:40.651157+0000 - INFO - sys.run_cmd_with_logger:32 -  18                  -1  1    493056  ultralytics.nn.modules.block.C2f             [384, 256, 1]
2023-10-24T01:08:40.655821+0000 - INFO - sys.run_cmd_with_logger:32 -  19                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
2023-10-24T01:08:40.656210+0000 - INFO - sys.run_cmd_with_logger:32 -  20             [-1, 9]  1         0  ultralytics.nn.modules.conv.Concat           [1]
2023-10-24T01:08:40.671960+0000 - INFO - sys.run_cmd_with_logger:32 -  21                  -1  1   1969152  ultralytics.nn.modules.block.C2f             [768, 512, 1]
2023-10-24T01:08:40.687185+0000 - INFO - sys.run_cmd_with_logger:32 -  22        [15, 18, 21]  1   2147008  ultralytics.nn.modules.head.Detect           [80, [128, 256, 512]]
2023-10-24T01:08:51.878774+0000 - INFO - sys.run_cmd_with_logger:32 - YOLOv8s summary: 225 layers, 11166560 parameters, 11166544 gradients, 28.7 GFLOPs
2023-10-24T01:08:51.879297+0000 - INFO - sys.run_cmd_with_logger:32 - 
2023-10-24T01:08:51.885047+0000 - INFO - sys.run_cmd_with_logger:32 - TensorBoard: Start with 'tensorboard --logdir runs/detect/train', view at http://localhost:6006/
2023-10-24T01:08:55.739030+0000 - INFO - sys.run_cmd_with_logger:32 - Freezing layer 'model.22.dfl.conv.weight'
2023-10-24T01:09:26.625978+0000 - INFO - sys.run_cmd_with_logger:32 - 
2023-10-24T01:09:26.626206+0000 - INFO - sys.run_cmd_with_logger:32 - train: Scanning /data/object-detection-yolov6/coco/labels/train2017.cache... 117266 images, 1021 backgrounds, 0 corrupt: 100%|██████████| 118287/118287 [00:00<?, ?it/s]
2023-10-24T01:09:28.232390+0000 - INFO - sys.run_cmd_with_logger:32 - train: Scanning /data/object-detection-yolov6/coco/labels/train2017.cache... 117266 images, 1021 backgrounds, 0 corrupt: 100%|██████████| 118287/118287 [00:00<?, ?it/s]
2023-10-24T01:09:30.203838+0000 - INFO - sys.run_cmd_with_logger:32 - 
2023-10-24T01:09:30.204406+0000 - INFO - sys.run_cmd_with_logger:32 - val: Scanning /data/object-detection-yolov6/coco/labels/val2017.cache... 4952 images, 48 backgrounds, 0 corrupt: 100%|██████████| 5000/5000 [00:00<?, ?it/s]
2023-10-24T01:09:30.258672+0000 - INFO - sys.run_cmd_with_logger:32 - val: Scanning /data/object-detection-yolov6/coco/labels/val2017.cache... 4952 images, 48 backgrounds, 0 corrupt: 100%|██████████| 5000/5000 [00:00<?, ?it/s]
2023-10-24T01:09:31.119089+0000 - INFO - sys.run_cmd_with_logger:32 - Plotting labels to runs/detect/train/labels.jpg...
2023-10-24T01:09:35.704396+0000 - INFO - sys.run_cmd_with_logger:32 - optimizer: AdamW(lr=0.000119, momentum=0.9) with parameter groups 57 weight(decay=0.0), 64 weight(decay=0.0005), 63 bias(decay=0.0)
2023-10-24T01:09:35.704949+0000 - INFO - sys.run_cmd_with_logger:32 - model is not a DataParallel model
2023-10-24T01:09:35.705215+0000 - INFO - sys.run_cmd_with_logger:32 - can't find file in /root/Projects/.checkpoints/yolov8/yolov8s_xgen.pt
2023-10-24T01:09:35.939356+0000 - INFO - sys.run_cmd_with_logger:32 - Traceback (most recent call last):
2023-10-24T01:09:35.939619+0000 - INFO - sys.run_cmd_with_logger:32 -   File "train_script_main.py", line 25, in <module>
2023-10-24T01:09:35.939834+0000 - INFO - sys.run_cmd_with_logger:32 -     training_main(args_ai=None)
2023-10-24T01:09:35.940261+0000 - INFO - sys.run_cmd_with_logger:32 -   File "train_script_main.py", line 21, in training_main
2023-10-24T01:09:35.940546+0000 - INFO - sys.run_cmd_with_logger:32 -     model.train(data=args.data, batch=args.batch, args=args, args_ai=args_ai, device=args_ai['general']['CUDA_VISIBLE_DEVICES'])
2023-10-24T01:09:35.940805+0000 - INFO - sys.run_cmd_with_logger:32 -   File "/root/output/YOLOv8_CoCo2017/20231024010820/yolov8_xgen/ultralytics/engine/model.py", line 357, in train
2023-10-24T01:09:35.941075+0000 - INFO - sys.run_cmd_with_logger:32 -     self.trainer.train()
2023-10-24T01:09:35.941430+0000 - INFO - sys.run_cmd_with_logger:32 -   File "/root/output/YOLOv8_CoCo2017/20231024010820/yolov8_xgen/ultralytics/engine/trainer.py", line 204, in train
2023-10-24T01:09:35.941700+0000 - INFO - sys.run_cmd_with_logger:32 -     self._do_train(world_size)
2023-10-24T01:09:35.941985+0000 - INFO - sys.run_cmd_with_logger:32 -   File "/root/output/YOLOv8_CoCo2017/20231024010820/yolov8_xgen/ultralytics/engine/trainer.py", line 314, in _do_train
2023-10-24T01:09:35.942273+0000 - INFO - sys.run_cmd_with_logger:32 -     self._setup_train(world_size)
2023-10-24T01:09:35.942505+0000 - INFO - sys.run_cmd_with_logger:32 -   File "/root/output/YOLOv8_CoCo2017/20231024010820/yolov8_xgen/ultralytics/engine/trainer.py", line 301, in _setup_train
2023-10-24T01:09:35.942840+0000 - INFO - sys.run_cmd_with_logger:32 -     xgen_load(self.model, args_ai=self.args_ai)
2023-10-24T01:09:35.943866+0000 - INFO - sys.run_cmd_with_logger:32 -   File "/usr/local/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-1.0.9-py3.7.egg/xgen_tools/helper.py", line 94, in __call__
2023-10-24T01:09:35.944016+0000 - INFO - sys.run_cmd_with_logger:32 -   File "/usr/local/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-1.0.9-py3.7.egg/xgen_tools/xgen_load.py", line 846, in xgen_load
2023-10-24T01:09:35.944200+0000 - INFO - sys.run_cmd_with_logger:32 - FileNotFoundError
Traceback (most recent call last):
  File "xgen_scripts.py", line 109, in main
    training(training_main, training_script_path=training_script_path, log_path=log_path)
  File "/usr/local/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_main-1.2.3-py3.7.egg/xgen/training/core.py", line 449, in training
    internal_data = train_module(job, training_main)
  File "/usr/local/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_main-1.2.3-py3.7.egg/xgen/training/train_module.py", line 184, in train_module
    args_ai = model_train_main(job, training_main)
  File "/usr/local/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_main-1.2.3-py3.7.egg/xgen/training/train_module.py", line 163, in model_train_main
    raise Exception('Training failed')
Exception: Training failed
2023-10-24T01:09:37.585222+0000 - ERROR - xgen_scripts.main:116 - Error found. Please check log file at /root/output/YOLOv8_CoCo2017/20231024010820/xgen-training.log
2023-10-24T01:09:37.585508+0000 - ERROR - xgen_scripts.main:117 - Cancel started session.
xinzhang-cocopie commented 10 months ago

@hsung2 We have fixed this issue. Please try it again.

hsung2 commented 10 months ago

Thanks, it worked.

On Mon, Oct 23, 2023 at 11:27 PM Xin Zhang @.***> wrote:

@hsung2 https://github.com/hsung2 We have fixed this issue. Please try it again.

— Reply to this email directly, view it on GitHub https://github.com/CoCoPIE-Group/XGen-Report/issues/38#issuecomment-1776458132, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYVG2RC654LFVLKJJYHI6V3YA4YQXAVCNFSM6AAAAAA6M7OFFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZWGQ2TQMJTGI . You are receiving this because you were mentioned.Message ID: @.***>

-- -- Hsin-Hsuan Sung