fsx950223
commented
3 years ago Change tf.distribute.MirroredStrategy() to tf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())
Open Ronald-Kray opened 3 years ago
fsx950223
commented
3 years ago Change tf.distribute.MirroredStrategy() to tf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())
Ronald-Kray
commented
3 years ago Change
tf.distribute.MirroredStrategy()totf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())
@fsx950223 Thanks for your reply. I changed the code, but another problem is coming out.
2021-03-31 11:33:51.518201: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
I0331 11:33:55.347332 14548 main.py:264] {'name': 'efficientdet-d0', 'act_type': 'swish', 'image_size': (512, 512), 'target_size': None, 'input_rand_hflip': True, 'jitter_min': 0.1, 'jitter_max': 2.0, 'autoaugment_policy': None, 'grid_mask': False, 'sample_image': None, 'map_freq': 5, 'num_classes': 2, 'seg_num_classes': 3, 'heads': ['object_detection'], 'skip_crowd_during_training': True, 'label_map': {1: 'urethane'}, 'max_instances_per_image': 100, 'regenerate_source_id': False, 'min_level': 3, 'max_level': 7, 'num_scales': 3, 'aspect_ratios': [1.0, 2.0, 0.5], 'anchor_scale': 4.0, 'is_training_bn': True, 'momentum': 0.9, 'optimizer': 'sgd', 'learning_rate': 0.08, 'lr_warmup_init': 0.008, 'lr_warmup_epoch': 1.0, 'first_lr_drop_epoch': 200.0, 'second_lr_drop_epoch': 250.0, 'poly_lr_power': 0.9, 'clip_gradients_norm': 10.0, 'num_epochs': 500, 'data_format': 'channels_last', 'mean_rgb': [123.675, 116.28, 103.53], 'stddev_rgb': [58.395, 57.120000000000005, 57.375], 'label_smoothing': 0.0, 'alpha': 0.25, 'gamma': 1.5, 'delta': 0.1, 'box_loss_weight': 50.0, 'iou_loss_type': None, 'iou_loss_weight': 1.0, 'weight_decay': 4e-05, 'strategy': 'gpus', 'mixed_precision': False, 'loss_scale': None, 'model_optimizations': {}, 'box_class_repeats': 3, 'fpn_cell_repeats': 3, 'fpn_num_filters': 64, 'separable_conv': True, 'apply_bn_for_resampling': True, 'conv_after_downsample': False, 'conv_bn_act_pattern': False, 'drop_remainder': True, 'nms_configs': {'method': 'gaussian', 'iou_thresh': None, 'score_thresh': 0.0, 'sigma': None, 'pyfunc': False, 'max_nms_inputs': 0, 'max_output_size': 100}, 'tflite_max_detections': 100, 'fpn_name': None, 'fpn_weight_method': None, 'fpn_config': None, 'survival_prob': None, 'img_summary_steps': None, 'lr_decay_method': 'cosine', 'moving_average_decay': 0.9998, 'ckpt_var_scope': None, 'skip_mismatch': True, 'backbone_name': 'efficientnet-b0', 'backbone_config': None, 'var_freeze_expr': '(efficientnet|fpn_cells|resample_p6)', 'use_keras_model': True, 'dataset_type': None, 'positives_momentum': None, 'grad_checkpoint': False, 'model_name': 'efficientdet-d0', 'iterations_per_loop': 100, 'model_dir': 'efficientdet/output', 'num_shards': 8, 'num_examples_per_epoch': 120000, 'backbone_ckpt': '', 'ckpt': '', 'val_json_file': None, 'testdev_dir': None, 'profile': True, 'mode': 'train'}
2021-03-31 11:33:55.348499: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-03-31 11:33:55.349727: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-03-31 11:33:55.455415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 11:33:55.455650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 11:33:55.455872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 11:33:55.456089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-03-31 11:33:55.463786: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-03-31 11:33:55.463911: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-03-31 11:33:55.469496: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-03-31 11:33:55.473970: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-03-31 11:33:55.496826: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-03-31 11:33:55.501236: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-03-31 11:33:55.502562: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-03-31 11:33:55.502849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2
2021-03-31 11:33:55.503347: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-03-31 11:33:56.095754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 11:33:56.096017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 11:33:56.096261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 11:33:56.096493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-03-31 11:33:56.096612: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-03-31 11:33:56.096728: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-03-31 11:33:56.096854: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-03-31 11:33:56.096974: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-03-31 11:33:56.097078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-03-31 11:33:56.097194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-03-31 11:33:56.097303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-03-31 11:33:56.097459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2
2021-03-31 11:33:57.693781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-31 11:33:57.693909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2
2021-03-31 11:33:57.693979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N N N
2021-03-31 11:33:57.694048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: N N N
2021-03-31 11:33:57.694117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: N N N
2021-03-31 11:33:57.694432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9417 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5)
2021-03-31 11:33:57.696209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9417 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5)
2021-03-31 11:33:57.697561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9417 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5)
2021-03-31 11:33:57.698785: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
INFO:tensorflow:Using MirroredStrategy with devices ('/replica:0/task:0/device:GPU:0', '/replica:0/task:0/device:GPU:1', '/replica:0/task:0/device:GPU:2')
I0331 11:33:57.708529 14548 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/replica:0/task:0/device:GPU:0', '/replica:0/task:0/device:GPU:1', '/replica:0/task:0/device:GPU:2')
INFO:tensorflow:Initializing RunConfig with distribution strategies.
I0331 11:33:57.849530 14548 run_config.py:584] Initializing RunConfig with distribution strategies.
INFO:tensorflow:Not using Distribute Coordinator.
I0331 11:33:57.849530 14548 estimator_training.py:167] Not using Distribute Coordinator.
INFO:tensorflow:Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}
I0331 11:33:57.849530 14548 estimator.py:191] Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}
INFO:tensorflow:Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}
I0331 11:33:57.850530 14548 estimator.py:191] Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}
INFO:tensorflow:The input_fn accepts an input_context which will be given by DistributionStrategy
I0331 11:33:57.859581 14548 estimator.py:1126] The input_fn accepts an input_context which will be given by DistributionStrategy
I0331 11:33:58.147531 14548 dataloader.py:85] target_size = (512, 512), output_size = (512, 512)
INFO:tensorflow:Calling model_fn.
I0331 11:33:58.988536 15140 api.py:479] Calling model_fn.
I0331 11:33:59.056532 15140 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x000002CD547E17B8>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False)
I0331 11:33:59.404538 15140 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 11:33:59.405542 15140 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 11:33:59.406539 15140 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 11:33:59.407540 15140 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 11:33:59.407540 15140 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 11:33:59.408538 15140 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 11:33:59.409537 15140 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 11:33:59.410539 15140 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0331 11:33:59.411539 15140 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 11:33:59.412538 15140 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 11:33:59.413537 15140 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 11:33:59.414539 15140 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 11:33:59.414539 15140 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 11:33:59.415539 15140 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 11:33:59.416538 15140 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 11:33:59.417538 15140 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0331 11:33:59.418539 15140 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 11:33:59.419539 15140 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 11:33:59.420538 15140 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 11:33:59.421538 15140 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 11:33:59.422538 15140 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 11:33:59.423539 15140 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 11:33:59.424538 15140 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 11:33:59.424538 15140 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
INFO:tensorflow:Calling model_fn.
I0331 11:33:59.552539 9656 api.py:479] Calling model_fn.
I0331 11:33:59.561539 9656 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x000002CD547E17B8>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False)
I0331 11:33:59.875541 9656 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 11:33:59.876541 9656 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 11:33:59.877544 9656 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 11:33:59.878541 9656 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 11:33:59.879543 9656 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 11:33:59.880542 9656 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 11:33:59.881542 9656 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 11:33:59.882541 9656 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0331 11:33:59.883542 9656 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 11:33:59.884542 9656 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 11:33:59.885541 9656 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 11:33:59.886540 9656 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 11:33:59.887542 9656 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 11:33:59.888540 9656 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 11:33:59.888540 9656 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 11:33:59.889540 9656 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0331 11:33:59.891541 9656 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 11:33:59.892540 9656 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 11:33:59.892540 9656 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 11:33:59.893540 9656 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 11:33:59.894541 9656 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 11:33:59.895545 9656 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 11:33:59.896541 9656 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 11:33:59.897541 9656 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
INFO:tensorflow:Calling model_fn.
I0331 11:33:59.978544 6308 api.py:479] Calling model_fn.
I0331 11:33:59.982542 6308 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x000002CD547E17B8>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False)
I0331 11:34:00.264543 6308 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 11:34:00.265544 6308 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 11:34:00.266543 6308 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 11:34:00.266543 6308 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 11:34:00.267543 6308 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 11:34:00.268542 6308 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 11:34:00.269542 6308 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 11:34:00.270542 6308 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0331 11:34:00.271543 6308 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 11:34:00.272545 6308 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 11:34:00.273543 6308 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 11:34:00.274542 6308 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 11:34:00.275545 6308 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 11:34:00.275545 6308 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 11:34:00.276542 6308 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 11:34:00.277542 6308 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0331 11:34:00.279544 6308 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 11:34:00.279544 6308 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 11:34:00.280543 6308 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 11:34:00.281543 6308 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 11:34:00.282544 6308 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 11:34:00.283544 6308 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 11:34:00.284543 6308 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 11:34:00.285542 6308 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0331 11:34:00.360543 14548 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:Error reported to Coordinator: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn *
model_fn_results = self._model_fn(features=features, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn *
params,
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn **
precision, model_fn, features)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision
outputs = mm(ii, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn
cls_out_list, box_out_list = model(inputs, params['is_training_bn'])
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call **
all_feats = self.backbone(inputs, training=training, features_only=True)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call **
outputs = self._stem(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call **
return self._relu_fn(self._bn(self._conv_stem(inputs), training=training))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call **
outputs = super().call(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call
keep_dims=keep_dims)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments
tf.distribute.ReduceOp.MEAN, shard_mean)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce
return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value)))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__
return self._d(self._f, a, k)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated
return _graph_mode_decorator(wrapped, args, kwargs)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator
input_ops=filtered_input_tensors, output_ops=flat_result)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp>
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr>
tf_vars = (get_variable_by_name(var_name) for var_name in var_names)
C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name
raise ValueError("Unsuccessful at finding variable {}.".format(var_name))
ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Traceback (most recent call last): File "C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) File "C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 670, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn *
model_fn_results = self._model_fn(features=features, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn *
params,
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn **
precision, model_fn, features)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision
outputs = mm(ii, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn
cls_out_list, box_out_list = model(inputs, params['is_training_bn'])
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call **
all_feats = self.backbone(inputs, training=training, features_only=True)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call **
outputs = self._stem(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call **
return self._relu_fn(self._bn(self._conv_stem(inputs), training=training))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call **
outputs = super().call(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call
keep_dims=keep_dims)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments
tf.distribute.ReduceOp.MEAN, shard_mean)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce
return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value)))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__
return self._d(self._f, a, k)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated
return _graph_mode_decorator(wrapped, args, kwargs)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator
input_ops=filtered_input_tensors, output_ops=flat_result)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp>
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr>
tf_vars = (get_variable_by_name(var_name) for var_name in var_names)
C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name
raise ValueError("Unsuccessful at finding variable {}.".format(var_name))
ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.I0331 11:34:00.424595 15140 coordinator.py:219] Error reported to Coordinator: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn *
model_fn_results = self._model_fn(features=features, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn *
params,
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn **
precision, model_fn, features)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision
outputs = mm(ii, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn
cls_out_list, box_out_list = model(inputs, params['is_training_bn'])
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call **
all_feats = self.backbone(inputs, training=training, features_only=True)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call **
outputs = self._stem(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call **
return self._relu_fn(self._bn(self._conv_stem(inputs), training=training))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call **
outputs = super().call(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call
keep_dims=keep_dims)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments
tf.distribute.ReduceOp.MEAN, shard_mean)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce
return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value)))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__
return self._d(self._f, a, k)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated
return _graph_mode_decorator(wrapped, args, kwargs)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator
input_ops=filtered_input_tensors, output_ops=flat_result)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp>
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr>
tf_vars = (get_variable_by_name(var_name) for var_name in var_names)
C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name
raise ValueError("Unsuccessful at finding variable {}.".format(var_name))
ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Traceback (most recent call last): File "C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) File "C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 670, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn *
model_fn_results = self._model_fn(features=features, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn *
params,
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn **
precision, model_fn, features)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision
outputs = mm(ii, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn
cls_out_list, box_out_list = model(inputs, params['is_training_bn'])
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call **
all_feats = self.backbone(inputs, training=training, features_only=True)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call **
outputs = self._stem(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call **
return self._relu_fn(self._bn(self._conv_stem(inputs), training=training))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call **
outputs = super().call(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call
keep_dims=keep_dims)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments
tf.distribute.ReduceOp.MEAN, shard_mean)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce
return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value)))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__
return self._d(self._f, a, k)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated
return _graph_mode_decorator(wrapped, args, kwargs)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator
input_ops=filtered_input_tensors, output_ops=flat_result)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp>
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr>
tf_vars = (get_variable_by_name(var_name) for var_name in var_names)
C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name
raise ValueError("Unsuccessful at finding variable {}.".format(var_name))
ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Traceback (most recent call last):
File "C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py", line 402, in
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn *
model_fn_results = self._model_fn(features=features, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn *
params,
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn **
precision, model_fn, features)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision
outputs = mm(ii, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn
cls_out_list, box_out_list = model(inputs, params['is_training_bn'])
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call **
all_feats = self.backbone(inputs, training=training, features_only=True)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call **
outputs = self._stem(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call **
return self._relu_fn(self._bn(self._conv_stem(inputs), training=training))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call **
outputs = super().call(inputs, training)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call
keep_dims=keep_dims)
C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments
tf.distribute.ReduceOp.MEAN, shard_mean)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce
return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value)))
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__
return self._d(self._f, a, k)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated
return _graph_mode_decorator(wrapped, args, kwargs)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator
input_ops=filtered_input_tensors, output_ops=flat_result)
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp>
tf_vars = [v for v in tf_vars if v is not None]
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr>
tf_vars = (get_variable_by_name(var_name) for var_name in var_names)
C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name
raise ValueError("Unsuccessful at finding variable {}.".format(var_name))
ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Process finished with exit code 1
fsx950223
commented
3 years ago Change
tf.distribute.MirroredStrategy()totf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())@fsx950223 Thanks for your reply. I changed the code, but another problem is coming out.
2021-03-31 11:33:51.518201: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll I0331 11:33:55.347332 14548 main.py:264] {'name': 'efficientdet-d0', 'act_type': 'swish', 'image_size': (512, 512), 'target_size': None, 'input_rand_hflip': True, 'jitter_min': 0.1, 'jitter_max': 2.0, 'autoaugment_policy': None, 'grid_mask': False, 'sample_image': None, 'map_freq': 5, 'num_classes': 2, 'seg_num_classes': 3, 'heads': ['object_detection'], 'skip_crowd_during_training': True, 'label_map': {1: 'urethane'}, 'max_instances_per_image': 100, 'regenerate_source_id': False, 'min_level': 3, 'max_level': 7, 'num_scales': 3, 'aspect_ratios': [1.0, 2.0, 0.5], 'anchor_scale': 4.0, 'is_training_bn': True, 'momentum': 0.9, 'optimizer': 'sgd', 'learning_rate': 0.08, 'lr_warmup_init': 0.008, 'lr_warmup_epoch': 1.0, 'first_lr_drop_epoch': 200.0, 'second_lr_drop_epoch': 250.0, 'poly_lr_power': 0.9, 'clip_gradients_norm': 10.0, 'num_epochs': 500, 'data_format': 'channels_last', 'mean_rgb': [123.675, 116.28, 103.53], 'stddev_rgb': [58.395, 57.120000000000005, 57.375], 'label_smoothing': 0.0, 'alpha': 0.25, 'gamma': 1.5, 'delta': 0.1, 'box_loss_weight': 50.0, 'iou_loss_type': None, 'iou_loss_weight': 1.0, 'weight_decay': 4e-05, 'strategy': 'gpus', 'mixed_precision': False, 'loss_scale': None, 'model_optimizations': {}, 'box_class_repeats': 3, 'fpn_cell_repeats': 3, 'fpn_num_filters': 64, 'separable_conv': True, 'apply_bn_for_resampling': True, 'conv_after_downsample': False, 'conv_bn_act_pattern': False, 'drop_remainder': True, 'nms_configs': {'method': 'gaussian', 'iou_thresh': None, 'score_thresh': 0.0, 'sigma': None, 'pyfunc': False, 'max_nms_inputs': 0, 'max_output_size': 100}, 'tflite_max_detections': 100, 'fpn_name': None, 'fpn_weight_method': None, 'fpn_config': None, 'survival_prob': None, 'img_summary_steps': None, 'lr_decay_method': 'cosine', 'moving_average_decay': 0.9998, 'ckpt_var_scope': None, 'skip_mismatch': True, 'backbone_name': 'efficientnet-b0', 'backbone_config': None, 'var_freeze_expr': '(efficientnet|fpn_cells|resample_p6)', 'use_keras_model': True, 'dataset_type': None, 'positives_momentum': None, 'grad_checkpoint': False, 'model_name': 'efficientdet-d0', 'iterations_per_loop': 100, 'model_dir': 'efficientdet/output', 'num_shards': 8, 'num_examples_per_epoch': 120000, 'backbone_ckpt': '', 'ckpt': '', 'val_json_file': None, 'testdev_dir': None, 'profile': True, 'mode': 'train'} 2021-03-31 11:33:55.348499: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-31 11:33:55.349727: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll 2021-03-31 11:33:55.455415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:55.455650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:55.455872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:55.456089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-03-31 11:33:55.463786: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-03-31 11:33:55.463911: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-03-31 11:33:55.469496: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-03-31 11:33:55.473970: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-03-31 11:33:55.496826: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-03-31 11:33:55.501236: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-03-31 11:33:55.502562: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-03-31 11:33:55.502849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2 2021-03-31 11:33:55.503347: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-03-31 11:33:56.095754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:56.096017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:56.096261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:56.096493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-03-31 11:33:56.096612: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-03-31 11:33:56.096728: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-03-31 11:33:56.096854: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-03-31 11:33:56.096974: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-03-31 11:33:56.097078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-03-31 11:33:56.097194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-03-31 11:33:56.097303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-03-31 11:33:56.097459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2 2021-03-31 11:33:57.693781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-31 11:33:57.693909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2 2021-03-31 11:33:57.693979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N N N 2021-03-31 11:33:57.694048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: N N N 2021-03-31 11:33:57.694117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: N N N 2021-03-31 11:33:57.694432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9417 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5) 2021-03-31 11:33:57.696209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9417 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5) 2021-03-31 11:33:57.697561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9417 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5) 2021-03-31 11:33:57.698785: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set INFO:tensorflow:Using MirroredStrategy with devices ('/replica:0/task:0/device:GPU:0', '/replica:0/task:0/device:GPU:1', '/replica:0/task:0/device:GPU:2') I0331 11:33:57.708529 14548 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/replica:0/task:0/device:GPU:0', '/replica:0/task:0/device:GPU:1', '/replica:0/task:0/device:GPU:2') INFO:tensorflow:Initializing RunConfig with distribution strategies. I0331 11:33:57.849530 14548 run_config.py:584] Initializing RunConfig with distribution strategies. INFO:tensorflow:Not using Distribute Coordinator. I0331 11:33:57.849530 14548 estimator_training.py:167] Not using Distribute Coordinator. INFO:tensorflow:Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} I0331 11:33:57.849530 14548 estimator.py:191] Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} INFO:tensorflow:Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} I0331 11:33:57.850530 14548 estimator.py:191] Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} INFO:tensorflow:The
input_fnaccepts aninput_contextwhich will be given by DistributionStrategy I0331 11:33:57.859581 14548 estimator.py:1126] Theinput_fnaccepts aninput_contextwhich will be given by DistributionStrategy I0331 11:33:58.147531 14548 dataloader.py:85] target_size = (512, 512), output_size = (512, 512) INFO:tensorflow:Calling model_fn. I0331 11:33:58.988536 15140 api.py:479] Calling model_fn. I0331 11:33:59.056532 15140 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x000002CD547E17B8>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False) I0331 11:33:59.404538 15140 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.405542 15140 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.406539 15140 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.407540 15140 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.407540 15140 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.408538 15140 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.409537 15140 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.410539 15140 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:33:59.411539 15140 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.412538 15140 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.413537 15140 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.414539 15140 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.414539 15140 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.415539 15140 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.416538 15140 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.417538 15140 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:33:59.418539 15140 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.419539 15140 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.420538 15140 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.421538 15140 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.422538 15140 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.423539 15140 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.424538 15140 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.424538 15140 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} INFO:tensorflow:Calling model_fn. I0331 11:33:59.552539 9656 api.py:479] Calling model_fn. I0331 11:33:59.561539 9656 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x000002CD547E17B8>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False) I0331 11:33:59.875541 9656 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.876541 9656 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.877544 9656 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.878541 9656 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.879543 9656 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.880542 9656 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.881542 9656 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.882541 9656 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:33:59.883542 9656 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.884542 9656 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.885541 9656 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.886540 9656 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.887542 9656 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.888540 9656 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.888540 9656 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.889540 9656 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:33:59.891541 9656 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.892540 9656 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.892540 9656 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.893540 9656 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.894541 9656 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.895545 9656 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.896541 9656 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.897541 9656 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} INFO:tensorflow:Calling model_fn. I0331 11:33:59.978544 6308 api.py:479] Calling model_fn. I0331 11:33:59.982542 6308 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x000002CD547E17B8>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False) I0331 11:34:00.264543 6308 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:34:00.265544 6308 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:34:00.266543 6308 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:34:00.266543 6308 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:34:00.267543 6308 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:34:00.268542 6308 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:34:00.269542 6308 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:34:00.270542 6308 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:34:00.271543 6308 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:34:00.272545 6308 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:34:00.273543 6308 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:34:00.274542 6308 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:34:00.275545 6308 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:34:00.275545 6308 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:34:00.276542 6308 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:34:00.277542 6308 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:34:00.279544 6308 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:34:00.279544 6308 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:34:00.280543 6308 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:34:00.281543 6308 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:34:00.282544 6308 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:34:00.283544 6308 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:34:00.284543 6308 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:34:00.285542 6308 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1 I0331 11:34:00.360543 14548 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1 INFO:tensorflow:Error reported to Coordinator: in user code:C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Traceback (most recent call last): File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 670, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.I0331 11:34:00.424595 15140 coordinator.py:219] Error reported to Coordinator: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Traceback (most recent call last): File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 670, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Traceback (most recent call last): File "C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py", line 402, in app.run(main) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\absl\app.py", line 303, in run _run_main(main, args) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\absl\app.py", line 251, in _run_main sys.exit(main(argv)) File "C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py", line 333, in main train_est.train(input_fn=train_input_fn, max_steps=train_steps) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 349, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1173, in _train_model return self._train_model_distributed(input_fn, hooks, saving_listeners) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1235, in _train_model_distributed self._config._train_distribute, input_fn, hooks, saving_listeners) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1319, in _actual_train_model_distributed self.config)) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 2730, in call_for_each_replica return self._call_for_each_replica(fn, args, kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_strategy.py", line 629, in _call_for_each_replica self._container_strategy(), fn, args, kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 93, in call_for_each_replica return _call_for_each_replica(strategy, fn, args, kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 234, in _call_for_each_replica coord.join(threads) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join six.reraise(self._exc_info_to_raise) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\six.py", line 703, in reraise raise value File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run self.main_result = self.main_fn(self.main_args, **self.main_kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 670, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Process finished with exit code 1
Could you try it with Keras implementation?
Ronald-Kray
commented
3 years ago Change
tf.distribute.MirroredStrategy()totf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())@fsx950223 Thanks for your reply. I changed the code, but another problem is coming out. 2021-03-31 11:33:51.518201: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll I0331 11:33:55.347332 14548 main.py:264] {'name': 'efficientdet-d0', 'act_type': 'swish', 'image_size': (512, 512), 'target_size': None, 'input_rand_hflip': True, 'jitter_min': 0.1, 'jitter_max': 2.0, 'autoaugment_policy': None, 'grid_mask': False, 'sample_image': None, 'map_freq': 5, 'num_classes': 2, 'seg_num_classes': 3, 'heads': ['object_detection'], 'skip_crowd_during_training': True, 'label_map': {1: 'urethane'}, 'max_instances_per_image': 100, 'regenerate_source_id': False, 'min_level': 3, 'max_level': 7, 'num_scales': 3, 'aspect_ratios': [1.0, 2.0, 0.5], 'anchor_scale': 4.0, 'is_training_bn': True, 'momentum': 0.9, 'optimizer': 'sgd', 'learning_rate': 0.08, 'lr_warmup_init': 0.008, 'lr_warmup_epoch': 1.0, 'first_lr_drop_epoch': 200.0, 'second_lr_drop_epoch': 250.0, 'poly_lr_power': 0.9, 'clip_gradients_norm': 10.0, 'num_epochs': 500, 'data_format': 'channels_last', 'mean_rgb': [123.675, 116.28, 103.53], 'stddev_rgb': [58.395, 57.120000000000005, 57.375], 'label_smoothing': 0.0, 'alpha': 0.25, 'gamma': 1.5, 'delta': 0.1, 'box_loss_weight': 50.0, 'iou_loss_type': None, 'iou_loss_weight': 1.0, 'weight_decay': 4e-05, 'strategy': 'gpus', 'mixed_precision': False, 'loss_scale': None, 'model_optimizations': {}, 'box_class_repeats': 3, 'fpn_cell_repeats': 3, 'fpn_num_filters': 64, 'separable_conv': True, 'apply_bn_for_resampling': True, 'conv_after_downsample': False, 'conv_bn_act_pattern': False, 'drop_remainder': True, 'nms_configs': {'method': 'gaussian', 'iou_thresh': None, 'score_thresh': 0.0, 'sigma': None, 'pyfunc': False, 'max_nms_inputs': 0, 'max_output_size': 100}, 'tflite_max_detections': 100, 'fpn_name': None, 'fpn_weight_method': None, 'fpn_config': None, 'survival_prob': None, 'img_summary_steps': None, 'lr_decay_method': 'cosine', 'moving_average_decay': 0.9998, 'ckpt_var_scope': None, 'skip_mismatch': True, 'backbone_name': 'efficientnet-b0', 'backbone_config': None, 'var_freeze_expr': '(efficientnet|fpn_cells|resample_p6)', 'use_keras_model': True, 'dataset_type': None, 'positives_momentum': None, 'grad_checkpoint': False, 'model_name': 'efficientdet-d0', 'iterations_per_loop': 100, 'model_dir': 'efficientdet/output', 'num_shards': 8, 'num_examples_per_epoch': 120000, 'backbone_ckpt': '', 'ckpt': '', 'val_json_file': None, 'testdev_dir': None, 'profile': True, 'mode': 'train'} 2021-03-31 11:33:55.348499: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-31 11:33:55.349727: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll 2021-03-31 11:33:55.455415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:55.455650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:55.455872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:55.456089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-03-31 11:33:55.463786: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-03-31 11:33:55.463911: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-03-31 11:33:55.469496: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-03-31 11:33:55.473970: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-03-31 11:33:55.496826: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-03-31 11:33:55.501236: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-03-31 11:33:55.502562: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-03-31 11:33:55.502849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2 2021-03-31 11:33:55.503347: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-03-31 11:33:56.095754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:56.096017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:56.096261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-31 11:33:56.096493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-03-31 11:33:56.096612: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-03-31 11:33:56.096728: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-03-31 11:33:56.096854: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-03-31 11:33:56.096974: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-03-31 11:33:56.097078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-03-31 11:33:56.097194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-03-31 11:33:56.097303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-03-31 11:33:56.097459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2 2021-03-31 11:33:57.693781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-31 11:33:57.693909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2 2021-03-31 11:33:57.693979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N N N 2021-03-31 11:33:57.694048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: N N N 2021-03-31 11:33:57.694117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: N N N 2021-03-31 11:33:57.694432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9417 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5) 2021-03-31 11:33:57.696209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9417 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5) 2021-03-31 11:33:57.697561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9417 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5) 2021-03-31 11:33:57.698785: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set INFO:tensorflow:Using MirroredStrategy with devices ('/replica:0/task:0/device:GPU:0', '/replica:0/task:0/device:GPU:1', '/replica:0/task:0/device:GPU:2') I0331 11:33:57.708529 14548 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/replica:0/task:0/device:GPU:0', '/replica:0/task:0/device:GPU:1', '/replica:0/task:0/device:GPU:2') INFO:tensorflow:Initializing RunConfig with distribution strategies. I0331 11:33:57.849530 14548 run_config.py:584] Initializing RunConfig with distribution strategies. INFO:tensorflow:Not using Distribute Coordinator. I0331 11:33:57.849530 14548 estimator_training.py:167] Not using Distribute Coordinator. INFO:tensorflow:Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} I0331 11:33:57.849530 14548 estimator.py:191] Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} INFO:tensorflow:Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} I0331 11:33:57.850530 14548 estimator.py:191] Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x000002CD554B6CF8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} INFO:tensorflow:The
input_fnaccepts aninput_contextwhich will be given by DistributionStrategy I0331 11:33:57.859581 14548 estimator.py:1126] Theinput_fnaccepts aninput_contextwhich will be given by DistributionStrategy I0331 11:33:58.147531 14548 dataloader.py:85] target_size = (512, 512), output_size = (512, 512) INFO:tensorflow:Calling model_fn. I0331 11:33:58.988536 15140 api.py:479] Calling model_fn. I0331 11:33:59.056532 15140 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x000002CD547E17B8>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False) I0331 11:33:59.404538 15140 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.405542 15140 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.406539 15140 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.407540 15140 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.407540 15140 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.408538 15140 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.409537 15140 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.410539 15140 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:33:59.411539 15140 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.412538 15140 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.413537 15140 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.414539 15140 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.414539 15140 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.415539 15140 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.416538 15140 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.417538 15140 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:33:59.418539 15140 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.419539 15140 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.420538 15140 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.421538 15140 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.422538 15140 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.423539 15140 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.424538 15140 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.424538 15140 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} INFO:tensorflow:Calling model_fn. I0331 11:33:59.552539 9656 api.py:479] Calling model_fn. I0331 11:33:59.561539 9656 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x000002CD547E17B8>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False) I0331 11:33:59.875541 9656 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.876541 9656 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.877544 9656 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.878541 9656 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.879543 9656 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.880542 9656 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.881542 9656 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.882541 9656 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:33:59.883542 9656 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.884542 9656 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.885541 9656 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.886540 9656 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.887542 9656 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.888540 9656 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.888540 9656 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.889540 9656 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:33:59.891541 9656 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:33:59.892540 9656 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:33:59.892540 9656 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:33:59.893540 9656 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:33:59.894541 9656 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:33:59.895545 9656 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:33:59.896541 9656 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:33:59.897541 9656 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} INFO:tensorflow:Calling model_fn. I0331 11:33:59.978544 6308 api.py:479] Calling model_fn. I0331 11:33:59.982542 6308 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x000002CD547E17B8>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False) I0331 11:34:00.264543 6308 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:34:00.265544 6308 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:34:00.266543 6308 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:34:00.266543 6308 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:34:00.267543 6308 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:34:00.268542 6308 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:34:00.269542 6308 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:34:00.270542 6308 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:34:00.271543 6308 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:34:00.272545 6308 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:34:00.273543 6308 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:34:00.274542 6308 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:34:00.275545 6308 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:34:00.275545 6308 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:34:00.276542 6308 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:34:00.277542 6308 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0331 11:34:00.279544 6308 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0331 11:34:00.279544 6308 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0331 11:34:00.280543 6308 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0331 11:34:00.281543 6308 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0331 11:34:00.282544 6308 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0331 11:34:00.283544 6308 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0331 11:34:00.284543 6308 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0331 11:34:00.285542 6308 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1 I0331 11:34:00.360543 14548 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1 INFO:tensorflow:Error reported to Coordinator: in user code:C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Traceback (most recent call last): File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 670, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.I0331 11:34:00.424595 15140 coordinator.py:219] Error reported to Coordinator: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Traceback (most recent call last): File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 670, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Traceback (most recent call last): File "C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py", line 402, in app.run(main) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\absl\app.py", line 303, in run _run_main(main, args) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\absl\app.py", line 251, in _run_main sys.exit(main(argv)) File "C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py", line 333, in main train_est.train(input_fn=train_input_fn, max_steps=train_steps) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 349, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1173, in _train_model return self._train_model_distributed(input_fn, hooks, saving_listeners) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1235, in _train_model_distributed self._config._train_distribute, input_fn, hooks, saving_listeners) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1319, in _actual_train_model_distributed self.config)) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 2730, in call_for_each_replica return self._call_for_each_replica(fn, args, kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_strategy.py", line 629, in _call_for_each_replica self._container_strategy(), fn, args, kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 93, in call_for_each_replica return _call_for_each_replica(strategy, fn, args, kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 234, in _call_for_each_replica coord.join(threads) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join six.reraise(self._exc_info_to_raise) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\six.py", line 703, in reraise raise value File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception yield File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run self.main_result = self.main_fn(self.main_args, **self.main_kwargs) File "C:\Users\ha485.conda\envs'Official3'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 670, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in user code:
C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py:1163 _call_model_fn * model_fn_results = self._model_fn(features=features, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:616 efficientdet_model_fn * params, C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:344 _model_fn ** precision, model_fn, features) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:631 build_model_with_precision outputs = mm(ii, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\det_model_fn.py:333 model_fn cls_out_list, box_out_list = model(inputs, params['is_training_bn']) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\keras\efficientdet_keras.py:897 call ** all_feats = self.backbone(inputs, training=training, features_only=True) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:734 call ** outputs = self._stem(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\backbone\efficientnet_model.py:528 call ** return self._relu_fn(self._bn(self._conv_stem(inputs), training=training)) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py:786 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:232 call ** outputs = super().call(inputs, training) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\keras\layers\normalization.py:810 call keep_dims=keep_dims) C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:223 _moments tf.distribute.ReduceOp.MEAN, shard_mean) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3089 all_reduce return nest.pack_sequence_as(value, grad_wrapper(*nest.flatten(value))) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:261 __call__ return self._d(self._f, a, k) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:217 decorated return _graph_mode_decorator(wrapped, args, kwargs) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:374 _graph_mode_decorator input_ops=filtered_input_tensors, output_ops=flat_result) C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 _get_dependent_variables tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:308 <listcomp> tf_vars = [v for v in tf_vars if v is not None] C:\Users\ha485\.conda\envs\'Official3'\lib\site-packages\tensorflow\python\ops\custom_gradient.py:307 <genexpr> tf_vars = (get_variable_by_name(var_name) for var_name in var_names) C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py:45 get_variable_by_name raise ValueError("Unsuccessful at finding variable {}.".format(var_name)) ValueError: Unsuccessful at finding variable efficientnet-b0/stem/conv2d/kernel/replica_1.Process finished with exit code 1
Could you try it with Keras implementation?
This is the log I tried with Keras train.py:( ImportError: cannot import name 'get_config'
2021-03-31 14:38:28.244179: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Traceback (most recent call last):
File "C:/Users/ha485/PycharmProjects/Official3/efficientdet/keras/train.py", line 22, in
fsx950223
commented
3 years ago What's your command and tensorflow version
Ronald-Kray
commented
3 years ago What's your command and TensorFlow version
I just ran below conditions. Tensorflow=2.4.0
def define_flags():
"""Define the flags."""
# Cloud TPU Cluster Resolvers
flags.DEFINE_string(
'tpu',
default=None,
help='The Cloud TPU to use for training. This should be either the name '
'used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 '
'url.')
flags.DEFINE_string(
'gcp_project',
default=None,
help='Project name for the Cloud TPU-enabled project. If not specified, '
'we will attempt to automatically detect the GCE project from metadata.')
flags.DEFINE_string(
'tpu_zone',
default=None,
help='GCE zone where the Cloud TPU is located in. If not specified, we '
'will attempt to automatically detect the GCE project from metadata.')
# Model specific paramenters
flags.DEFINE_string( 'eval_master', default='', help='GRPC URL of the eval master. Set to an appropriate value when ' 'running on CPU/GPU') flags.DEFINE_string('eval_name', default=None, help='Eval job name')
flags.DEFINE_enum('strategy', 'gpus', ['tpu', 'gpus', ''],
'Training: gpus for multi-gpu, if None, use TF default.')
flags.DEFINE_integer(
'num_cores', default=8, help='Number of TPU cores for training')
flags.DEFINE_bool('use_fake_data', False, 'Use fake input.')
flags.DEFINE_bool(
'use_xla', False,
'Use XLA even if strategy is not tpu. If strategy is tpu, always use XLA,'
' and this flag has no effect.')
flags.DEFINE_string('model_dir', 'efficientdet/output', 'Location of model_dir')
flags.DEFINE_string('pretrained_ckpt', None,
'Start training from this EfficientDet checkpoint.')
flags.DEFINE_string(
'hparams', 'C:\Users\ha485\PycharmProjects\Official3\efficientdet\configs\config.yaml', 'Comma separated k=v pairs of hyperparameters or a module'
' containing attributes to use as hyperparameters.')
flags.DEFINE_integer('batch_size', 48, 'training batch size')
flags.DEFINE_integer('eval_samples', 5000, 'The number of samples for '
'evaluation.')
flags.DEFINE_integer('steps_per_execution', 1,
'Number of steps per training execution.')
flags.DEFINE_string(
'train_file_pattern', 'efficientdet/tfrecord/train/.tfrecord',
'Glob for train data files (e.g., COCO train - minival set)')
flags.DEFINE_string('val_file_pattern', 'efficientdet/tfrecord/val/.tfrecord',
'Glob for evaluation tfrecords (e.g., COCO val2017 set)')
flags.DEFINE_string(
'val_json_file', None,
'COCO validation JSON containing golden bounding boxes. If None, use the '
'ground truth from the dataloader. Ignored if testdev_dir is not None.')
flags.DEFINE_enum('mode', 'traineval', ['train', 'traineval'],
'job mode: train, traineval.')
flags.DEFINE_string(
'hub_module_url', None, 'TF-Hub path/url to EfficientDet module.'
'If specified, pretrained_ckpt flag should not be used.')
flags.DEFINE_integer('num_examples_per_epoch', 120000,
'Number of examples in one epoch')
flags.DEFINE_integer('num_epochs', None, 'Number of epochs for training')
flags.DEFINE_string('model_name', 'efficientdet-d0', 'Model name.')
flags.DEFINE_bool('debug', False, 'Enable debug mode')
flags.DEFINE_integer(
'tf_random_seed', 111111,
'Fixed random seed for deterministic execution across runs for debugging.'
)
flags.DEFINE_bool('profile', False, 'Enable profile mode')
fsx950223
commented
3 years ago Add PYTHONPATH=./ in front of command or uninstall keras
Ronald-Kray
commented
3 years ago Add
PYTHONPATH=./in front of command or uninstall keras
Could you give me an example?
fsx950223
commented
3 years ago Add
PYTHONPATH=./in front of command or uninstall kerasCould you give me an example?
pip uninstall keras.
Ronald-Kray
commented
3 years ago Add
PYTHONPATH=./in front of command or uninstall kerasCould you give me an example?
pip uninstall keras. it seemt not to be installed
fsx950223
commented
3 years ago Add
PYTHONPATH=./in front of command or uninstall kerasCould you give me an example?
pip uninstall keras. it seemt not to be installed
pip uninstall Keras
Ronald-Kray
commented
3 years ago pip uninstall Keras
It is the same result. Should I reinstall the whole env?
fsx950223
commented
3 years ago Remove C:\Users\ha485.conda\envs'Official3'\lib\site-packages\keras folder
Ronald-Kray
commented
3 years ago Remove
C:\Users\ha485.conda\envs'Official3'\lib\site-packages\kerasfolder
is it right?
In main.py
2021-03-31 15:27:04.838966: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Traceback (most recent call last):
File "C:/Users/ha485/PycharmProjects/Official3/efficientdet/main.py", line 61, in
In train.py
2021-03-31 15:28:30.466713: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Traceback (most recent call last):
File "C:/Users/ha485/PycharmProjects/Official3/efficientdet/keras/train.py", line 22, in
fsx950223
commented
3 years ago Oh, revert the folder and set environment variable PYTHONPATH=./ before import keras.
E.g:
PYTHONPATH=./ python keras/train.py
--train_file_pattern=tfrecord/pascal*.tfrecord \
--val_file_pattern=tfrecord/pascal*.tfrecord \
--val_file_pattern=tfrecord/*.json \
--model_name=efficientdet-d0 \
--model_dir=/tmp/efficientdet-d0-finetune \
--pretrained_ckpt=efficientdet-d0 \
--batch_size=64 \
--eval_samples=1024 \
--num_examples_per_epoch=5717 --num_epochs=50 \
--hparams=voc_config.yamlPYTHONPATH=./ python keras/train.py --train_file_pattern=tfrecord/pascal.tfrecord \ --val_file_pattern=tfrecord/pascal.tfrecord \ --val_file_pattern=tfrecord/*.json \ --model_name=efficientdet-d0 \ --model_dir=/tmp/efficientdet-d0-finetune \ --pretrained_ckpt=efficientdet-d0 \ --batch_size=64 \ --eval_samples=1024 \ --num_examples_per_epoch=5717 --num_epochs=50 \ --hparams=voc_config.yaml
Here is my command. I have got another error.
PYTHONPATH=./python efficientdet/keras/train.py --train_file_pattern=tfrecord/train/.tfrecord --val_file_pattern=tfrecord/val/dangerous-materials.tfrecord --val_file_pattern=efficientdet/tfrecord/*.json --model_name=efficientdet-d0 --model_dir=efficientdet/output --batch_size=64 --eval_samples=1024 --num_examples_per_epoch=5717 --num_epochs=50 --hparams=configs/config.yaml
('Official3') C:\Users\ha485\PycharmProjects\Official3\efficientdet>PYTHONPATH=./python efficientdet/keras/train.py --train_file_pattern=tfrecord/train/.tfrecord --val_file_pattern=tfrecord/val/dangerous- materials.tfrecord --val_file_pattern=efficientdet/tfrecord/*.json --model_name=efficientdet-d0 --model_dir=efficientdet/output --batch_size=64 --eval_samples=1024 --num_examples_per_epoch=5717 --num_epo chs=50 --hparams=configs/config.yaml 'PYTHONPATH' is not recognized as an internal or external command, operable program or batch file.
fsx950223
commented
3 years ago Why PYTHONPATH=./python? Could you follow my example?
Ronald-Kray
commented
3 years ago Why
PYTHONPATH=./python? Could you follow my example?
Still the same error..
('Official3') C:\Users\ha485\PycharmProjects\Official3\efficientdet>PYTHONPATH=./ python efficientdet/keras/train.py --train_file_pattern=tfrecord/train/.tfrecord --val_file_pattern=tfrecord/val/dangerous -materials.tfrecord --val_file_pattern=efficientdet/tfrecord/*.json --model_name=efficientdet-d0 --model_dir=efficientdet/output --batch_size=64 --eval_samples=1024 --num_examples_per_epoch=5717 --num_ep ochs=50 --hparams=configs/config.yaml 'PYTHONPATH' is not recognized as an internal or external command, operable program or batch file.
juliangrosshauser
commented
3 years ago Here's how you can set the PYTHONPATH on Windows: https://docs.microsoft.com/en-us/windows/python/faqs#what-is-pythonpath-
Ronald-Kray
commented
3 years ago @juliangrosshauser @fsx950223 Thanks for your reply. I've reinstalled the whole env to fix the problem. (Win 10, Tensorflow 2.4.1, Keras trainning) I've tried 3 approaches. 1. just run, 2. use the command, 3. use PYTHONPATH=./
1. Just run
def define_flags():
"""Define the flags."""
# Cloud TPU Cluster Resolvers
flags.DEFINE_string(
'tpu',
default=None,
help='The Cloud TPU to use for training. This should be either the name '
'used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 '
'url.')
flags.DEFINE_string(
'gcp_project',
default=None,
help='Project name for the Cloud TPU-enabled project. If not specified, '
'we will attempt to automatically detect the GCE project from metadata.')
flags.DEFINE_string(
'tpu_zone',
default=None,
help='GCE zone where the Cloud TPU is located in. If not specified, we '
'will attempt to automatically detect the GCE project from metadata.')
# Model specific paramenters
flags.DEFINE_string(
'eval_master',
default='',
help='GRPC URL of the eval master. Set to an appropriate value when '
'running on CPU/GPU')
flags.DEFINE_string('eval_name', default=None, help='Eval job name')
flags.DEFINE_enum('strategy', 'gpus', ['tpu', 'gpus', ''],
'Training: gpus for multi-gpu, if None, use TF default.')
flags.DEFINE_integer(
'num_cores', default=8, help='Number of TPU cores for training')
flags.DEFINE_bool('use_fake_data', False, 'Use fake input.')
flags.DEFINE_bool(
'use_xla', False,
'Use XLA even if strategy is not tpu. If strategy is tpu, always use XLA,'
' and this flag has no effect.')
flags.DEFINE_string('model_dir', 'efficientdet/ckpt/efficientdet-d0', 'Location of model_dir')
flags.DEFINE_string('pretrained_ckpt', None,
'Start training from this EfficientDet checkpoint.')
flags.DEFINE_string(
'hparams', 'C:\Users\ha485\PycharmProjects\Official4\efficientdet\configs\config.yaml', 'Comma separated k=v pairs of hyperparameters or a module'
' containing attributes to use as hyperparameters.')
flags.DEFINE_integer('batch_size', 64, 'training batch size')
flags.DEFINE_integer('eval_samples', 5000, 'The number of samples for '
'evaluation.')
flags.DEFINE_integer('steps_per_execution', 1,
'Number of steps per training execution.')
flags.DEFINE_string(
'train_file_pattern', 'C:\Users\ha485\PycharmProjects\Official4\efficientdet\dataset\tfrecord\train.tfrecord',
'Glob for train data files (e.g., COCO train - minival set)')
flags.DEFINE_string('val_file_pattern', 'C:\Users\ha485\PycharmProjects\Official4\efficientdet\dataset\tfrecord\val.tfrecord',
'Glob for evaluation tfrecords (e.g., COCO val2017 set)')
flags.DEFINE_string(
'val_json_file', 'C:\Users\ha485\PycharmProjects\Official4\efficientdet\dataset\tfrecord\_annotations.coco.json',
'COCO validation JSON containing golden bounding boxes. If None, use the '
'ground truth from the dataloader. Ignored if testdev_dir is not None.')
flags.DEFINE_enum('mode', 'train', ['train', 'traineval'],
'job mode: train, traineval.')
flags.DEFINE_string(
'hub_module_url', None, 'TF-Hub path/url to EfficientDet module.'
'If specified, pretrained_ckpt flag should not be used.')
flags.DEFINE_integer('num_examples_per_epoch', 120000,
'Number of examples in one epoch')
flags.DEFINE_integer('num_epochs', 100, 'Number of epochs for training')
flags.DEFINE_string('model_name', 'efficientdet-d0', 'Model name.')
flags.DEFINE_bool('debug', False, 'Enable debug mode')
flags.DEFINE_integer(
'tf_random_seed', 111111,
'Fixed random seed for deterministic execution across runs for debugging.'
)
flags.DEFINE_bool('profile', False, 'Enable profile mode')
1. Just run(Log)
2021-03-31 21:45:51.732639: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-03-31 21:45:56.100830: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-03-31 21:45:56.101850: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-03-31 21:45:56.224416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 21:45:56.224722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 21:45:56.225021: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 21:45:56.225302: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-03-31 21:45:56.233050: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-03-31 21:45:56.233172: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-03-31 21:45:56.237597: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-03-31 21:45:56.239292: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-03-31 21:45:56.248584: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-03-31 21:45:56.251838: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-03-31 21:45:56.253259: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-03-31 21:45:56.253850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2
2021-03-31 21:45:56.254872: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-03-31 21:45:56.793195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 21:45:56.793893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 21:45:56.794567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-03-31 21:45:56.795208: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-03-31 21:45:56.795535: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-03-31 21:45:56.795866: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-03-31 21:45:56.796194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-03-31 21:45:56.796510: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-03-31 21:45:56.796831: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-03-31 21:45:56.797152: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-03-31 21:45:56.797475: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-03-31 21:45:56.797954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2
2021-03-31 21:45:58.480610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-31 21:45:58.480741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2
2021-03-31 21:45:58.480815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N N N
2021-03-31 21:45:58.480885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: N N N
2021-03-31 21:45:58.480955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: N N N
2021-03-31 21:45:58.481264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9417 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5)
2021-03-31 21:45:58.482926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9417 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5)
2021-03-31 21:45:58.484407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9417 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5)
2021-03-31 21:45:58.485654: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0331 21:45:58.489144 20144 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0331 21:45:58.491145 20144 train.py:182] All devices: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU')]
I0331 21:45:59.623721 20144 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x0000020F227B3400>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False)
I0331 21:45:59.919737 20144 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 21:45:59.920737 20144 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 21:45:59.921738 20144 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 21:45:59.922737 20144 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 21:45:59.922737 20144 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 21:45:59.923737 20144 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 21:45:59.924737 20144 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 21:45:59.925737 20144 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0331 21:45:59.926736 20144 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 21:45:59.927736 20144 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 21:45:59.927736 20144 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 21:45:59.928739 20144 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 21:45:59.929737 20144 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 21:45:59.930739 20144 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 21:45:59.931739 20144 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 21:45:59.932738 20144 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0331 21:45:59.933737 20144 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0331 21:45:59.934738 20144 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0331 21:45:59.935739 20144 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0331 21:45:59.935739 20144 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0331 21:45:59.936738 20144 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0331 21:45:59.937737 20144 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0331 21:45:59.938739 20144 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0331 21:45:59.939739 20144 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:00.642779 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:00.644779 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:00.649782 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:00.651780 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:00.718782 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:00.728784 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:1402: UserWarning: layer.updates will be removed in a future version. This property should not be used in TensorFlow 2.0, as updates are applied automatically.
warnings.warn('layer.updates will be removed in a future version. '
I0331 21:46:02.137863 20144 api.py:461] Built stem stem : (None, 256, 256, 32)
I0331 21:46:04.146338 20144 api.py:461] Block blocks_0 input shape: (None, 256, 256, 32)
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:04.180340 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:04.182337 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:04.186338 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:04.188339 20144 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0331 21:46:04.212341 20144 api.py:461] DWConv shape: (None, 256, 256, 32)
I0331 21:46:04.342348 20144 api.py:461] Built SE se : (None, 1, 1, 32)
I0331 21:46:04.392351 20144 api.py:461] Project shape: (None, 256, 256, 16)
I0331 21:46:04.436354 20144 api.py:461] Block blocks_1 input shape: (None, 256, 256, 16)
I0331 21:46:04.487356 20144 api.py:461] Expand shape: (None, 256, 256, 96)
I0331 21:46:04.537359 20144 api.py:461] DWConv shape: (None, 128, 128, 96)
I0331 21:46:04.573361 20144 api.py:461] Built SE se : (None, 1, 1, 96)
I0331 21:46:04.622365 20144 api.py:461] Project shape: (None, 128, 128, 24)
I0331 21:46:04.634365 20144 api.py:461] Block blocks_2 input shape: (None, 128, 128, 24)
I0331 21:46:04.686368 20144 api.py:461] Expand shape: (None, 128, 128, 144)
I0331 21:46:04.738371 20144 api.py:461] DWConv shape: (None, 128, 128, 144)
I0331 21:46:04.773373 20144 api.py:461] Built SE se : (None, 1, 1, 144)
I0331 21:46:04.823375 20144 api.py:461] Project shape: (None, 128, 128, 24)
I0331 21:46:04.836376 20144 api.py:461] Block blocks_3 input shape: (None, 128, 128, 24)
I0331 21:46:04.886379 20144 api.py:461] Expand shape: (None, 128, 128, 144)
I0331 21:46:04.941382 20144 api.py:461] DWConv shape: (None, 64, 64, 144)
I0331 21:46:04.976384 20144 api.py:461] Built SE se : (None, 1, 1, 144)
I0331 21:46:05.025387 20144 api.py:461] Project shape: (None, 64, 64, 40)
I0331 21:46:05.038388 20144 api.py:461] Block blocks_4 input shape: (None, 64, 64, 40)
I0331 21:46:05.089391 20144 api.py:461] Expand shape: (None, 64, 64, 240)
I0331 21:46:05.144395 20144 api.py:461] DWConv shape: (None, 64, 64, 240)
I0331 21:46:05.192397 20144 api.py:461] Built SE se : (None, 1, 1, 240)
I0331 21:46:05.254400 20144 api.py:461] Project shape: (None, 64, 64, 40)
I0331 21:46:05.267401 20144 api.py:461] Block blocks_5 input shape: (None, 64, 64, 40)
I0331 21:46:05.318404 20144 api.py:461] Expand shape: (None, 64, 64, 240)
I0331 21:46:05.369407 20144 api.py:461] DWConv shape: (None, 32, 32, 240)
I0331 21:46:05.404408 20144 api.py:461] Built SE se : (None, 1, 1, 240)
I0331 21:46:05.454412 20144 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:05.466412 20144 api.py:461] Block blocks_6 input shape: (None, 32, 32, 80)
I0331 21:46:05.516415 20144 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:05.568418 20144 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:05.605420 20144 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:05.654423 20144 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:05.667424 20144 api.py:461] Block blocks_7 input shape: (None, 32, 32, 80)
I0331 21:46:05.739426 20144 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:05.801431 20144 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:05.837433 20144 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:05.900437 20144 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:05.913438 20144 api.py:461] Block blocks_8 input shape: (None, 32, 32, 80)
I0331 21:46:05.962440 20144 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:06.013443 20144 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:06.048445 20144 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:06.098448 20144 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:06.112450 20144 api.py:461] Block blocks_9 input shape: (None, 32, 32, 112)
I0331 21:46:06.171452 20144 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:06.243454 20144 api.py:461] DWConv shape: (None, 32, 32, 672)
I0331 21:46:06.289459 20144 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:06.338463 20144 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:06.351463 20144 api.py:461] Block blocks_10 input shape: (None, 32, 32, 112)
I0331 21:46:06.402465 20144 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:06.454469 20144 api.py:461] DWConv shape: (None, 32, 32, 672)
I0331 21:46:06.489470 20144 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:06.538473 20144 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:06.551474 20144 api.py:461] Block blocks_11 input shape: (None, 32, 32, 112)
I0331 21:46:06.605477 20144 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:06.655480 20144 api.py:461] DWConv shape: (None, 16, 16, 672)
I0331 21:46:06.690482 20144 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:06.740485 20144 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:06.753486 20144 api.py:461] Block blocks_12 input shape: (None, 16, 16, 192)
I0331 21:46:06.831487 20144 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:06.901540 20144 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:06.938543 20144 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:06.987544 20144 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:06.999545 20144 api.py:461] Block blocks_13 input shape: (None, 16, 16, 192)
I0331 21:46:07.049548 20144 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:07.104552 20144 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:07.143557 20144 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:07.198559 20144 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:07.211558 20144 api.py:461] Block blocks_14 input shape: (None, 16, 16, 192)
I0331 21:46:07.265559 20144 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:07.318562 20144 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:07.354565 20144 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:07.404567 20144 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:07.416569 20144 api.py:461] Block blocks_15 input shape: (None, 16, 16, 192)
I0331 21:46:07.467572 20144 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:07.526574 20144 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:07.562576 20144 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:07.613579 20144 api.py:461] Project shape: (None, 16, 16, 320)
I0331 21:46:13.015685 20144 train_lib.py:218] LR schedule method: cosine
I0331 21:46:13.015685 20144 train_lib.py:296] Use SGD optimizer
I0331 21:46:13.431708 20144 dataloader.py:85] target_size = (512, 512), output_size = (512, 512)
2021-03-31 21:46:14.501822: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/100
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:15.131805 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:15.192819 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:15.266815 18380 api.py:461] Built stem stem : (None, 256, 256, 32)
I0331 21:46:15.278813 18380 api.py:461] Block blocks_0 input shape: (None, 256, 256, 32)
I0331 21:46:15.327817 15064 api.py:461] Built stem stem : (None, 256, 256, 32)
I0331 21:46:15.339817 15064 api.py:461] Block blocks_0 input shape: (None, 256, 256, 32)
I0331 21:46:15.412822 8152 api.py:461] Built stem stem : (None, 256, 256, 32)
I0331 21:46:15.424822 8152 api.py:461] Block blocks_0 input shape: (None, 256, 256, 32)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:15.447824 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:15.509827 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:15.581832 18380 api.py:461] DWConv shape: (None, 256, 256, 32)
I0331 21:46:15.604832 18380 api.py:461] Built SE se : (None, 1, 1, 32)
I0331 21:46:15.654835 15064 api.py:461] DWConv shape: (None, 256, 256, 32)
I0331 21:46:15.676836 15064 api.py:461] Built SE se : (None, 1, 1, 32)
I0331 21:46:15.746840 8152 api.py:461] DWConv shape: (None, 256, 256, 32)
I0331 21:46:15.769842 8152 api.py:461] Built SE se : (None, 1, 1, 32)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:15.792843 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:15.852846 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:15.907850 18380 api.py:461] Project shape: (None, 256, 256, 16)
I0331 21:46:15.920850 18380 api.py:461] Block blocks_1 input shape: (None, 256, 256, 16)
I0331 21:46:15.987854 15064 api.py:461] Project shape: (None, 256, 256, 16)
I0331 21:46:15.999855 15064 api.py:461] Block blocks_1 input shape: (None, 256, 256, 16)
I0331 21:46:16.066865 8152 api.py:461] Project shape: (None, 256, 256, 16)
I0331 21:46:16.079859 8152 api.py:461] Block blocks_1 input shape: (None, 256, 256, 16)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:16.101861 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:16.162054 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:16.237078 18380 api.py:461] Expand shape: (None, 256, 256, 96)
I0331 21:46:16.310087 15064 api.py:461] Expand shape: (None, 256, 256, 96)
I0331 21:46:16.380091 8152 api.py:461] Expand shape: (None, 256, 256, 96)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:16.404093 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:16.467089 20144 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1
I0331 21:46:16.542095 18380 api.py:461] DWConv shape: (None, 128, 128, 96)
I0331 21:46:16.566103 18380 api.py:461] Built SE se : (None, 1, 1, 96)
I0331 21:46:16.638105 15064 api.py:461] DWConv shape: (None, 128, 128, 96)
I0331 21:46:16.659107 15064 api.py:461] Built SE se : (None, 1, 1, 96)
I0331 21:46:16.742113 8152 api.py:461] DWConv shape: (None, 128, 128, 96)
I0331 21:46:16.763111 8152 api.py:461] Built SE se : (None, 1, 1, 96)
I0331 21:46:16.922117 18380 api.py:461] Project shape: (None, 128, 128, 24)
I0331 21:46:16.935119 18380 api.py:461] Block blocks_2 input shape: (None, 128, 128, 24)
I0331 21:46:17.001127 15064 api.py:461] Project shape: (None, 128, 128, 24)
I0331 21:46:17.015126 15064 api.py:461] Block blocks_2 input shape: (None, 128, 128, 24)
I0331 21:46:17.082446 8152 api.py:461] Project shape: (None, 128, 128, 24)
I0331 21:46:17.094446 8152 api.py:461] Block blocks_2 input shape: (None, 128, 128, 24)
I0331 21:46:17.250455 18380 api.py:461] Expand shape: (None, 128, 128, 144)
I0331 21:46:17.322458 15064 api.py:461] Expand shape: (None, 128, 128, 144)
I0331 21:46:17.391463 8152 api.py:461] Expand shape: (None, 128, 128, 144)
I0331 21:46:17.548427 18380 api.py:461] DWConv shape: (None, 128, 128, 144)
I0331 21:46:17.570428 18380 api.py:461] Built SE se : (None, 1, 1, 144)
I0331 21:46:17.645434 15064 api.py:461] DWConv shape: (None, 128, 128, 144)
I0331 21:46:17.666434 15064 api.py:461] Built SE se : (None, 1, 1, 144)
I0331 21:46:17.737440 8152 api.py:461] DWConv shape: (None, 128, 128, 144)
I0331 21:46:17.759440 8152 api.py:461] Built SE se : (None, 1, 1, 144)
I0331 21:46:17.936445 18380 api.py:461] Project shape: (None, 128, 128, 24)
I0331 21:46:17.948447 18380 api.py:461] Block blocks_3 input shape: (None, 128, 128, 24)
I0331 21:46:18.121460 15064 api.py:461] Project shape: (None, 128, 128, 24)
I0331 21:46:18.133460 15064 api.py:461] Block blocks_3 input shape: (None, 128, 128, 24)
I0331 21:46:18.180468 8152 api.py:461] Project shape: (None, 128, 128, 24)
I0331 21:46:18.193465 8152 api.py:461] Block blocks_3 input shape: (None, 128, 128, 24)
I0331 21:46:18.350471 18380 api.py:461] Expand shape: (None, 128, 128, 144)
I0331 21:46:18.399474 15064 api.py:461] Expand shape: (None, 128, 128, 144)
I0331 21:46:18.481479 8152 api.py:461] Expand shape: (None, 128, 128, 144)
I0331 21:46:18.641443 18380 api.py:461] DWConv shape: (None, 64, 64, 144)
I0331 21:46:18.663444 18380 api.py:461] Built SE se : (None, 1, 1, 144)
I0331 21:46:18.713447 15064 api.py:461] DWConv shape: (None, 64, 64, 144)
I0331 21:46:18.734448 15064 api.py:461] Built SE se : (None, 1, 1, 144)
I0331 21:46:18.805452 8152 api.py:461] DWConv shape: (None, 64, 64, 144)
I0331 21:46:18.827455 8152 api.py:461] Built SE se : (None, 1, 1, 144)
I0331 21:46:18.982464 18380 api.py:461] Project shape: (None, 64, 64, 40)
I0331 21:46:18.997463 18380 api.py:461] Block blocks_4 input shape: (None, 64, 64, 40)
I0331 21:46:19.044466 15064 api.py:461] Project shape: (None, 64, 64, 40)
I0331 21:46:19.056467 15064 api.py:461] Block blocks_4 input shape: (None, 64, 64, 40)
I0331 21:46:19.124469 8152 api.py:461] Project shape: (None, 64, 64, 40)
I0331 21:46:19.136473 8152 api.py:461] Block blocks_4 input shape: (None, 64, 64, 40)
I0331 21:46:19.287878 18380 api.py:461] Expand shape: (None, 64, 64, 240)
I0331 21:46:19.354880 15064 api.py:461] Expand shape: (None, 64, 64, 240)
I0331 21:46:19.425889 8152 api.py:461] Expand shape: (None, 64, 64, 240)
I0331 21:46:19.583894 18380 api.py:461] DWConv shape: (None, 64, 64, 240)
I0331 21:46:19.606899 18380 api.py:461] Built SE se : (None, 1, 1, 240)
I0331 21:46:19.654902 15064 api.py:461] DWConv shape: (None, 64, 64, 240)
I0331 21:46:19.675905 15064 api.py:461] Built SE se : (None, 1, 1, 240)
I0331 21:46:19.745908 8152 api.py:461] DWConv shape: (None, 64, 64, 240)
I0331 21:46:19.766909 8152 api.py:461] Built SE se : (None, 1, 1, 240)
I0331 21:46:19.923930 18380 api.py:461] Project shape: (None, 64, 64, 40)
I0331 21:46:19.936929 18380 api.py:461] Block blocks_5 input shape: (None, 64, 64, 40)
I0331 21:46:19.983934 15064 api.py:461] Project shape: (None, 64, 64, 40)
I0331 21:46:19.995934 15064 api.py:461] Block blocks_5 input shape: (None, 64, 64, 40)
I0331 21:46:20.059941 8152 api.py:461] Project shape: (None, 64, 64, 40)
I0331 21:46:20.071943 8152 api.py:461] Block blocks_5 input shape: (None, 64, 64, 40)
I0331 21:46:20.229951 18380 api.py:461] Expand shape: (None, 64, 64, 240)
I0331 21:46:20.301951 15064 api.py:461] Expand shape: (None, 64, 64, 240)
I0331 21:46:20.370954 8152 api.py:461] Expand shape: (None, 64, 64, 240)
I0331 21:46:20.541965 18380 api.py:461] DWConv shape: (None, 32, 32, 240)
I0331 21:46:20.563966 18380 api.py:461] Built SE se : (None, 1, 1, 240)
I0331 21:46:20.611969 15064 api.py:461] DWConv shape: (None, 32, 32, 240)
I0331 21:46:20.632970 15064 api.py:461] Built SE se : (None, 1, 1, 240)
I0331 21:46:20.704973 8152 api.py:461] DWConv shape: (None, 32, 32, 240)
I0331 21:46:20.725980 8152 api.py:461] Built SE se : (None, 1, 1, 240)
I0331 21:46:20.881984 18380 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:20.894985 18380 api.py:461] Block blocks_6 input shape: (None, 32, 32, 80)
I0331 21:46:20.957986 15064 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:20.974989 15064 api.py:461] Block blocks_6 input shape: (None, 32, 32, 80)
I0331 21:46:21.019996 8152 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:21.032996 8152 api.py:461] Block blocks_6 input shape: (None, 32, 32, 80)
I0331 21:46:21.207008 18380 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:21.279024 15064 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:21.349010 8152 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:21.508278 18380 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:21.530278 18380 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:21.580281 15064 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:21.602283 15064 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:21.673289 8152 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:21.695289 8152 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:21.854295 18380 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:21.867297 18380 api.py:461] Block blocks_7 input shape: (None, 32, 32, 80)
I0331 21:46:21.935303 15064 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:21.947301 15064 api.py:461] Block blocks_7 input shape: (None, 32, 32, 80)
I0331 21:46:22.005305 8152 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:22.018306 8152 api.py:461] Block blocks_7 input shape: (None, 32, 32, 80)
I0331 21:46:22.191312 18380 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:22.278320 15064 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:22.328331 8152 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:22.489329 18380 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:22.511335 18380 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:22.559338 15064 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:22.580339 15064 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:22.650342 8152 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:22.672343 8152 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:22.831499 18380 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:22.843501 18380 api.py:461] Block blocks_8 input shape: (None, 32, 32, 80)
I0331 21:46:22.889503 15064 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:22.901504 15064 api.py:461] Block blocks_8 input shape: (None, 32, 32, 80)
I0331 21:46:22.981509 8152 api.py:461] Project shape: (None, 32, 32, 80)
I0331 21:46:23.003513 8152 api.py:461] Block blocks_8 input shape: (None, 32, 32, 80)
I0331 21:46:23.162523 18380 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:23.212522 15064 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:23.285525 8152 api.py:461] Expand shape: (None, 32, 32, 480)
I0331 21:46:23.447541 18380 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:23.469541 18380 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:23.519540 15064 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:23.541541 15064 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:23.612544 8152 api.py:461] DWConv shape: (None, 32, 32, 480)
I0331 21:46:23.633548 8152 api.py:461] Built SE se : (None, 1, 1, 480)
I0331 21:46:23.798553 18380 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:23.810555 18380 api.py:461] Block blocks_9 input shape: (None, 32, 32, 112)
I0331 21:46:23.878564 15064 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:23.890564 15064 api.py:461] Block blocks_9 input shape: (None, 32, 32, 112)
I0331 21:46:23.935063 8152 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:23.948062 8152 api.py:461] Block blocks_9 input shape: (None, 32, 32, 112)
I0331 21:46:24.142064 18380 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:24.213065 15064 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:24.284067 8152 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:24.443081 18380 api.py:461] DWConv shape: (None, 32, 32, 672)
I0331 21:46:24.466083 18380 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:24.535083 15064 api.py:461] DWConv shape: (None, 32, 32, 672)
I0331 21:46:24.557084 15064 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:24.626086 8152 api.py:461] DWConv shape: (None, 32, 32, 672)
I0331 21:46:24.647090 8152 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:24.803099 18380 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:24.815099 18380 api.py:461] Block blocks_10 input shape: (None, 32, 32, 112)
I0331 21:46:24.980109 15064 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:24.993110 15064 api.py:461] Block blocks_10 input shape: (None, 32, 32, 112)
I0331 21:46:25.072114 8152 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:25.085116 8152 api.py:461] Block blocks_10 input shape: (None, 32, 32, 112)
I0331 21:46:25.246585 18380 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:25.319592 15064 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:25.390596 8152 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:25.550601 18380 api.py:461] DWConv shape: (None, 32, 32, 672)
I0331 21:46:25.572605 18380 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:25.644610 15064 api.py:461] DWConv shape: (None, 32, 32, 672)
I0331 21:46:25.665611 15064 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:25.736617 8152 api.py:461] DWConv shape: (None, 32, 32, 672)
I0331 21:46:25.757617 8152 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:25.913621 18380 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:25.926623 18380 api.py:461] Block blocks_11 input shape: (None, 32, 32, 112)
I0331 21:46:25.995631 15064 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:26.007633 15064 api.py:461] Block blocks_11 input shape: (None, 32, 32, 112)
I0331 21:46:26.074635 8152 api.py:461] Project shape: (None, 32, 32, 112)
I0331 21:46:26.087636 8152 api.py:461] Block blocks_11 input shape: (None, 32, 32, 112)
I0331 21:46:26.248643 18380 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:26.318650 15064 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:26.389653 8152 api.py:461] Expand shape: (None, 32, 32, 672)
I0331 21:46:26.548658 18380 api.py:461] DWConv shape: (None, 16, 16, 672)
I0331 21:46:26.569665 18380 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:26.618666 15064 api.py:461] DWConv shape: (None, 16, 16, 672)
I0331 21:46:26.640667 15064 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:26.712671 8152 api.py:461] DWConv shape: (None, 16, 16, 672)
I0331 21:46:26.733672 8152 api.py:461] Built SE se : (None, 1, 1, 672)
I0331 21:46:26.890678 18380 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:26.903682 18380 api.py:461] Block blocks_12 input shape: (None, 16, 16, 192)
I0331 21:46:26.953686 15064 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:26.965685 15064 api.py:461] Block blocks_12 input shape: (None, 16, 16, 192)
I0331 21:46:27.037692 8152 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:27.050691 8152 api.py:461] Block blocks_12 input shape: (None, 16, 16, 192)
I0331 21:46:27.234698 18380 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:27.310706 15064 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:27.383710 8152 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:27.545717 18380 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:27.567720 18380 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:27.639726 15064 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:27.660726 15064 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:27.731730 8152 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:27.752730 8152 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:27.907734 18380 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:27.919738 18380 api.py:461] Block blocks_13 input shape: (None, 16, 16, 192)
I0331 21:46:27.991746 15064 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:28.004745 15064 api.py:461] Block blocks_13 input shape: (None, 16, 16, 192)
I0331 21:46:28.091751 8152 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:28.105751 8152 api.py:461] Block blocks_13 input shape: (None, 16, 16, 192)
I0331 21:46:28.274757 18380 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:28.329764 15064 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:28.384766 8152 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:28.542772 18380 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:28.565776 18380 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:28.640782 15064 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:28.662782 15064 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:28.732786 8152 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:28.754789 8152 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:28.923794 18380 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:28.936795 18380 api.py:461] Block blocks_14 input shape: (None, 16, 16, 192)
I0331 21:46:29.006798 15064 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:29.018802 15064 api.py:461] Block blocks_14 input shape: (None, 16, 16, 192)
I0331 21:46:29.082806 8152 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:29.107809 8152 api.py:461] Block blocks_14 input shape: (None, 16, 16, 192)
I0331 21:46:29.262817 18380 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:29.339905 15064 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:29.392908 8152 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:29.553921 18380 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:29.575922 18380 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:29.626933 15064 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:29.648926 15064 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:29.719926 8152 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:29.740928 8152 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:29.895710 18380 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:29.908711 18380 api.py:461] Block blocks_15 input shape: (None, 16, 16, 192)
I0331 21:46:29.955714 15064 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:29.967716 15064 api.py:461] Block blocks_15 input shape: (None, 16, 16, 192)
I0331 21:46:30.013717 8152 api.py:461] Project shape: (None, 16, 16, 192)
I0331 21:46:30.025719 8152 api.py:461] Block blocks_15 input shape: (None, 16, 16, 192)
I0331 21:46:30.188728 18380 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:30.266729 15064 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:30.342732 8152 api.py:461] Expand shape: (None, 16, 16, 1152)
I0331 21:46:30.506742 18380 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:30.527744 18380 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:30.582747 15064 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:30.625747 15064 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:30.675755 8152 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0331 21:46:30.696756 8152 api.py:461] Built SE se : (None, 1, 1, 1152)
I0331 21:46:30.867762 18380 api.py:461] Project shape: (None, 16, 16, 320)
I0331 21:46:30.940771 15064 api.py:461] Project shape: (None, 16, 16, 320)
I0331 21:46:30.988032 8152 api.py:461] Project shape: (None, 16, 16, 320)
INFO:tensorflow:Error reported to Coordinator: Failed to convert object of type <class 'list'> to Tensor. Contents: [None, 64, 64, -1]. Consider casting elements to a supported type.
Traceback (most recent call last):
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 549, in make_tensor_proto
str_values = [compat.as_bytes(x) for x in proto_values]
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 549, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception
yield
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run
self.main_result = self.main_fn(*self.main_args, self.main_kwargs)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 667, in wrapper
return converted_call(f, args, kwargs, options=options)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 396, in converted_call
return _call_unconverted(f, args, kwargs, options)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 478, in _call_unconverted
return f(*args, *kwargs)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\keras\engine\training.py", line 788, in run_step
outputs = model.train_step(data)
File "C:\Users\ha485\PycharmProjects\Official4\efficientdet\keras\train_lib.py", line 766, in train_step
loss_vals)
File "C:\Users\ha485\PycharmProjects\Official4\efficientdet\keras\train_lib.py", line 674, in _detection_loss
[bs, width, height, -1])
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
return target(args, kwargs)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\ops\array_ops.py", line 195, in reshape
result = gen_array_ops.reshape(tensor, shape, name)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 8377, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 525, in _apply_op_helper
raise err
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 515, in _apply_op_helper
preferred_dtype=default_dtype)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\profiler\trace.py", line 163, in wrapped
return func(*args, **kwargs)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\ops.py", line 1540, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\constant_op.py", line 339, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\constant_op.py", line 265, in constant
allow_broadcast=True)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\constant_op.py", line 283, in _constant_impl
allow_broadcast=allow_broadcast))
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 553, in make_tensor_proto
"supported type." % (type(values), values))
TypeError: Failed to convert object of type <class 'list'> to Tensor. Contents: [None, 64, 64, -1]. Consider casting elements to a supported type.
I0331 21:46:48.000990 18380 coordinator.py:219] Error reported to Coordinator: Failed to convert object of type <class 'list'> to Tensor. Contents: [None, 64, 64, -1]. Consider casting elements to a supported type.
Traceback (most recent call last):
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 549, in make_tensor_proto
str_values = [compat.as_bytes(x) for x in proto_values]
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 549, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\training\coordinator.py", line 297, in stop_on_exception
yield
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py", line 323, in run
self.main_result = self.main_fn(*self.main_args, self.main_kwargs)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 667, in wrapper
return converted_call(f, args, kwargs, options=options)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 396, in converted_call
return _call_unconverted(f, args, kwargs, options)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 478, in _call_unconverted
return f(*args, *kwargs)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\keras\engine\training.py", line 788, in run_step
outputs = model.train_step(data)
File "C:\Users\ha485\PycharmProjects\Official4\efficientdet\keras\train_lib.py", line 766, in train_step
loss_vals)
File "C:\Users\ha485\PycharmProjects\Official4\efficientdet\keras\train_lib.py", line 674, in _detection_loss
[bs, width, height, -1])
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
return target(args, kwargs)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\ops\array_ops.py", line 195, in reshape
result = gen_array_ops.reshape(tensor, shape, name)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 8377, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 525, in _apply_op_helper
raise err
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 515, in _apply_op_helper
preferred_dtype=default_dtype)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\profiler\trace.py", line 163, in wrapped
return func(*args, kwargs)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\ops.py", line 1540, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\constant_op.py", line 339, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\constant_op.py", line 265, in constant
allow_broadcast=True)
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\constant_op.py", line 283, in _constant_impl
allow_broadcast=allow_broadcast))
File "C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 553, in make_tensor_proto
"supported type." % (type(values), values))
TypeError: Failed to convert object of type <class 'list'> to Tensor. Contents: [None, 64, 64, -1]. Consider casting elements to a supported type.
Traceback (most recent call last):
File "C:/Users/ha485/PycharmProjects/Official4/efficientdet/keras/train.py", line 280, in
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\keras\engine\training.py:805 train_function *
return step_function(self, iterator)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\keras\engine\training.py:795 step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:1259 run
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2730 call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\distribute\mirrored_strategy.py:629 _call_for_each_replica
self._container_strategy(), fn, args, kwargs)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py:93 call_for_each_replica
return _call_for_each_replica(strategy, fn, args, kwargs)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py:234 _call_for_each_replica
coord.join(threads)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\training\coordinator.py:389 join
six.reraise(*self._exc_info_to_raise)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\six.py:703 reraise
raise value
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\training\coordinator.py:297 stop_on_exception
yield
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\distribute\mirrored_run.py:323 run
self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\keras\engine\training.py:788 run_step **
outputs = model.train_step(data)
C:\Users\ha485\PycharmProjects\Official4\efficientdet\keras\train_lib.py:766 train_step
loss_vals)
C:\Users\ha485\PycharmProjects\Official4\efficientdet\keras\train_lib.py:674 _detection_loss
[bs, width, height, -1])
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\util\dispatch.py:201 wrapper
return target(*args, **kwargs)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\ops\array_ops.py:195 reshape
result = gen_array_ops.reshape(tensor, shape, name)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\ops\gen_array_ops.py:8377 reshape
"Reshape", tensor=tensor, shape=shape, name=name)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\op_def_library.py:525 _apply_op_helper
raise err
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\op_def_library.py:515 _apply_op_helper
preferred_dtype=default_dtype)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\profiler\trace.py:163 wrapped
return func(*args, **kwargs)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\ops.py:1540 convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\constant_op.py:339 _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\constant_op.py:265 constant
allow_broadcast=True)
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\constant_op.py:283 _constant_impl
allow_broadcast=allow_broadcast))
C:\Users\ha485\.conda\envs\'Official4'\lib\site-packages\tensorflow\python\framework\tensor_util.py:553 make_tensor_proto
"supported type." % (type(values), values))
**TypeError: Failed to convert object of type <class 'list'> to Tensor. Contents: [None, 64, 64, -1]. Consider casting elements to a supported type.**@juliangrosshauser @fsx950223 Thanks for your reply. I've reinstalled the whole env to fix the problem. (Win 10, Tensorflow 2.4.1, Keras trainning) I've tried 3 approaches.
1.just run, 2. use the command, 3. use PYTHONPATH=./
2. use the command python keras/train.py --train_file_pattern=tfrecord/train/.tfrecord --val_file_pattern=tfrecord/val//.tfrecord --val_file_pattern=efficientdet/tfrecord/*.json --model_name=efficientdet-d0 --model_dir=efficientdet/output --batch_size=64 --num_epochs=50 --hparams=configs/config.yaml
2. use the command(Log)
('Official4') C:\Users\ha485\PycharmProjects\Official4\efficientdet>python keras/train.py --train_file_pattern=tfrecord/train/.tfrecord --val_file_pattern=tfrecord/val//.tfrecord --val_file_pattern=effic
ientdet/tfrecord/*.json --model_name=efficientdet-d0 --model_dir=efficientdet/output --batch_size=64 --num_epochs=50 --hparams=configs/config.yaml
2021-03-31 21:55:24.521238: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Traceback (most recent call last):
File "keras/train.py", line 22, in
Ronald-Kray
commented
3 years ago @juliangrosshauser @fsx950223 Thanks for your reply. I've reinstalled the whole env to fix the problem. (Win 10, Tensorflow 2.4.1, Keras trainning) I've tried 3 approaches. I set the variable env, but I can't use PYTHONPATH=./
1.just run, 2. use the command, 3. use PYTHONPATH=./
3. use PYTHONPATH=./ PYTHONPATH=./ python keras/train.py --train_file_pattern=tfrecord/train/.tfrecord --val_file_pattern=tfrecord/val//.tfrecord --val_file_pattern=efficientdet/tfrecord/*.json --model_name=efficientdet-d0 --model_dir=efficientdet/output --batch_size=64 --num_epochs=50 --hparams=configs/config.yaml
3. use PYTHONPATH=./(Log) 'PYTHONPATH' is not recognized as an internal or external command, operable program or batch file.
fsx950223
commented
3 years ago Just tun it, set batch_size=48 and tf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())
Ronald-Kray
commented
3 years ago Just tun it, set
batch_size=48andtf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())
@fsx950223 Thank you for your reply. The code stays still for a while. It seems like 1 epoch. Is it correct?
2021-04-01 13:32:49.676100: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-01 13:32:54.460897: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-01 13:32:54.463221: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-04-01 13:32:54.574630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:54.574931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:54.575162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:54.575380: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-01 13:32:54.597229: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-01 13:32:54.597362: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-01 13:32:54.602366: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-01 13:32:54.604210: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-01 13:32:54.614894: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-01 13:32:54.618698: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-01 13:32:54.619884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-04-01 13:32:54.620124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2
2021-04-01 13:32:54.620621: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-01 13:32:55.147768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:55.148033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:55.148278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:55.148515: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-01 13:32:55.148638: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-01 13:32:55.148772: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-01 13:32:55.148899: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-01 13:32:55.149006: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-01 13:32:55.149116: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-01 13:32:55.149227: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-01 13:32:55.149337: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-04-01 13:32:55.149509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2
2021-04-01 13:32:56.745811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-01 13:32:56.745937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2
2021-04-01 13:32:56.746007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N N N
2021-04-01 13:32:56.746075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: N N N
2021-04-01 13:32:56.746143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: N N N
2021-04-01 13:32:56.746447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9417 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5)
2021-04-01 13:32:56.748183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9417 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5)
2021-04-01 13:32:56.749546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9417 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5)
2021-04-01 13:32:56.750724: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0401 13:32:56.753754 19644 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0401 13:32:56.753754 19644 train.py:182] All devices: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU')]
I0401 13:32:57.880417 19644 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x00000195631E3400>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False)
I0401 13:32:58.177420 19644 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0401 13:32:58.178419 19644 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0401 13:32:58.179418 19644 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0401 13:32:58.180418 19644 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0401 13:32:58.180418 19644 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0401 13:32:58.181418 19644 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0401 13:32:58.182418 19644 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0401 13:32:58.183420 19644 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0401 13:32:58.184418 19644 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0401 13:32:58.185418 19644 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0401 13:32:58.185418 19644 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0401 13:32:58.186418 19644 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0401 13:32:58.187418 19644 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0401 13:32:58.188419 19644 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0401 13:32:58.189418 19644 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0401 13:32:58.189418 19644 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0401 13:32:58.191419 19644 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0401 13:32:58.191419 19644 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0401 13:32:58.192420 19644 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0401 13:32:58.193420 19644 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0401 13:32:58.194419 19644 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0401 13:32:58.195420 19644 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0401 13:32:58.196420 19644 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0401 13:32:58.197419 19644 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.813437 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.815436 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.820438 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.822438 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.881438 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.894438 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:1402: UserWarning: layer.updates will be removed in a future version. This property should not be used in TensorFlow 2.0, as updates are applied automatically.
warnings.warn('layer.updates will be removed in a future version. '
I0401 13:33:00.264466 19644 api.py:461] Built stem stem : (None, 256, 256, 32)
I0401 13:33:02.286643 19644 api.py:461] Block blocks_0 input shape: (None, 256, 256, 32)
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.322646 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.324648 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.328646 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.330646 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.352647 19644 api.py:461] DWConv shape: (None, 256, 256, 32)
I0401 13:33:02.523651 19644 api.py:461] Built SE se : (None, 1, 1, 32)
I0401 13:33:02.577652 19644 api.py:461] Project shape: (None, 256, 256, 16)
I0401 13:33:02.623652 19644 api.py:461] Block blocks_1 input shape: (None, 256, 256, 16)
I0401 13:33:02.675653 19644 api.py:461] Expand shape: (None, 256, 256, 96)
I0401 13:33:02.725654 19644 api.py:461] DWConv shape: (None, 128, 128, 96)
I0401 13:33:02.762655 19644 api.py:461] Built SE se : (None, 1, 1, 96)
I0401 13:33:02.838654 19644 api.py:461] Project shape: (None, 128, 128, 24)
I0401 13:33:02.854655 19644 api.py:461] Block blocks_2 input shape: (None, 128, 128, 24)
I0401 13:33:02.911658 19644 api.py:461] Expand shape: (None, 128, 128, 144)
I0401 13:33:02.961659 19644 api.py:461] DWConv shape: (None, 128, 128, 144)
I0401 13:33:02.996660 19644 api.py:461] Built SE se : (None, 1, 1, 144)
I0401 13:33:03.046661 19644 api.py:461] Project shape: (None, 128, 128, 24)
I0401 13:33:03.059662 19644 api.py:461] Block blocks_3 input shape: (None, 128, 128, 24)
I0401 13:33:03.110662 19644 api.py:461] Expand shape: (None, 128, 128, 144)
I0401 13:33:03.161663 19644 api.py:461] DWConv shape: (None, 64, 64, 144)
I0401 13:33:03.195664 19644 api.py:461] Built SE se : (None, 1, 1, 144)
I0401 13:33:03.245665 19644 api.py:461] Project shape: (None, 64, 64, 40)
I0401 13:33:03.258665 19644 api.py:461] Block blocks_4 input shape: (None, 64, 64, 40)
I0401 13:33:03.309667 19644 api.py:461] Expand shape: (None, 64, 64, 240)
I0401 13:33:03.360667 19644 api.py:461] DWConv shape: (None, 64, 64, 240)
I0401 13:33:03.396668 19644 api.py:461] Built SE se : (None, 1, 1, 240)
I0401 13:33:03.445669 19644 api.py:461] Project shape: (None, 64, 64, 40)
I0401 13:33:03.458669 19644 api.py:461] Block blocks_5 input shape: (None, 64, 64, 40)
I0401 13:33:03.510671 19644 api.py:461] Expand shape: (None, 64, 64, 240)
I0401 13:33:03.570672 19644 api.py:461] DWConv shape: (None, 32, 32, 240)
I0401 13:33:03.608672 19644 api.py:461] Built SE se : (None, 1, 1, 240)
I0401 13:33:03.659673 19644 api.py:461] Project shape: (None, 32, 32, 80)
I0401 13:33:03.672674 19644 api.py:461] Block blocks_6 input shape: (None, 32, 32, 80)
I0401 13:33:03.723674 19644 api.py:461] Expand shape: (None, 32, 32, 480)
I0401 13:33:03.774676 19644 api.py:461] DWConv shape: (None, 32, 32, 480)
I0401 13:33:03.811676 19644 api.py:461] Built SE se : (None, 1, 1, 480)
I0401 13:33:03.860677 19644 api.py:461] Project shape: (None, 32, 32, 80)
I0401 13:33:03.873677 19644 api.py:461] Block blocks_7 input shape: (None, 32, 32, 80)
I0401 13:33:03.923678 19644 api.py:461] Expand shape: (None, 32, 32, 480)
I0401 13:33:03.973680 19644 api.py:461] DWConv shape: (None, 32, 32, 480)
I0401 13:33:04.007680 19644 api.py:461] Built SE se : (None, 1, 1, 480)
I0401 13:33:04.059682 19644 api.py:461] Project shape: (None, 32, 32, 80)
I0401 13:33:04.079680 19644 api.py:461] Block blocks_8 input shape: (None, 32, 32, 80)
I0401 13:33:04.143683 19644 api.py:461] Expand shape: (None, 32, 32, 480)
I0401 13:33:04.194801 19644 api.py:461] DWConv shape: (None, 32, 32, 480)
I0401 13:33:04.241802 19644 api.py:461] Built SE se : (None, 1, 1, 480)
I0401 13:33:04.303804 19644 api.py:461] Project shape: (None, 32, 32, 112)
I0401 13:33:04.316805 19644 api.py:461] Block blocks_9 input shape: (None, 32, 32, 112)
I0401 13:33:04.366807 19644 api.py:461] Expand shape: (None, 32, 32, 672)
I0401 13:33:04.417807 19644 api.py:461] DWConv shape: (None, 32, 32, 672)
I0401 13:33:04.465809 19644 api.py:461] Built SE se : (None, 1, 1, 672)
I0401 13:33:04.527812 19644 api.py:461] Project shape: (None, 32, 32, 112)
I0401 13:33:04.541812 19644 api.py:461] Block blocks_10 input shape: (None, 32, 32, 112)
I0401 13:33:04.598810 19644 api.py:461] Expand shape: (None, 32, 32, 672)
I0401 13:33:04.652812 19644 api.py:461] DWConv shape: (None, 32, 32, 672)
I0401 13:33:04.687812 19644 api.py:461] Built SE se : (None, 1, 1, 672)
I0401 13:33:04.738812 19644 api.py:461] Project shape: (None, 32, 32, 112)
I0401 13:33:04.751813 19644 api.py:461] Block blocks_11 input shape: (None, 32, 32, 112)
I0401 13:33:04.803815 19644 api.py:461] Expand shape: (None, 32, 32, 672)
I0401 13:33:04.853815 19644 api.py:461] DWConv shape: (None, 16, 16, 672)
I0401 13:33:04.887815 19644 api.py:461] Built SE se : (None, 1, 1, 672)
I0401 13:33:04.937820 19644 api.py:461] Project shape: (None, 16, 16, 192)
I0401 13:33:04.949817 19644 api.py:461] Block blocks_12 input shape: (None, 16, 16, 192)
I0401 13:33:05.001821 19644 api.py:461] Expand shape: (None, 16, 16, 1152)
I0401 13:33:05.053819 19644 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0401 13:33:05.088820 19644 api.py:461] Built SE se : (None, 1, 1, 1152)
I0401 13:33:05.137821 19644 api.py:461] Project shape: (None, 16, 16, 192)
I0401 13:33:05.149821 19644 api.py:461] Block blocks_13 input shape: (None, 16, 16, 192)
I0401 13:33:05.200822 19644 api.py:461] Expand shape: (None, 16, 16, 1152)
I0401 13:33:05.252825 19644 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0401 13:33:05.290823 19644 api.py:461] Built SE se : (None, 1, 1, 1152)
I0401 13:33:05.339825 19644 api.py:461] Project shape: (None, 16, 16, 192)
I0401 13:33:05.352825 19644 api.py:461] Block blocks_14 input shape: (None, 16, 16, 192)
I0401 13:33:05.402827 19644 api.py:461] Expand shape: (None, 16, 16, 1152)
I0401 13:33:05.452827 19644 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0401 13:33:05.496834 19644 api.py:461] Built SE se : (None, 1, 1, 1152)
I0401 13:33:05.562833 19644 api.py:461] Project shape: (None, 16, 16, 192)
I0401 13:33:05.576830 19644 api.py:461] Block blocks_15 input shape: (None, 16, 16, 192)
I0401 13:33:05.633169 19644 api.py:461] Expand shape: (None, 16, 16, 1152)
I0401 13:33:05.684171 19644 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0401 13:33:05.719170 19644 api.py:461] Built SE se : (None, 1, 1, 1152)
I0401 13:33:05.772173 19644 api.py:461] Project shape: (None, 16, 16, 320)
I0401 13:33:10.935725 19644 train_lib.py:218] LR schedule method: cosine
I0401 13:33:10.936728 19644 train_lib.py:296] Use SGD optimizer
I0401 13:33:11.334733 19644 dataloader.py:85] target_size = (512, 512), output_size = (512, 512)
2021-04-01 13:33:12.377972: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/100
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.002767 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.069780 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.143770 5584 api.py:461] Built stem stem : (16, 256, 256, 32)
I0401 13:33:13.155770 5584 api.py:461] Block blocks_0 input shape: (16, 256, 256, 32)
I0401 13:33:13.202770 15184 api.py:461] Built stem stem : (16, 256, 256, 32)
I0401 13:33:13.214773 15184 api.py:461] Block blocks_0 input shape: (16, 256, 256, 32)
I0401 13:33:13.285774 16568 api.py:461] Built stem stem : (16, 256, 256, 32)
I0401 13:33:13.299774 16568 api.py:461] Block blocks_0 input shape: (16, 256, 256, 32)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.323771 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.398787 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.476475 5584 api.py:461] DWConv shape: (16, 256, 256, 32)
I0401 13:33:13.499487 5584 api.py:461] Built SE se : (16, 1, 1, 32)
I0401 13:33:13.554477 15184 api.py:461] DWConv shape: (16, 256, 256, 32)
I0401 13:33:13.577477 15184 api.py:461] Built SE se : (16, 1, 1, 32)
I0401 13:33:13.650478 16568 api.py:461] DWConv shape: (16, 256, 256, 32)
I0401 13:33:13.689482 16568 api.py:461] Built SE se : (16, 1, 1, 32)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.718396 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.811389 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.892391 5584 api.py:461] Project shape: (16, 256, 256, 16)
I0401 13:33:13.905393 5584 api.py:461] Block blocks_1 input shape: (16, 256, 256, 16)
I0401 13:33:13.953395 15184 api.py:461] Project shape: (16, 256, 256, 16)
I0401 13:33:13.965397 15184 api.py:461] Block blocks_1 input shape: (16, 256, 256, 16)
I0401 13:33:14.043401 16568 api.py:461] Project shape: (16, 256, 256, 16)
I0401 13:33:14.055398 16568 api.py:461] Block blocks_1 input shape: (16, 256, 256, 16)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.077399 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.184402 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.282399 5584 api.py:461] Expand shape: (16, 256, 256, 96)
I0401 13:33:14.356401 15184 api.py:461] Expand shape: (16, 256, 256, 96)
I0401 13:33:14.409404 16568 api.py:461] Expand shape: (16, 256, 256, 96)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.431405 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.571408 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.676407 5584 api.py:461] DWConv shape: (16, 128, 128, 96)
I0401 13:33:14.699410 5584 api.py:461] Built SE se : (16, 1, 1, 96)
I0401 13:33:14.756411 15184 api.py:461] DWConv shape: (16, 128, 128, 96)
I0401 13:33:14.777412 15184 api.py:461] Built SE se : (16, 1, 1, 96)
I0401 13:33:14.856413 16568 api.py:461] DWConv shape: (16, 128, 128, 96)
I0401 13:33:14.877413 16568 api.py:461] Built SE se : (16, 1, 1, 96)
I0401 13:33:15.144311 5584 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:15.157311 5584 api.py:461] Block blocks_2 input shape: (16, 128, 128, 24)
I0401 13:33:15.214312 15184 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:15.227313 15184 api.py:461] Block blocks_2 input shape: (16, 128, 128, 24)
I0401 13:33:15.307315 16568 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:15.319317 16568 api.py:461] Block blocks_2 input shape: (16, 128, 128, 24)
I0401 13:33:15.600446 5584 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:15.663447 15184 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:15.750446 16568 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:16.027456 5584 api.py:461] DWConv shape: (16, 128, 128, 144)
I0401 13:33:16.048456 5584 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:16.138456 15184 api.py:461] DWConv shape: (16, 128, 128, 144)
I0401 13:33:16.159457 15184 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:16.250456 16568 api.py:461] DWConv shape: (16, 128, 128, 144)
I0401 13:33:16.273456 16568 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:16.611545 5584 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:16.625548 5584 api.py:461] Block blocks_3 input shape: (16, 128, 128, 24)
I0401 13:33:16.696546 15184 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:16.815549 15184 api.py:461] Block blocks_3 input shape: (16, 128, 128, 24)
I0401 13:33:16.911551 16568 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:16.924551 16568 api.py:461] Block blocks_3 input shape: (16, 128, 128, 24)
I0401 13:33:17.257809 5584 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:17.331811 15184 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:17.430813 16568 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:17.787820 5584 api.py:461] DWConv shape: (16, 64, 64, 144)
I0401 13:33:17.809821 5584 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:17.888824 15184 api.py:461] DWConv shape: (16, 64, 64, 144)
I0401 13:33:17.909823 15184 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:18.015828 16568 api.py:461] DWConv shape: (16, 64, 64, 144)
I0401 13:33:18.037827 16568 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:18.408649 5584 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:18.420649 5584 api.py:461] Block blocks_4 input shape: (16, 64, 64, 40)
I0401 13:33:18.504652 15184 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:18.517652 15184 api.py:461] Block blocks_4 input shape: (16, 64, 64, 40)
I0401 13:33:18.618656 16568 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:18.630653 16568 api.py:461] Block blocks_4 input shape: (16, 64, 64, 40)
I0401 13:33:19.016661 5584 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:19.106663 15184 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:19.222663 16568 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:19.616448 5584 api.py:461] DWConv shape: (16, 64, 64, 240)
I0401 13:33:19.638448 5584 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:19.751451 15184 api.py:461] DWConv shape: (16, 64, 64, 240)
I0401 13:33:19.772450 15184 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:19.872453 16568 api.py:461] DWConv shape: (16, 64, 64, 240)
I0401 13:33:19.894454 16568 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:20.365462 5584 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:20.378463 5584 api.py:461] Block blocks_5 input shape: (16, 64, 64, 40)
I0401 13:33:20.503467 15184 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:20.517466 15184 api.py:461] Block blocks_5 input shape: (16, 64, 64, 40)
I0401 13:33:20.654469 16568 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:20.668469 16568 api.py:461] Block blocks_5 input shape: (16, 64, 64, 40)
I0401 13:33:21.145476 5584 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:21.257478 15184 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:21.422487 16568 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:21.967557 5584 api.py:461] DWConv shape: (16, 32, 32, 240)
I0401 13:33:22.006561 5584 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:22.167561 15184 api.py:461] DWConv shape: (16, 32, 32, 240)
I0401 13:33:22.188561 15184 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:22.334564 16568 api.py:461] DWConv shape: (16, 32, 32, 240)
I0401 13:33:22.356566 16568 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:22.959574 5584 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:22.971574 5584 api.py:461] Block blocks_6 input shape: (16, 32, 32, 80)
I0401 13:33:23.117581 15184 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:23.129580 15184 api.py:461] Block blocks_6 input shape: (16, 32, 32, 80)
I0401 13:33:23.270585 16568 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:23.291584 16568 api.py:461] Block blocks_6 input shape: (16, 32, 32, 80)
I0401 13:33:23.875598 5584 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:24.055599 15184 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:24.193602 16568 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:24.842617 5584 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:24.864617 5584 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:25.049619 15184 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:25.071619 15184 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:25.240623 16568 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:25.262624 16568 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:25.938634 5584 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:25.951634 5584 api.py:461] Block blocks_7 input shape: (16, 32, 32, 80)
I0401 13:33:26.123642 15184 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:26.144641 15184 api.py:461] Block blocks_7 input shape: (16, 32, 32, 80)
I0401 13:33:26.321645 16568 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:26.334645 16568 api.py:461] Block blocks_7 input shape: (16, 32, 32, 80)
I0401 13:33:27.039657 5584 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:27.221662 15184 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:27.386666 16568 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:28.142679 5584 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:28.163679 5584 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:28.356688 15184 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:28.378686 15184 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:28.563693 16568 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:28.588694 16568 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:29.354706 5584 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:29.367706 5584 api.py:461] Block blocks_8 input shape: (16, 32, 32, 80)
I0401 13:33:29.544710 15184 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:29.557710 15184 api.py:461] Block blocks_8 input shape: (16, 32, 32, 80)
I0401 13:33:29.792715 16568 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:29.804715 16568 api.py:461] Block blocks_8 input shape: (16, 32, 32, 80)
I0401 13:33:30.620502 5584 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:30.840506 15184 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:31.069518 16568 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:31.927490 5584 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:31.949491 5584 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:32.193496 15184 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:32.214496 15184 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:32.455504 16568 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:32.479505 16568 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:33.374524 5584 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:33.387524 5584 api.py:461] Block blocks_9 input shape: (16, 32, 32, 112)
I0401 13:33:33.626529 15184 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:33.639529 15184 api.py:461] Block blocks_9 input shape: (16, 32, 32, 112)
I0401 13:33:33.858532 16568 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:33.870533 16568 api.py:461] Block blocks_9 input shape: (16, 32, 32, 112)
I0401 13:33:34.821553 5584 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:35.067557 15184 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:35.341562 16568 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:36.334579 5584 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:36.356580 5584 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:36.600589 15184 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:36.623590 15184 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:36.896599 16568 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:36.918594 16568 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:37.980613 5584 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:37.993613 5584 api.py:461] Block blocks_10 input shape: (16, 32, 32, 112)
I0401 13:33:38.286622 15184 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:38.299623 15184 api.py:461] Block blocks_10 input shape: (16, 32, 32, 112)
I0401 13:33:38.579628 16568 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:38.717630 16568 api.py:461] Block blocks_10 input shape: (16, 32, 32, 112)
I0401 13:33:39.862655 5584 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:40.169661 15184 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:40.435666 16568 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:41.644690 5584 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:41.668691 5584 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:41.990698 15184 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:42.012701 15184 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:42.314705 16568 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:42.335704 16568 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:43.568892 5584 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:43.582893 5584 api.py:461] Block blocks_11 input shape: (16, 32, 32, 112)
I0401 13:33:43.894900 15184 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:43.907900 15184 api.py:461] Block blocks_11 input shape: (16, 32, 32, 112)
I0401 13:33:44.222905 16568 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:44.235906 16568 api.py:461] Block blocks_11 input shape: (16, 32, 32, 112)
I0401 13:33:45.598932 5584 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:45.925939 15184 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:46.265945 16568 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:47.628980 5584 api.py:461] DWConv shape: (16, 16, 16, 672)
I0401 13:33:47.651974 5584 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:47.991981 15184 api.py:461] DWConv shape: (16, 16, 16, 672)
I0401 13:33:48.013982 15184 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:48.381986 16568 api.py:461] DWConv shape: (16, 16, 16, 672)
I0401 13:33:48.403987 16568 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:49.867761 5584 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:49.879762 5584 api.py:461] Block blocks_12 input shape: (16, 16, 16, 192)
I0401 13:33:50.239769 15184 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:50.251769 15184 api.py:461] Block blocks_12 input shape: (16, 16, 16, 192)
I0401 13:33:50.591777 16568 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:50.604779 16568 api.py:461] Block blocks_12 input shape: (16, 16, 16, 192)
I0401 13:33:52.058806 5584 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:33:52.412813 15184 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:33:52.801823 16568 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:33:54.358854 5584 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:33:54.380852 5584 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:33:54.779860 15184 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:33:54.800862 15184 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:33:55.233870 16568 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:33:55.255873 16568 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:33:56.933907 5584 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:56.945905 5584 api.py:461] Block blocks_13 input shape: (16, 16, 16, 192)
I0401 13:33:57.329912 15184 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:57.342914 15184 api.py:461] Block blocks_13 input shape: (16, 16, 16, 192)
I0401 13:33:57.790924 16568 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:57.802922 16568 api.py:461] Block blocks_13 input shape: (16, 16, 16, 192)
I0401 13:33:59.529071 5584 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:33:59.965076 15184 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:00.406087 16568 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:02.171123 5584 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:02.192123 5584 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:02.642130 15184 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:02.666131 15184 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:03.150315 16568 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:03.173315 16568 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:05.084357 5584 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:05.096357 5584 api.py:461] Block blocks_14 input shape: (16, 16, 16, 192)
I0401 13:34:05.540365 15184 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:05.552365 15184 api.py:461] Block blocks_14 input shape: (16, 16, 16, 192)
I0401 13:34:06.076376 16568 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:06.088376 16568 api.py:461] Block blocks_14 input shape: (16, 16, 16, 192)
I0401 13:34:08.042416 5584 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:08.531427 15184 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:09.074437 16568 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:11.080477 5584 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:11.102478 5584 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:11.606487 15184 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:11.628488 15184 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:12.192500 16568 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:12.214500 16568 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:14.278542 5584 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:14.290542 5584 api.py:461] Block blocks_15 input shape: (16, 16, 16, 192)
I0401 13:34:14.802552 15184 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:14.815554 15184 api.py:461] Block blocks_15 input shape: (16, 16, 16, 192)
I0401 13:34:15.389564 16568 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:15.402565 16568 api.py:461] Block blocks_15 input shape: (16, 16, 16, 192)
I0401 13:34:17.565609 5584 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:18.096620 15184 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:18.679629 16568 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:20.964677 5584 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:20.986677 5584 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:21.531689 15184 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:21.552688 15184 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:22.151701 16568 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:22.174701 16568 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:24.511750 5584 api.py:461] Project shape: (16, 16, 16, 320)
I0401 13:34:25.103761 15184 api.py:461] Project shape: (16, 16, 16, 320)
I0401 13:34:25.726771 16568 api.py:461] Project shape: (16, 16, 16, 320)
Ronald-Kray
commented
3 years ago @fsx950223 Could you give me a comment for me, please?
C:\Users\ha485.conda\envs\'Official4'\python.exe C:/Users/ha485/PycharmProjects/Official4/efficientdet/keras/train.py
2021-04-01 13:32:49.676100: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-01 13:32:54.460897: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-01 13:32:54.463221: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-04-01 13:32:54.574630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:54.574931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:54.575162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:54.575380: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-01 13:32:54.597229: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-01 13:32:54.597362: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-01 13:32:54.602366: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-01 13:32:54.604210: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-01 13:32:54.614894: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-01 13:32:54.618698: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-01 13:32:54.619884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-04-01 13:32:54.620124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2
2021-04-01 13:32:54.620621: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-01 13:32:55.147768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:55.148033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:55.148278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties:
pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2021-04-01 13:32:55.148515: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-01 13:32:55.148638: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-01 13:32:55.148772: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-01 13:32:55.148899: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-01 13:32:55.149006: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-01 13:32:55.149116: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-01 13:32:55.149227: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-01 13:32:55.149337: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-04-01 13:32:55.149509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2
2021-04-01 13:32:56.745811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-01 13:32:56.745937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2
2021-04-01 13:32:56.746007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N N N
2021-04-01 13:32:56.746075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: N N N
2021-04-01 13:32:56.746143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: N N N
2021-04-01 13:32:56.746447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9417 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5)
2021-04-01 13:32:56.748183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9417 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5)
2021-04-01 13:32:56.749546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9417 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5)
2021-04-01 13:32:56.750724: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0401 13:32:56.753754 19644 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
I0401 13:32:56.753754 19644 train.py:182] All devices: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU')]
I0401 13:32:57.880417 19644 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x00000195631E3400>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False)
I0401 13:32:58.177420 19644 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0401 13:32:58.178419 19644 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0401 13:32:58.179418 19644 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0401 13:32:58.180418 19644 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0401 13:32:58.180418 19644 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0401 13:32:58.181418 19644 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0401 13:32:58.182418 19644 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0401 13:32:58.183420 19644 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0401 13:32:58.184418 19644 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0401 13:32:58.185418 19644 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0401 13:32:58.185418 19644 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0401 13:32:58.186418 19644 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0401 13:32:58.187418 19644 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0401 13:32:58.188419 19644 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0401 13:32:58.189418 19644 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0401 13:32:58.189418 19644 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I0401 13:32:58.191419 19644 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I0401 13:32:58.191419 19644 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I0401 13:32:58.192420 19644 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I0401 13:32:58.193420 19644 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I0401 13:32:58.194419 19644 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I0401 13:32:58.195420 19644 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I0401 13:32:58.196420 19644 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I0401 13:32:58.197419 19644 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.813437 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.815436 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.820438 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.822438 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.881438 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:32:58.894438 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
C:\Users\ha485.conda\envs\'Official4'\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:1402: UserWarning: layer.updates will be removed in a future version. This property should not be used in TensorFlow 2.0, as updates are applied automatically.
warnings.warn('layer.updates will be removed in a future version. '
I0401 13:33:00.264466 19644 api.py:461] Built stem stem : (None, 256, 256, 32)
I0401 13:33:02.286643 19644 api.py:461] Block blocks_0 input shape: (None, 256, 256, 32)
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.322646 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.324648 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.328646 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.330646 19644 cross_device_ops.py:565] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
I0401 13:33:02.352647 19644 api.py:461] DWConv shape: (None, 256, 256, 32)
I0401 13:33:02.523651 19644 api.py:461] Built SE se : (None, 1, 1, 32)
I0401 13:33:02.577652 19644 api.py:461] Project shape: (None, 256, 256, 16)
I0401 13:33:02.623652 19644 api.py:461] Block blocks_1 input shape: (None, 256, 256, 16)
I0401 13:33:02.675653 19644 api.py:461] Expand shape: (None, 256, 256, 96)
I0401 13:33:02.725654 19644 api.py:461] DWConv shape: (None, 128, 128, 96)
I0401 13:33:02.762655 19644 api.py:461] Built SE se : (None, 1, 1, 96)
I0401 13:33:02.838654 19644 api.py:461] Project shape: (None, 128, 128, 24)
I0401 13:33:02.854655 19644 api.py:461] Block blocks_2 input shape: (None, 128, 128, 24)
I0401 13:33:02.911658 19644 api.py:461] Expand shape: (None, 128, 128, 144)
I0401 13:33:02.961659 19644 api.py:461] DWConv shape: (None, 128, 128, 144)
I0401 13:33:02.996660 19644 api.py:461] Built SE se : (None, 1, 1, 144)
I0401 13:33:03.046661 19644 api.py:461] Project shape: (None, 128, 128, 24)
I0401 13:33:03.059662 19644 api.py:461] Block blocks_3 input shape: (None, 128, 128, 24)
I0401 13:33:03.110662 19644 api.py:461] Expand shape: (None, 128, 128, 144)
I0401 13:33:03.161663 19644 api.py:461] DWConv shape: (None, 64, 64, 144)
I0401 13:33:03.195664 19644 api.py:461] Built SE se : (None, 1, 1, 144)
I0401 13:33:03.245665 19644 api.py:461] Project shape: (None, 64, 64, 40)
I0401 13:33:03.258665 19644 api.py:461] Block blocks_4 input shape: (None, 64, 64, 40)
I0401 13:33:03.309667 19644 api.py:461] Expand shape: (None, 64, 64, 240)
I0401 13:33:03.360667 19644 api.py:461] DWConv shape: (None, 64, 64, 240)
I0401 13:33:03.396668 19644 api.py:461] Built SE se : (None, 1, 1, 240)
I0401 13:33:03.445669 19644 api.py:461] Project shape: (None, 64, 64, 40)
I0401 13:33:03.458669 19644 api.py:461] Block blocks_5 input shape: (None, 64, 64, 40)
I0401 13:33:03.510671 19644 api.py:461] Expand shape: (None, 64, 64, 240)
I0401 13:33:03.570672 19644 api.py:461] DWConv shape: (None, 32, 32, 240)
I0401 13:33:03.608672 19644 api.py:461] Built SE se : (None, 1, 1, 240)
I0401 13:33:03.659673 19644 api.py:461] Project shape: (None, 32, 32, 80)
I0401 13:33:03.672674 19644 api.py:461] Block blocks_6 input shape: (None, 32, 32, 80)
I0401 13:33:03.723674 19644 api.py:461] Expand shape: (None, 32, 32, 480)
I0401 13:33:03.774676 19644 api.py:461] DWConv shape: (None, 32, 32, 480)
I0401 13:33:03.811676 19644 api.py:461] Built SE se : (None, 1, 1, 480)
I0401 13:33:03.860677 19644 api.py:461] Project shape: (None, 32, 32, 80)
I0401 13:33:03.873677 19644 api.py:461] Block blocks_7 input shape: (None, 32, 32, 80)
I0401 13:33:03.923678 19644 api.py:461] Expand shape: (None, 32, 32, 480)
I0401 13:33:03.973680 19644 api.py:461] DWConv shape: (None, 32, 32, 480)
I0401 13:33:04.007680 19644 api.py:461] Built SE se : (None, 1, 1, 480)
I0401 13:33:04.059682 19644 api.py:461] Project shape: (None, 32, 32, 80)
I0401 13:33:04.079680 19644 api.py:461] Block blocks_8 input shape: (None, 32, 32, 80)
I0401 13:33:04.143683 19644 api.py:461] Expand shape: (None, 32, 32, 480)
I0401 13:33:04.194801 19644 api.py:461] DWConv shape: (None, 32, 32, 480)
I0401 13:33:04.241802 19644 api.py:461] Built SE se : (None, 1, 1, 480)
I0401 13:33:04.303804 19644 api.py:461] Project shape: (None, 32, 32, 112)
I0401 13:33:04.316805 19644 api.py:461] Block blocks_9 input shape: (None, 32, 32, 112)
I0401 13:33:04.366807 19644 api.py:461] Expand shape: (None, 32, 32, 672)
I0401 13:33:04.417807 19644 api.py:461] DWConv shape: (None, 32, 32, 672)
I0401 13:33:04.465809 19644 api.py:461] Built SE se : (None, 1, 1, 672)
I0401 13:33:04.527812 19644 api.py:461] Project shape: (None, 32, 32, 112)
I0401 13:33:04.541812 19644 api.py:461] Block blocks_10 input shape: (None, 32, 32, 112)
I0401 13:33:04.598810 19644 api.py:461] Expand shape: (None, 32, 32, 672)
I0401 13:33:04.652812 19644 api.py:461] DWConv shape: (None, 32, 32, 672)
I0401 13:33:04.687812 19644 api.py:461] Built SE se : (None, 1, 1, 672)
I0401 13:33:04.738812 19644 api.py:461] Project shape: (None, 32, 32, 112)
I0401 13:33:04.751813 19644 api.py:461] Block blocks_11 input shape: (None, 32, 32, 112)
I0401 13:33:04.803815 19644 api.py:461] Expand shape: (None, 32, 32, 672)
I0401 13:33:04.853815 19644 api.py:461] DWConv shape: (None, 16, 16, 672)
I0401 13:33:04.887815 19644 api.py:461] Built SE se : (None, 1, 1, 672)
I0401 13:33:04.937820 19644 api.py:461] Project shape: (None, 16, 16, 192)
I0401 13:33:04.949817 19644 api.py:461] Block blocks_12 input shape: (None, 16, 16, 192)
I0401 13:33:05.001821 19644 api.py:461] Expand shape: (None, 16, 16, 1152)
I0401 13:33:05.053819 19644 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0401 13:33:05.088820 19644 api.py:461] Built SE se : (None, 1, 1, 1152)
I0401 13:33:05.137821 19644 api.py:461] Project shape: (None, 16, 16, 192)
I0401 13:33:05.149821 19644 api.py:461] Block blocks_13 input shape: (None, 16, 16, 192)
I0401 13:33:05.200822 19644 api.py:461] Expand shape: (None, 16, 16, 1152)
I0401 13:33:05.252825 19644 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0401 13:33:05.290823 19644 api.py:461] Built SE se : (None, 1, 1, 1152)
I0401 13:33:05.339825 19644 api.py:461] Project shape: (None, 16, 16, 192)
I0401 13:33:05.352825 19644 api.py:461] Block blocks_14 input shape: (None, 16, 16, 192)
I0401 13:33:05.402827 19644 api.py:461] Expand shape: (None, 16, 16, 1152)
I0401 13:33:05.452827 19644 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0401 13:33:05.496834 19644 api.py:461] Built SE se : (None, 1, 1, 1152)
I0401 13:33:05.562833 19644 api.py:461] Project shape: (None, 16, 16, 192)
I0401 13:33:05.576830 19644 api.py:461] Block blocks_15 input shape: (None, 16, 16, 192)
I0401 13:33:05.633169 19644 api.py:461] Expand shape: (None, 16, 16, 1152)
I0401 13:33:05.684171 19644 api.py:461] DWConv shape: (None, 16, 16, 1152)
I0401 13:33:05.719170 19644 api.py:461] Built SE se : (None, 1, 1, 1152)
I0401 13:33:05.772173 19644 api.py:461] Project shape: (None, 16, 16, 320)
I0401 13:33:10.935725 19644 train_lib.py:218] LR schedule method: cosine
I0401 13:33:10.936728 19644 train_lib.py:296] Use SGD optimizer
I0401 13:33:11.334733 19644 dataloader.py:85] target_size = (512, 512), output_size = (512, 512)
2021-04-01 13:33:12.377972: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/100
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.002767 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.069780 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.143770 5584 api.py:461] Built stem stem : (16, 256, 256, 32)
I0401 13:33:13.155770 5584 api.py:461] Block blocks_0 input shape: (16, 256, 256, 32)
I0401 13:33:13.202770 15184 api.py:461] Built stem stem : (16, 256, 256, 32)
I0401 13:33:13.214773 15184 api.py:461] Block blocks_0 input shape: (16, 256, 256, 32)
I0401 13:33:13.285774 16568 api.py:461] Built stem stem : (16, 256, 256, 32)
I0401 13:33:13.299774 16568 api.py:461] Block blocks_0 input shape: (16, 256, 256, 32)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.323771 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.398787 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.476475 5584 api.py:461] DWConv shape: (16, 256, 256, 32)
I0401 13:33:13.499487 5584 api.py:461] Built SE se : (16, 1, 1, 32)
I0401 13:33:13.554477 15184 api.py:461] DWConv shape: (16, 256, 256, 32)
I0401 13:33:13.577477 15184 api.py:461] Built SE se : (16, 1, 1, 32)
I0401 13:33:13.650478 16568 api.py:461] DWConv shape: (16, 256, 256, 32)
I0401 13:33:13.689482 16568 api.py:461] Built SE se : (16, 1, 1, 32)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.718396 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.811389 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:13.892391 5584 api.py:461] Project shape: (16, 256, 256, 16)
I0401 13:33:13.905393 5584 api.py:461] Block blocks_1 input shape: (16, 256, 256, 16)
I0401 13:33:13.953395 15184 api.py:461] Project shape: (16, 256, 256, 16)
I0401 13:33:13.965397 15184 api.py:461] Block blocks_1 input shape: (16, 256, 256, 16)
I0401 13:33:14.043401 16568 api.py:461] Project shape: (16, 256, 256, 16)
I0401 13:33:14.055398 16568 api.py:461] Block blocks_1 input shape: (16, 256, 256, 16)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.077399 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.184402 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.282399 5584 api.py:461] Expand shape: (16, 256, 256, 96)
I0401 13:33:14.356401 15184 api.py:461] Expand shape: (16, 256, 256, 96)
I0401 13:33:14.409404 16568 api.py:461] Expand shape: (16, 256, 256, 96)
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.431405 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.571408 19644 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = hierarchical_copy, num_packs = 1
I0401 13:33:14.676407 5584 api.py:461] DWConv shape: (16, 128, 128, 96)
I0401 13:33:14.699410 5584 api.py:461] Built SE se : (16, 1, 1, 96)
I0401 13:33:14.756411 15184 api.py:461] DWConv shape: (16, 128, 128, 96)
I0401 13:33:14.777412 15184 api.py:461] Built SE se : (16, 1, 1, 96)
I0401 13:33:14.856413 16568 api.py:461] DWConv shape: (16, 128, 128, 96)
I0401 13:33:14.877413 16568 api.py:461] Built SE se : (16, 1, 1, 96)
I0401 13:33:15.144311 5584 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:15.157311 5584 api.py:461] Block blocks_2 input shape: (16, 128, 128, 24)
I0401 13:33:15.214312 15184 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:15.227313 15184 api.py:461] Block blocks_2 input shape: (16, 128, 128, 24)
I0401 13:33:15.307315 16568 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:15.319317 16568 api.py:461] Block blocks_2 input shape: (16, 128, 128, 24)
I0401 13:33:15.600446 5584 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:15.663447 15184 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:15.750446 16568 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:16.027456 5584 api.py:461] DWConv shape: (16, 128, 128, 144)
I0401 13:33:16.048456 5584 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:16.138456 15184 api.py:461] DWConv shape: (16, 128, 128, 144)
I0401 13:33:16.159457 15184 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:16.250456 16568 api.py:461] DWConv shape: (16, 128, 128, 144)
I0401 13:33:16.273456 16568 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:16.611545 5584 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:16.625548 5584 api.py:461] Block blocks_3 input shape: (16, 128, 128, 24)
I0401 13:33:16.696546 15184 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:16.815549 15184 api.py:461] Block blocks_3 input shape: (16, 128, 128, 24)
I0401 13:33:16.911551 16568 api.py:461] Project shape: (16, 128, 128, 24)
I0401 13:33:16.924551 16568 api.py:461] Block blocks_3 input shape: (16, 128, 128, 24)
I0401 13:33:17.257809 5584 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:17.331811 15184 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:17.430813 16568 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 13:33:17.787820 5584 api.py:461] DWConv shape: (16, 64, 64, 144)
I0401 13:33:17.809821 5584 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:17.888824 15184 api.py:461] DWConv shape: (16, 64, 64, 144)
I0401 13:33:17.909823 15184 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:18.015828 16568 api.py:461] DWConv shape: (16, 64, 64, 144)
I0401 13:33:18.037827 16568 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 13:33:18.408649 5584 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:18.420649 5584 api.py:461] Block blocks_4 input shape: (16, 64, 64, 40)
I0401 13:33:18.504652 15184 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:18.517652 15184 api.py:461] Block blocks_4 input shape: (16, 64, 64, 40)
I0401 13:33:18.618656 16568 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:18.630653 16568 api.py:461] Block blocks_4 input shape: (16, 64, 64, 40)
I0401 13:33:19.016661 5584 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:19.106663 15184 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:19.222663 16568 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:19.616448 5584 api.py:461] DWConv shape: (16, 64, 64, 240)
I0401 13:33:19.638448 5584 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:19.751451 15184 api.py:461] DWConv shape: (16, 64, 64, 240)
I0401 13:33:19.772450 15184 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:19.872453 16568 api.py:461] DWConv shape: (16, 64, 64, 240)
I0401 13:33:19.894454 16568 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:20.365462 5584 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:20.378463 5584 api.py:461] Block blocks_5 input shape: (16, 64, 64, 40)
I0401 13:33:20.503467 15184 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:20.517466 15184 api.py:461] Block blocks_5 input shape: (16, 64, 64, 40)
I0401 13:33:20.654469 16568 api.py:461] Project shape: (16, 64, 64, 40)
I0401 13:33:20.668469 16568 api.py:461] Block blocks_5 input shape: (16, 64, 64, 40)
I0401 13:33:21.145476 5584 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:21.257478 15184 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:21.422487 16568 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 13:33:21.967557 5584 api.py:461] DWConv shape: (16, 32, 32, 240)
I0401 13:33:22.006561 5584 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:22.167561 15184 api.py:461] DWConv shape: (16, 32, 32, 240)
I0401 13:33:22.188561 15184 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:22.334564 16568 api.py:461] DWConv shape: (16, 32, 32, 240)
I0401 13:33:22.356566 16568 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 13:33:22.959574 5584 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:22.971574 5584 api.py:461] Block blocks_6 input shape: (16, 32, 32, 80)
I0401 13:33:23.117581 15184 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:23.129580 15184 api.py:461] Block blocks_6 input shape: (16, 32, 32, 80)
I0401 13:33:23.270585 16568 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:23.291584 16568 api.py:461] Block blocks_6 input shape: (16, 32, 32, 80)
I0401 13:33:23.875598 5584 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:24.055599 15184 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:24.193602 16568 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:24.842617 5584 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:24.864617 5584 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:25.049619 15184 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:25.071619 15184 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:25.240623 16568 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:25.262624 16568 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:25.938634 5584 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:25.951634 5584 api.py:461] Block blocks_7 input shape: (16, 32, 32, 80)
I0401 13:33:26.123642 15184 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:26.144641 15184 api.py:461] Block blocks_7 input shape: (16, 32, 32, 80)
I0401 13:33:26.321645 16568 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:26.334645 16568 api.py:461] Block blocks_7 input shape: (16, 32, 32, 80)
I0401 13:33:27.039657 5584 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:27.221662 15184 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:27.386666 16568 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:28.142679 5584 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:28.163679 5584 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:28.356688 15184 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:28.378686 15184 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:28.563693 16568 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:28.588694 16568 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:29.354706 5584 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:29.367706 5584 api.py:461] Block blocks_8 input shape: (16, 32, 32, 80)
I0401 13:33:29.544710 15184 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:29.557710 15184 api.py:461] Block blocks_8 input shape: (16, 32, 32, 80)
I0401 13:33:29.792715 16568 api.py:461] Project shape: (16, 32, 32, 80)
I0401 13:33:29.804715 16568 api.py:461] Block blocks_8 input shape: (16, 32, 32, 80)
I0401 13:33:30.620502 5584 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:30.840506 15184 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:31.069518 16568 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 13:33:31.927490 5584 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:31.949491 5584 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:32.193496 15184 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:32.214496 15184 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:32.455504 16568 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 13:33:32.479505 16568 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 13:33:33.374524 5584 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:33.387524 5584 api.py:461] Block blocks_9 input shape: (16, 32, 32, 112)
I0401 13:33:33.626529 15184 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:33.639529 15184 api.py:461] Block blocks_9 input shape: (16, 32, 32, 112)
I0401 13:33:33.858532 16568 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:33.870533 16568 api.py:461] Block blocks_9 input shape: (16, 32, 32, 112)
I0401 13:33:34.821553 5584 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:35.067557 15184 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:35.341562 16568 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:36.334579 5584 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:36.356580 5584 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:36.600589 15184 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:36.623590 15184 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:36.896599 16568 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:36.918594 16568 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:37.980613 5584 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:37.993613 5584 api.py:461] Block blocks_10 input shape: (16, 32, 32, 112)
I0401 13:33:38.286622 15184 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:38.299623 15184 api.py:461] Block blocks_10 input shape: (16, 32, 32, 112)
I0401 13:33:38.579628 16568 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:38.717630 16568 api.py:461] Block blocks_10 input shape: (16, 32, 32, 112)
I0401 13:33:39.862655 5584 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:40.169661 15184 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:40.435666 16568 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:41.644690 5584 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:41.668691 5584 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:41.990698 15184 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:42.012701 15184 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:42.314705 16568 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 13:33:42.335704 16568 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:43.568892 5584 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:43.582893 5584 api.py:461] Block blocks_11 input shape: (16, 32, 32, 112)
I0401 13:33:43.894900 15184 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:43.907900 15184 api.py:461] Block blocks_11 input shape: (16, 32, 32, 112)
I0401 13:33:44.222905 16568 api.py:461] Project shape: (16, 32, 32, 112)
I0401 13:33:44.235906 16568 api.py:461] Block blocks_11 input shape: (16, 32, 32, 112)
I0401 13:33:45.598932 5584 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:45.925939 15184 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:46.265945 16568 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 13:33:47.628980 5584 api.py:461] DWConv shape: (16, 16, 16, 672)
I0401 13:33:47.651974 5584 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:47.991981 15184 api.py:461] DWConv shape: (16, 16, 16, 672)
I0401 13:33:48.013982 15184 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:48.381986 16568 api.py:461] DWConv shape: (16, 16, 16, 672)
I0401 13:33:48.403987 16568 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 13:33:49.867761 5584 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:49.879762 5584 api.py:461] Block blocks_12 input shape: (16, 16, 16, 192)
I0401 13:33:50.239769 15184 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:50.251769 15184 api.py:461] Block blocks_12 input shape: (16, 16, 16, 192)
I0401 13:33:50.591777 16568 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:50.604779 16568 api.py:461] Block blocks_12 input shape: (16, 16, 16, 192)
I0401 13:33:52.058806 5584 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:33:52.412813 15184 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:33:52.801823 16568 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:33:54.358854 5584 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:33:54.380852 5584 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:33:54.779860 15184 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:33:54.800862 15184 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:33:55.233870 16568 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:33:55.255873 16568 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:33:56.933907 5584 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:56.945905 5584 api.py:461] Block blocks_13 input shape: (16, 16, 16, 192)
I0401 13:33:57.329912 15184 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:57.342914 15184 api.py:461] Block blocks_13 input shape: (16, 16, 16, 192)
I0401 13:33:57.790924 16568 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:33:57.802922 16568 api.py:461] Block blocks_13 input shape: (16, 16, 16, 192)
I0401 13:33:59.529071 5584 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:33:59.965076 15184 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:00.406087 16568 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:02.171123 5584 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:02.192123 5584 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:02.642130 15184 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:02.666131 15184 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:03.150315 16568 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:03.173315 16568 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:05.084357 5584 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:05.096357 5584 api.py:461] Block blocks_14 input shape: (16, 16, 16, 192)
I0401 13:34:05.540365 15184 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:05.552365 15184 api.py:461] Block blocks_14 input shape: (16, 16, 16, 192)
I0401 13:34:06.076376 16568 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:06.088376 16568 api.py:461] Block blocks_14 input shape: (16, 16, 16, 192)
I0401 13:34:08.042416 5584 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:08.531427 15184 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:09.074437 16568 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:11.080477 5584 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:11.102478 5584 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:11.606487 15184 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:11.628488 15184 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:12.192500 16568 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:12.214500 16568 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:14.278542 5584 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:14.290542 5584 api.py:461] Block blocks_15 input shape: (16, 16, 16, 192)
I0401 13:34:14.802552 15184 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:14.815554 15184 api.py:461] Block blocks_15 input shape: (16, 16, 16, 192)
I0401 13:34:15.389564 16568 api.py:461] Project shape: (16, 16, 16, 192)
I0401 13:34:15.402565 16568 api.py:461] Block blocks_15 input shape: (16, 16, 16, 192)
I0401 13:34:17.565609 5584 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:18.096620 15184 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:18.679629 16568 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 13:34:20.964677 5584 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:20.986677 5584 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:21.531689 15184 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:21.552688 15184 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:22.151701 16568 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 13:34:22.174701 16568 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 13:34:24.511750 5584 api.py:461] Project shape: (16, 16, 16, 320)
I0401 13:34:25.103761 15184 api.py:461] Project shape: (16, 16, 16, 320)
I0401 13:34:25.726771 16568 api.py:461] Project shape: (16, 16, 16, 320)
I0401 15:38:59.010392 16916 api.py:461] Built stem stem : (16, 256, 256, 32)
I0401 15:38:59.024394 16916 api.py:461] Block blocks_0 input shape: (16, 256, 256, 32)
I0401 15:38:59.093078 17900 api.py:461] Built stem stem : (16, 256, 256, 32)
I0401 15:38:59.106080 17900 api.py:461] Block blocks_0 input shape: (16, 256, 256, 32)
I0401 15:38:59.158078 22420 api.py:461] Built stem stem : (16, 256, 256, 32)
I0401 15:38:59.171082 22420 api.py:461] Block blocks_0 input shape: (16, 256, 256, 32)
I0401 15:38:59.329978 16916 api.py:461] DWConv shape: (16, 256, 256, 32)
I0401 15:38:59.354979 16916 api.py:461] Built SE se : (16, 1, 1, 32)
I0401 15:38:59.421983 17900 api.py:461] DWConv shape: (16, 256, 256, 32)
I0401 15:38:59.446986 17900 api.py:461] Built SE se : (16, 1, 1, 32)
I0401 15:38:59.517992 22420 api.py:461] DWConv shape: (16, 256, 256, 32)
I0401 15:38:59.551992 22420 api.py:461] Built SE se : (16, 1, 1, 32)
I0401 15:38:59.728790 16916 api.py:461] Project shape: (16, 256, 256, 16)
I0401 15:38:59.741792 16916 api.py:461] Block blocks_1 input shape: (16, 256, 256, 16)
I0401 15:38:59.794792 17900 api.py:461] Project shape: (16, 256, 256, 16)
I0401 15:38:59.808793 17900 api.py:461] Block blocks_1 input shape: (16, 256, 256, 16)
I0401 15:38:59.866796 22420 api.py:461] Project shape: (16, 256, 256, 16)
I0401 15:38:59.880795 22420 api.py:461] Block blocks_1 input shape: (16, 256, 256, 16)
I0401 15:39:00.054802 16916 api.py:461] Expand shape: (16, 256, 256, 96)
I0401 15:39:00.133805 17900 api.py:461] Expand shape: (16, 256, 256, 96)
I0401 15:39:00.213809 22420 api.py:461] Expand shape: (16, 256, 256, 96)
I0401 15:39:00.410519 16916 api.py:461] DWConv shape: (16, 128, 128, 96)
I0401 15:39:00.434519 16916 api.py:461] Built SE se : (16, 1, 1, 96)
I0401 15:39:00.496523 17900 api.py:461] DWConv shape: (16, 128, 128, 96)
I0401 15:39:00.522521 17900 api.py:461] Built SE se : (16, 1, 1, 96)
I0401 15:39:00.592525 22420 api.py:461] DWConv shape: (16, 128, 128, 96)
I0401 15:39:00.620528 22420 api.py:461] Built SE se : (16, 1, 1, 96)
I0401 15:39:00.844985 16916 api.py:461] Project shape: (16, 128, 128, 24)
I0401 15:39:00.858987 16916 api.py:461] Block blocks_2 input shape: (16, 128, 128, 24)
I0401 15:39:00.924990 17900 api.py:461] Project shape: (16, 128, 128, 24)
I0401 15:39:00.940989 17900 api.py:461] Block blocks_2 input shape: (16, 128, 128, 24)
I0401 15:39:01.022993 22420 api.py:461] Project shape: (16, 128, 128, 24)
I0401 15:39:01.035991 22420 api.py:461] Block blocks_2 input shape: (16, 128, 128, 24)
I0401 15:39:01.322329 16916 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 15:39:01.409332 17900 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 15:39:01.503993 22420 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 15:39:01.777566 16916 api.py:461] DWConv shape: (16, 128, 128, 144)
I0401 15:39:01.800569 16916 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 15:39:01.894571 17900 api.py:461] DWConv shape: (16, 128, 128, 144)
I0401 15:39:01.918572 17900 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 15:39:02.009575 22420 api.py:461] DWConv shape: (16, 128, 128, 144)
I0401 15:39:02.032576 22420 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 15:39:02.352696 16916 api.py:461] Project shape: (16, 128, 128, 24)
I0401 15:39:02.366694 16916 api.py:461] Block blocks_3 input shape: (16, 128, 128, 24)
I0401 15:39:02.440698 17900 api.py:461] Project shape: (16, 128, 128, 24)
I0401 15:39:02.454698 17900 api.py:461] Block blocks_3 input shape: (16, 128, 128, 24)
I0401 15:39:02.557702 22420 api.py:461] Project shape: (16, 128, 128, 24)
I0401 15:39:02.582701 22420 api.py:461] Block blocks_3 input shape: (16, 128, 128, 24)
I0401 15:39:02.902499 16916 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 15:39:02.987502 17900 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 15:39:03.077502 22420 api.py:461] Expand shape: (16, 128, 128, 144)
I0401 15:39:03.443333 16916 api.py:461] DWConv shape: (16, 64, 64, 144)
I0401 15:39:03.468336 16916 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 15:39:03.562338 17900 api.py:461] DWConv shape: (16, 64, 64, 144)
I0401 15:39:03.589344 17900 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 15:39:03.719345 22420 api.py:461] DWConv shape: (16, 64, 64, 144)
I0401 15:39:03.743347 22420 api.py:461] Built SE se : (16, 1, 1, 144)
I0401 15:39:04.147769 16916 api.py:461] Project shape: (16, 64, 64, 40)
I0401 15:39:04.162770 16916 api.py:461] Block blocks_4 input shape: (16, 64, 64, 40)
I0401 15:39:04.252776 17900 api.py:461] Project shape: (16, 64, 64, 40)
I0401 15:39:04.266772 17900 api.py:461] Block blocks_4 input shape: (16, 64, 64, 40)
I0401 15:39:04.381554 22420 api.py:461] Project shape: (16, 64, 64, 40)
I0401 15:39:04.395572 22420 api.py:461] Block blocks_4 input shape: (16, 64, 64, 40)
I0401 15:39:04.819159 16916 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 15:39:04.923162 17900 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 15:39:05.064167 22420 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 15:39:05.503554 16916 api.py:461] DWConv shape: (16, 64, 64, 240)
I0401 15:39:05.528555 16916 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 15:39:05.674564 17900 api.py:461] DWConv shape: (16, 64, 64, 240)
I0401 15:39:05.697562 17900 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 15:39:05.832540 22420 api.py:461] DWConv shape: (16, 64, 64, 240)
I0401 15:39:05.857541 22420 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 15:39:06.306557 16916 api.py:461] Project shape: (16, 64, 64, 40)
I0401 15:39:06.321309 16916 api.py:461] Block blocks_5 input shape: (16, 64, 64, 40)
I0401 15:39:06.433740 17900 api.py:461] Project shape: (16, 64, 64, 40)
I0401 15:39:06.447739 17900 api.py:461] Block blocks_5 input shape: (16, 64, 64, 40)
I0401 15:39:06.600376 22420 api.py:461] Project shape: (16, 64, 64, 40)
I0401 15:39:06.615373 22420 api.py:461] Block blocks_5 input shape: (16, 64, 64, 40)
I0401 15:39:07.109296 16916 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 15:39:07.250299 17900 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 15:39:07.394169 22420 api.py:461] Expand shape: (16, 64, 64, 240)
I0401 15:39:07.936873 16916 api.py:461] DWConv shape: (16, 32, 32, 240)
I0401 15:39:07.960874 16916 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 15:39:08.131876 17900 api.py:461] DWConv shape: (16, 32, 32, 240)
I0401 15:39:08.155878 17900 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 15:39:08.307885 22420 api.py:461] DWConv shape: (16, 32, 32, 240)
I0401 15:39:08.331942 22420 api.py:461] Built SE se : (16, 1, 1, 240)
I0401 15:39:08.869233 16916 api.py:461] Project shape: (16, 32, 32, 80)
I0401 15:39:08.882894 16916 api.py:461] Block blocks_6 input shape: (16, 32, 32, 80)
I0401 15:39:09.020048 17900 api.py:461] Project shape: (16, 32, 32, 80)
I0401 15:39:09.033916 17900 api.py:461] Block blocks_6 input shape: (16, 32, 32, 80)
I0401 15:39:09.172925 22420 api.py:461] Project shape: (16, 32, 32, 80)
I0401 15:39:09.185925 22420 api.py:461] Block blocks_6 input shape: (16, 32, 32, 80)
I0401 15:39:09.749299 16916 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 15:39:09.929079 17900 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 15:39:10.119951 22420 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 15:39:10.715917 16916 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 15:39:10.739947 16916 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 15:39:10.927558 17900 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 15:39:10.951558 17900 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 15:39:11.152567 22420 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 15:39:11.176567 22420 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 15:39:11.877509 16916 api.py:461] Project shape: (16, 32, 32, 80)
I0401 15:39:11.892511 16916 api.py:461] Block blocks_7 input shape: (16, 32, 32, 80)
I0401 15:39:12.071060 17900 api.py:461] Project shape: (16, 32, 32, 80)
I0401 15:39:12.085063 17900 api.py:461] Block blocks_7 input shape: (16, 32, 32, 80)
I0401 15:39:12.272069 22420 api.py:461] Project shape: (16, 32, 32, 80)
I0401 15:39:12.286069 22420 api.py:461] Block blocks_7 input shape: (16, 32, 32, 80)
I0401 15:39:12.992987 16916 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 15:39:13.172992 17900 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 15:39:13.373000 22420 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 15:39:14.149003 16916 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 15:39:14.173004 16916 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 15:39:14.383013 17900 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 15:39:14.406014 17900 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 15:39:14.619016 22420 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 15:39:14.656013 22420 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 15:39:15.502216 16916 api.py:461] Project shape: (16, 32, 32, 80)
I0401 15:39:15.516217 16916 api.py:461] Block blocks_8 input shape: (16, 32, 32, 80)
I0401 15:39:15.727767 17900 api.py:461] Project shape: (16, 32, 32, 80)
I0401 15:39:15.741768 17900 api.py:461] Block blocks_8 input shape: (16, 32, 32, 80)
I0401 15:39:15.965816 22420 api.py:461] Project shape: (16, 32, 32, 80)
I0401 15:39:15.981335 22420 api.py:461] Block blocks_8 input shape: (16, 32, 32, 80)
I0401 15:39:16.803018 16916 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 15:39:17.056537 17900 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 15:39:17.299546 22420 api.py:461] Expand shape: (16, 32, 32, 480)
I0401 15:39:18.231281 16916 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 15:39:18.257286 16916 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 15:39:18.522499 17900 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 15:39:18.546128 17900 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 15:39:18.795136 22420 api.py:461] DWConv shape: (16, 32, 32, 480)
I0401 15:39:18.819141 22420 api.py:461] Built SE se : (16, 1, 1, 480)
I0401 15:39:19.751963 16916 api.py:461] Project shape: (16, 32, 32, 112)
I0401 15:39:19.766966 16916 api.py:461] Block blocks_9 input shape: (16, 32, 32, 112)
I0401 15:39:20.006977 17900 api.py:461] Project shape: (16, 32, 32, 112)
I0401 15:39:20.021978 17900 api.py:461] Block blocks_9 input shape: (16, 32, 32, 112)
I0401 15:39:20.275798 22420 api.py:461] Project shape: (16, 32, 32, 112)
I0401 15:39:20.289806 22420 api.py:461] Block blocks_9 input shape: (16, 32, 32, 112)
I0401 15:39:21.260990 16916 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 15:39:21.508632 17900 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 15:39:21.787514 22420 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 15:39:22.824112 16916 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 15:39:22.849112 16916 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 15:39:23.109702 17900 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 15:39:23.144701 17900 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 15:39:23.419717 22420 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 15:39:23.442718 22420 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 15:39:24.619537 16916 api.py:461] Project shape: (16, 32, 32, 112)
I0401 15:39:24.643168 16916 api.py:461] Block blocks_10 input shape: (16, 32, 32, 112)
I0401 15:39:24.935180 17900 api.py:461] Project shape: (16, 32, 32, 112)
I0401 15:39:24.949185 17900 api.py:461] Block blocks_10 input shape: (16, 32, 32, 112)
I0401 15:39:25.246673 22420 api.py:461] Project shape: (16, 32, 32, 112)
I0401 15:39:25.260672 22420 api.py:461] Block blocks_10 input shape: (16, 32, 32, 112)
I0401 15:39:26.404814 16916 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 15:39:26.714298 17900 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 15:39:27.025310 22420 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 15:39:28.265676 16916 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 15:39:28.289678 16916 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 15:39:28.589688 17900 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 15:39:28.614690 17900 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 15:39:28.938765 22420 api.py:461] DWConv shape: (16, 32, 32, 672)
I0401 15:39:28.963766 22420 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 15:39:30.262865 16916 api.py:461] Project shape: (16, 32, 32, 112)
I0401 15:39:30.276865 16916 api.py:461] Block blocks_11 input shape: (16, 32, 32, 112)
I0401 15:39:30.624887 17900 api.py:461] Project shape: (16, 32, 32, 112)
I0401 15:39:30.639887 17900 api.py:461] Block blocks_11 input shape: (16, 32, 32, 112)
I0401 15:39:30.992428 22420 api.py:461] Project shape: (16, 32, 32, 112)
I0401 15:39:31.008429 22420 api.py:461] Block blocks_11 input shape: (16, 32, 32, 112)
I0401 15:39:32.422469 16916 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 15:39:32.770440 17900 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 15:39:33.134454 22420 api.py:461] Expand shape: (16, 32, 32, 672)
I0401 15:39:34.529197 16916 api.py:461] DWConv shape: (16, 16, 16, 672)
I0401 15:39:34.553198 16916 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 15:39:34.964609 17900 api.py:461] DWConv shape: (16, 16, 16, 672)
I0401 15:39:34.988609 17900 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 15:39:35.377516 22420 api.py:461] DWConv shape: (16, 16, 16, 672)
I0401 15:39:35.400516 22420 api.py:461] Built SE se : (16, 1, 1, 672)
I0401 15:39:36.892176 16916 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:39:36.906173 16916 api.py:461] Block blocks_12 input shape: (16, 16, 16, 192)
I0401 15:39:37.317821 17900 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:39:37.331824 17900 api.py:461] Block blocks_12 input shape: (16, 16, 16, 192)
I0401 15:39:37.709830 22420 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:39:37.724829 22420 api.py:461] Block blocks_12 input shape: (16, 16, 16, 192)
I0401 15:39:39.327305 16916 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:39:39.762373 17900 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:39:40.198852 22420 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:39:41.867616 16916 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:39:41.891792 16916 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:39:42.332813 17900 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:39:42.356814 17900 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:39:42.798847 22420 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:39:42.822847 22420 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:39:44.530074 16916 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:39:44.544077 16916 api.py:461] Block blocks_13 input shape: (16, 16, 16, 192)
I0401 15:39:44.979925 17900 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:39:44.993924 17900 api.py:461] Block blocks_13 input shape: (16, 16, 16, 192)
I0401 15:39:45.460961 22420 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:39:45.474961 22420 api.py:461] Block blocks_13 input shape: (16, 16, 16, 192)
I0401 15:39:47.309737 16916 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:39:47.773467 17900 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:39:48.288810 22420 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:39:50.130834 16916 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:39:50.170835 16916 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:39:50.651040 17900 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:39:50.677036 17900 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:39:51.220761 22420 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:39:51.246762 22420 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:39:53.414636 16916 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:39:53.428636 16916 api.py:461] Block blocks_14 input shape: (16, 16, 16, 192)
I0401 15:39:53.932619 17900 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:39:53.947630 17900 api.py:461] Block blocks_14 input shape: (16, 16, 16, 192)
I0401 15:39:54.996667 22420 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:39:55.011666 22420 api.py:461] Block blocks_14 input shape: (16, 16, 16, 192)
I0401 15:39:57.128908 16916 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:39:57.632880 17900 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:39:58.186950 22420 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:40:00.312306 16916 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:40:00.336311 16916 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:40:00.883792 17900 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:40:00.908785 17900 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:40:01.505554 22420 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:40:01.530238 22420 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:40:03.810959 16916 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:40:03.824958 16916 api.py:461] Block blocks_15 input shape: (16, 16, 16, 192)
I0401 15:40:04.410109 17900 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:40:04.424109 17900 api.py:461] Block blocks_15 input shape: (16, 16, 16, 192)
I0401 15:40:05.031138 22420 api.py:461] Project shape: (16, 16, 16, 192)
I0401 15:40:05.045139 22420 api.py:461] Block blocks_15 input shape: (16, 16, 16, 192)
I0401 15:40:07.372060 16916 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:40:07.993764 17900 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:40:08.643059 22420 api.py:461] Expand shape: (16, 16, 16, 1152)
I0401 15:40:11.131124 16916 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:40:11.156115 16916 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:40:11.785039 17900 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:40:11.809991 17900 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:40:12.476264 22420 api.py:461] DWConv shape: (16, 16, 16, 1152)
I0401 15:40:12.500264 22420 api.py:461] Built SE se : (16, 1, 1, 1152)
I0401 15:40:15.115189 16916 api.py:461] Project shape: (16, 16, 16, 320)
I0401 15:40:15.778236 17900 api.py:461] Project shape: (16, 16, 16, 320)
I0401 15:40:16.493865 22420 api.py:461] Project shape: (16, 16, 16, 320)
Traceback (most recent call last):
File "C:/Users/ha485/PycharmProjects/Official4/efficientdet/keras/train.py", line 280, in
Function call stack: fn_with_cond -> fn_with_cond -> fn_with_cond
Process finished with exit code 1
fsx950223
commented
3 years ago max_instances_per_image: 300
Ronald-Kray
commented
3 years ago @fsx950223 I'm training EfficientDet-D0 for 100epoch. 48 batch size. Currently, it is 91 epochs It takes almost 6days. Is it normal?
nikhilparmar
commented
2 years ago
max_instances_per_image: 300
Hi, I am also facing the same issue with instances count using the tflite model maker on EfficientDet. How to set this max_instances_per_image: 300 ? I am a newbie. Kindly help thanks
I training on custom data set based on Window 10. I don't know what happened in the training. Anyone can tell me about this?
2021-03-30 22:53:06.333435: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll I0330 22:53:10.040188 8636 main.py:264] {'name': 'efficientdet-d0', 'act_type': 'swish', 'image_size': (512, 512), 'target_size': None, 'input_rand_hflip': False, 'jitter_min': 1.0, 'jitter_max': 1.0, 'autoaugment_policy': None, 'grid_mask': False, 'sample_image': None, 'map_freq': 5, 'num_classes': 2, 'seg_num_classes': 3, 'heads': ['object_detection'], 'skip_crowd_during_training': True, 'label_map': {1: 'wheat_head'}, 'max_instances_per_image': 200, 'regenerate_source_id': False, 'min_level': 3, 'max_level': 7, 'num_scales': 3, 'aspect_ratios': [1.0, 2.0, 0.5], 'anchor_scale': 4, 'is_training_bn': True, 'momentum': 0.9, 'optimizer': 'adam', 'learning_rate': 0.008, 'lr_warmup_init': 0.0008, 'lr_warmup_epoch': 1.0, 'first_lr_drop_epoch': 100, 'second_lr_drop_epoch': 150, 'poly_lr_power': 0.9, 'clip_gradients_norm': 10.0, 'num_epochs': 500, 'data_format': 'channels_last', 'mean_rgb': [123.675, 116.28, 103.53], 'stddev_rgb': [58.395, 57.120000000000005, 57.375], 'label_smoothing': 0.0, 'alpha': 0.25, 'gamma': 1.5, 'delta': 0.1, 'box_loss_weight': 50.0, 'iou_loss_type': None, 'iou_loss_weight': 1.0, 'weight_decay': 4e-05, 'strategy': 'gpus', 'mixed_precision': True, 'loss_scale': None, 'model_optimizations': {}, 'box_class_repeats': 3, 'fpn_cell_repeats': 3, 'fpn_num_filters': 64, 'separable_conv': True, 'apply_bn_for_resampling': True, 'conv_after_downsample': False, 'conv_bn_act_pattern': False, 'drop_remainder': True, 'nms_configs': {'method': 'gaussian', 'iou_thresh': None, 'score_thresh': 0.0, 'sigma': None, 'pyfunc': False, 'max_nms_inputs': 0, 'max_output_size': 400}, 'tflite_max_detections': 100, 'fpn_name': None, 'fpn_weight_method': None, 'fpn_config': None, 'survival_prob': None, 'img_summary_steps': None, 'lr_decay_method': 'cosine', 'moving_average_decay': 0.9998, 'ckpt_var_scope': None, 'skip_mismatch': True, 'backbone_name': 'efficientnet-b0', 'backbone_config': None, 'var_freeze_expr': '(efficientnet|fpn_cells|resample_p6)', 'use_keras_model': True, 'dataset_type': None, 'positives_momentum': None, 'grad_checkpoint': True, 'model_name': 'efficientdet-d0', 'iterations_per_loop': 100, 'model_dir': 'efficientdet/output', 'num_shards': 8, 'num_examples_per_epoch': 120000, 'backbone_ckpt': '', 'ckpt': None, 'val_json_file': None, 'testdev_dir': None, 'profile': True, 'mode': 'train'} 2021-03-30 22:53:10.040833: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-30 22:53:10.041786: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll 2021-03-30 22:53:10.151856: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 22:53:10.152097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 22:53:10.152328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 22:53:10.152547: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-03-30 22:53:10.164560: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-03-30 22:53:10.164683: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-03-30 22:53:10.168937: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-03-30 22:53:10.170749: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-03-30 22:53:10.180667: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-03-30 22:53:10.183902: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-03-30 22:53:10.185030: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-03-30 22:53:10.185319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2 2021-03-30 22:53:10.186492: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-03-30 22:53:10.785507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 22:53:10.785788: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 22:53:10.786057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 22:53:10.786309: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-03-30 22:53:10.786438: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-03-30 22:53:10.786563: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-03-30 22:53:10.786688: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-03-30 22:53:10.786811: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-03-30 22:53:10.786934: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-03-30 22:53:10.787067: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-03-30 22:53:10.787190: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-03-30 22:53:10.787376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2 2021-03-30 22:53:12.459250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-30 22:53:12.459387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2 2021-03-30 22:53:12.459462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N N N 2021-03-30 22:53:12.459538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: N N N 2021-03-30 22:53:12.459611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: N N N 2021-03-30 22:53:12.459932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9417 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5) 2021-03-30 22:53:12.461823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9417 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5) 2021-03-30 22:53:12.463243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9417 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5) 2021-03-30 22:53:12.464383: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-30 22:53:12.468311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 22:53:12.468563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 22:53:12.468809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 22:53:12.469044: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-03-30 22:53:12.469161: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-03-30 22:53:12.469275: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-03-30 22:53:12.469391: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-03-30 22:53:12.469505: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-03-30 22:53:12.469633: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-03-30 22:53:12.469752: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-03-30 22:53:12.469871: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-03-30 22:53:12.470041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2 2021-03-30 22:53:12.470288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-30 22:53:12.470415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2 2021-03-30 22:53:12.470495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N N N 2021-03-30 22:53:12.470605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: N N N 2021-03-30 22:53:12.470684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: N N N 2021-03-30 22:53:12.470913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/device:GPU:0 with 9417 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5) 2021-03-30 22:53:12.471142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/device:GPU:1 with 9417 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5) 2021-03-30 22:53:12.471370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/device:GPU:2 with 9417 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5) 2021-03-30 22:53:12.478751: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set INFO:tensorflow:Using MirroredStrategy with devices ('/replica:0/task:0/device:GPU:0', '/replica:0/task:0/device:GPU:1', '/replica:0/task:0/device:GPU:2') I0330 22:53:12.479525 8636 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/replica:0/task:0/device:GPU:0', '/replica:0/task:0/device:GPU:1', '/replica:0/task:0/device:GPU:2') INFO:tensorflow:Initializing RunConfig with distribution strategies. I0330 22:53:12.611532 8636 run_config.py:584] Initializing RunConfig with distribution strategies. INFO:tensorflow:Not using Distribute Coordinator. I0330 22:53:12.612533 8636 estimator_training.py:167] Not using Distribute Coordinator. INFO:tensorflow:Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x0000014453763160>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} I0330 22:53:12.612533 8636 estimator.py:191] Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x0000014453763160>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} INFO:tensorflow:Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x0000014453763160>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} I0330 22:53:12.613531 8636 estimator.py:191] Using config: {'_model_dir': 'efficientdet/output', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.python.distribute.mirrored_strategy.MirroredStrategyV1 object at 0x0000014453763160>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None} INFO:tensorflow:The
input_fnaccepts aninput_contextwhich will be given by DistributionStrategy I0330 22:53:12.619532 8636 estimator.py:1126] Theinput_fnaccepts aninput_contextwhich will be given by DistributionStrategy I0330 22:53:12.865545 8636 dataloader.py:85] target_size = (512, 512), output_size = (512, 512) INFO:tensorflow:Calling model_fn. I0330 22:53:13.863601 12768 api.py:479] Calling model_fn. I0330 22:53:13.907603 12768 utils.py:595] use mixed precision policy name mixed_float16 WARNING:tensorflow:From C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:596: The name tf.keras.layers.enable_v2_dtype_behavior is deprecated. Please use tf.compat.v1.keras.layers.enable_v2_dtype_behavior instead.W0330 22:53:13.907603 12768 module_wrapper.py:138] From C:\Users\ha485\PycharmProjects\Official3\efficientdet\utils.py:596: The name tf.keras.layers.enable_v2_dtype_behavior is deprecated. Please use tf.compat.v1.keras.layers.enable_v2_dtype_behavior instead.
2021-03-30 22:53:13.908210: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-30 22:53:13.908455: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-30 22:53:13.908612: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set INFO:tensorflow:Mixed precision compatibility check (mixed_float16): OK Your GPUs will likely run quickly with dtype policy mixed_float16 as they all have compute capability of at least 7.0 I0330 22:53:13.908603 12768 device_compatibility_check.py:129] Mixed precision compatibility check (mixed_float16): OK Your GPUs will likely run quickly with dtype policy mixed_float16 as they all have compute capability of at least 7.0 I0330 22:53:13.916605 12768 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x0000014452A7C840>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=True) I0330 22:53:14.280624 12768 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0330 22:53:14.281623 12768 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0330 22:53:14.282623 12768 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0330 22:53:14.283624 12768 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0330 22:53:14.283624 12768 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0330 22:53:14.284625 12768 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0330 22:53:14.285623 12768 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0330 22:53:14.286623 12768 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0330 22:53:14.287625 12768 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0330 22:53:14.288625 12768 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0330 22:53:14.289627 12768 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0330 22:53:14.290624 12768 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0330 22:53:14.291623 12768 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0330 22:53:14.292623 12768 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0330 22:53:14.292623 12768 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0330 22:53:14.293624 12768 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0330 22:53:14.295625 12768 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0330 22:53:14.298624 12768 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0330 22:53:14.300625 12768 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0330 22:53:14.303626 12768 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0330 22:53:14.305628 12768 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0330 22:53:14.308626 12768 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0330 22:53:14.311626 12768 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0330 22:53:14.313625 12768 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} INFO:tensorflow:Calling model_fn. I0330 22:53:14.496645 14952 api.py:479] Calling model_fn. I0330 22:53:14.498648 14952 utils.py:595] use mixed precision policy name mixed_float16 I0330 22:53:14.508637 14952 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x0000014452A7C840>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=True) I0330 22:53:14.866657 14952 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0330 22:53:14.867657 14952 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0330 22:53:14.868656 14952 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0330 22:53:14.868656 14952 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0330 22:53:14.869656 14952 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0330 22:53:14.870656 14952 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0330 22:53:14.871659 14952 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0330 22:53:14.872656 14952 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0330 22:53:14.873656 14952 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0330 22:53:14.874657 14952 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0330 22:53:14.875657 14952 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0330 22:53:14.876656 14952 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0330 22:53:14.877657 14952 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0330 22:53:14.878657 14952 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0330 22:53:14.879658 14952 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0330 22:53:14.880656 14952 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0330 22:53:14.881656 14952 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0330 22:53:14.882657 14952 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0330 22:53:14.883658 14952 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0330 22:53:14.884656 14952 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0330 22:53:14.884656 14952 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0330 22:53:14.885657 14952 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0330 22:53:14.886658 14952 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0330 22:53:14.887660 14952 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} INFO:tensorflow:Calling model_fn. I0330 22:53:14.994664 10288 api.py:479] Calling model_fn. I0330 22:53:14.996665 10288 utils.py:595] use mixed precision policy name mixed_float16 I0330 22:53:15.006665 10288 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x0000014452A7C840>, act_type='swish'), batch_norm=<class 'utils.SyncBatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=True) I0330 22:53:15.345682 10288 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0330 22:53:15.346683 10288 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0330 22:53:15.347683 10288 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0330 22:53:15.348682 10288 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0330 22:53:15.349682 10288 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0330 22:53:15.350683 10288 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0330 22:53:15.350683 10288 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0330 22:53:15.351684 10288 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0330 22:53:15.353682 10288 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0330 22:53:15.353682 10288 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0330 22:53:15.354683 10288 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0330 22:53:15.355684 10288 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0330 22:53:15.356685 10288 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0330 22:53:15.357686 10288 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0330 22:53:15.358683 10288 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0330 22:53:15.359685 10288 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} I0330 22:53:15.360684 10288 efficientdet_keras.py:760] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]} I0330 22:53:15.361684 10288 efficientdet_keras.py:760] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]} I0330 22:53:15.362683 10288 efficientdet_keras.py:760] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]} I0330 22:53:15.363683 10288 efficientdet_keras.py:760] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]} I0330 22:53:15.364684 10288 efficientdet_keras.py:760] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]} I0330 22:53:15.365684 10288 efficientdet_keras.py:760] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]} I0330 22:53:15.365684 10288 efficientdet_keras.py:760] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]} I0330 22:53:15.366683 10288 efficientdet_keras.py:760] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]} INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:15.444687 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:15.505589 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:15.566602 12768 efficientnet_model.py:735] Built stem stem : (21, 256, 256, 32) I0330 22:53:15.566602 12768 efficientnet_model.py:374] Block blocks_0 input shape: (21, 256, 256, 32) I0330 22:53:15.633604 14952 efficientnet_model.py:735] Built stem stem_1 : (21, 256, 256, 32) I0330 22:53:15.634605 14952 efficientnet_model.py:374] Block blocks_0 input shape: (21, 256, 256, 32) I0330 22:53:15.683604 10288 efficientnet_model.py:735] Built stem stem_2 : (21, 256, 256, 32) I0330 22:53:15.685605 10288 efficientnet_model.py:374] Block blocks_0 input shape: (21, 256, 256, 32) INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:15.706609 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:15.781659 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:15.871713 12768 efficientnet_model.py:393] DWConv shape: (21, 256, 256, 32) I0330 22:53:15.927664 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 32) I0330 22:53:16.015670 14952 efficientnet_model.py:393] DWConv shape: (21, 256, 256, 32) I0330 22:53:16.046671 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 32) I0330 22:53:16.083668 10288 efficientnet_model.py:393] DWConv shape: (21, 256, 256, 32) I0330 22:53:16.097673 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 32) INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:16.109674 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:16.159672 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:16.215900 12768 efficientnet_model.py:414] Project shape: (21, 256, 256, 16) I0330 22:53:16.229901 12768 efficientnet_model.py:374] Block blocks_1 input shape: (21, 256, 256, 16) I0330 22:53:16.312902 14952 efficientnet_model.py:414] Project shape: (21, 256, 256, 16) I0330 22:53:16.328903 14952 efficientnet_model.py:374] Block blocks_1 input shape: (21, 256, 256, 16) I0330 22:53:16.350910 10288 efficientnet_model.py:414] Project shape: (21, 256, 256, 16) I0330 22:53:16.363910 10288 efficientnet_model.py:374] Block blocks_1 input shape: (21, 256, 256, 16) INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:16.380912 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:16.439913 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:16.498916 12768 efficientnet_model.py:390] Expand shape: (21, 256, 256, 96) I0330 22:53:16.587917 14952 efficientnet_model.py:390] Expand shape: (21, 256, 256, 96) I0330 22:53:16.613919 10288 efficientnet_model.py:390] Expand shape: (21, 256, 256, 96) INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:16.624921 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:16.680977 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:53:16.811934 12768 efficientnet_model.py:393] DWConv shape: (21, 128, 128, 96) I0330 22:53:16.878933 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 96) I0330 22:53:16.983940 14952 efficientnet_model.py:393] DWConv shape: (21, 128, 128, 96) I0330 22:53:17.000944 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 96) I0330 22:53:17.043947 10288 efficientnet_model.py:393] DWConv shape: (21, 128, 128, 96) I0330 22:53:17.060949 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 96) I0330 22:53:17.186955 12768 efficientnet_model.py:414] Project shape: (21, 128, 128, 24) I0330 22:53:17.209950 12768 efficientnet_model.py:374] Block blocks_2 input shape: (21, 128, 128, 24) I0330 22:53:17.299962 14952 efficientnet_model.py:414] Project shape: (21, 128, 128, 24) I0330 22:53:17.320965 14952 efficientnet_model.py:374] Block blocks_2 input shape: (21, 128, 128, 24) I0330 22:53:17.362965 10288 efficientnet_model.py:414] Project shape: (21, 128, 128, 24) I0330 22:53:17.383965 10288 efficientnet_model.py:374] Block blocks_2 input shape: (21, 128, 128, 24) I0330 22:53:17.509972 12768 efficientnet_model.py:390] Expand shape: (21, 128, 128, 144) I0330 22:53:17.600974 14952 efficientnet_model.py:390] Expand shape: (21, 128, 128, 144) I0330 22:53:17.625976 10288 efficientnet_model.py:390] Expand shape: (21, 128, 128, 144) I0330 22:53:17.754985 12768 efficientnet_model.py:393] DWConv shape: (21, 128, 128, 144) I0330 22:53:17.809988 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 144) I0330 22:53:17.897990 14952 efficientnet_model.py:393] DWConv shape: (21, 128, 128, 144) I0330 22:53:17.913991 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 144) I0330 22:53:17.958999 10288 efficientnet_model.py:393] DWConv shape: (21, 128, 128, 144) I0330 22:53:17.974999 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 144) I0330 22:53:18.113009 12768 efficientnet_model.py:414] Project shape: (21, 128, 128, 24) I0330 22:53:18.140007 12768 efficientnet_model.py:374] Block blocks_3 input shape: (21, 128, 128, 24) I0330 22:53:18.229007 14952 efficientnet_model.py:414] Project shape: (21, 128, 128, 24) I0330 22:53:18.251011 14952 efficientnet_model.py:374] Block blocks_3 input shape: (21, 128, 128, 24) I0330 22:53:18.293017 10288 efficientnet_model.py:414] Project shape: (21, 128, 128, 24) I0330 22:53:18.316020 10288 efficientnet_model.py:374] Block blocks_3 input shape: (21, 128, 128, 24) I0330 22:53:18.447739 12768 efficientnet_model.py:390] Expand shape: (21, 128, 128, 144) I0330 22:53:18.536738 14952 efficientnet_model.py:390] Expand shape: (21, 128, 128, 144) I0330 22:53:18.582746 10288 efficientnet_model.py:390] Expand shape: (21, 128, 128, 144) I0330 22:53:18.714442 12768 efficientnet_model.py:393] DWConv shape: (21, 64, 64, 144) I0330 22:53:18.784447 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 144) I0330 22:53:18.875450 14952 efficientnet_model.py:393] DWConv shape: (21, 64, 64, 144) I0330 22:53:18.888454 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 144) I0330 22:53:18.933459 10288 efficientnet_model.py:393] DWConv shape: (21, 64, 64, 144) I0330 22:53:18.949459 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 144) I0330 22:53:19.076461 12768 efficientnet_model.py:414] Project shape: (21, 64, 64, 40) I0330 22:53:19.101466 12768 efficientnet_model.py:374] Block blocks_4 input shape: (21, 64, 64, 40) I0330 22:53:19.188467 14952 efficientnet_model.py:414] Project shape: (21, 64, 64, 40) I0330 22:53:19.211467 14952 efficientnet_model.py:374] Block blocks_4 input shape: (21, 64, 64, 40) I0330 22:53:19.233472 10288 efficientnet_model.py:414] Project shape: (21, 64, 64, 40) I0330 22:53:19.255475 10288 efficientnet_model.py:374] Block blocks_4 input shape: (21, 64, 64, 40) I0330 22:53:19.367479 12768 efficientnet_model.py:390] Expand shape: (21, 64, 64, 240) I0330 22:53:19.435483 14952 efficientnet_model.py:390] Expand shape: (21, 64, 64, 240) I0330 22:53:19.480482 10288 efficientnet_model.py:390] Expand shape: (21, 64, 64, 240) I0330 22:53:19.639912 12768 efficientnet_model.py:393] DWConv shape: (21, 64, 64, 240) I0330 22:53:19.715920 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 240) I0330 22:53:19.822928 14952 efficientnet_model.py:393] DWConv shape: (21, 64, 64, 240) I0330 22:53:19.838928 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 240) I0330 22:53:19.861929 10288 efficientnet_model.py:393] DWConv shape: (21, 64, 64, 240) I0330 22:53:19.875930 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 240) I0330 22:53:20.015982 12768 efficientnet_model.py:414] Project shape: (21, 64, 64, 40) I0330 22:53:20.041939 12768 efficientnet_model.py:374] Block blocks_5 input shape: (21, 64, 64, 40) I0330 22:53:20.131942 14952 efficientnet_model.py:414] Project shape: (21, 64, 64, 40) I0330 22:53:20.178942 14952 efficientnet_model.py:374] Block blocks_5 input shape: (21, 64, 64, 40) I0330 22:53:20.204948 10288 efficientnet_model.py:414] Project shape: (21, 64, 64, 40) I0330 22:53:20.249950 10288 efficientnet_model.py:374] Block blocks_5 input shape: (21, 64, 64, 40) I0330 22:53:20.375952 12768 efficientnet_model.py:390] Expand shape: (21, 64, 64, 240) I0330 22:53:20.464964 14952 efficientnet_model.py:390] Expand shape: (21, 64, 64, 240) I0330 22:53:20.491964 10288 efficientnet_model.py:390] Expand shape: (21, 64, 64, 240) I0330 22:53:20.642971 12768 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 240) I0330 22:53:20.694974 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 240) I0330 22:53:20.789983 14952 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 240) I0330 22:53:20.828024 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 240) I0330 22:53:20.867983 10288 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 240) I0330 22:53:20.880985 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 240) I0330 22:53:21.013914 12768 efficientnet_model.py:414] Project shape: (21, 32, 32, 80) I0330 22:53:21.049971 12768 efficientnet_model.py:374] Block blocks_6 input shape: (21, 32, 32, 80) I0330 22:53:21.199494 14952 efficientnet_model.py:414] Project shape: (21, 32, 32, 80) I0330 22:53:21.227494 14952 efficientnet_model.py:374] Block blocks_6 input shape: (21, 32, 32, 80) I0330 22:53:21.278495 10288 efficientnet_model.py:414] Project shape: (21, 32, 32, 80) I0330 22:53:21.306500 10288 efficientnet_model.py:374] Block blocks_6 input shape: (21, 32, 32, 80) I0330 22:53:21.539514 12768 efficientnet_model.py:390] Expand shape: (21, 32, 32, 480) I0330 22:53:21.634521 14952 efficientnet_model.py:390] Expand shape: (21, 32, 32, 480) I0330 22:53:21.686521 10288 efficientnet_model.py:390] Expand shape: (21, 32, 32, 480) I0330 22:53:21.817972 12768 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 480) I0330 22:53:21.893971 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 480) I0330 22:53:22.002985 14952 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 480) I0330 22:53:22.017984 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 480) I0330 22:53:22.044988 10288 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 480) I0330 22:53:22.059986 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 480) I0330 22:53:22.194992 12768 efficientnet_model.py:414] Project shape: (21, 32, 32, 80) I0330 22:53:22.245994 12768 efficientnet_model.py:374] Block blocks_7 input shape: (21, 32, 32, 80) I0330 22:53:22.312000 14952 efficientnet_model.py:414] Project shape: (21, 32, 32, 80) I0330 22:53:22.339000 14952 efficientnet_model.py:374] Block blocks_7 input shape: (21, 32, 32, 80) I0330 22:53:22.381003 10288 efficientnet_model.py:414] Project shape: (21, 32, 32, 80) I0330 22:53:22.411004 10288 efficientnet_model.py:374] Block blocks_7 input shape: (21, 32, 32, 80) I0330 22:53:22.544064 12768 efficientnet_model.py:390] Expand shape: (21, 32, 32, 480) I0330 22:53:22.648018 14952 efficientnet_model.py:390] Expand shape: (21, 32, 32, 480) I0330 22:53:22.674020 10288 efficientnet_model.py:390] Expand shape: (21, 32, 32, 480) I0330 22:53:22.825028 12768 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 480) I0330 22:53:22.900035 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 480) I0330 22:53:22.970036 14952 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 480) I0330 22:53:22.984045 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 480) I0330 22:53:23.031041 10288 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 480) I0330 22:53:23.054042 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 480) I0330 22:53:23.182699 12768 efficientnet_model.py:414] Project shape: (21, 32, 32, 80) I0330 22:53:23.211695 12768 efficientnet_model.py:374] Block blocks_8 input shape: (21, 32, 32, 80) I0330 22:53:23.276699 14952 efficientnet_model.py:414] Project shape: (21, 32, 32, 80) I0330 22:53:23.304700 14952 efficientnet_model.py:374] Block blocks_8 input shape: (21, 32, 32, 80) I0330 22:53:23.352707 10288 efficientnet_model.py:414] Project shape: (21, 32, 32, 80) I0330 22:53:23.394703 10288 efficientnet_model.py:374] Block blocks_8 input shape: (21, 32, 32, 80) I0330 22:53:23.534590 12768 efficientnet_model.py:390] Expand shape: (21, 32, 32, 480) I0330 22:53:23.625594 14952 efficientnet_model.py:390] Expand shape: (21, 32, 32, 480) I0330 22:53:23.675594 10288 efficientnet_model.py:390] Expand shape: (21, 32, 32, 480) I0330 22:53:23.806298 12768 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 480) I0330 22:53:23.860300 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 480) I0330 22:53:23.949301 14952 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 480) I0330 22:53:23.965306 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 480) I0330 22:53:23.991311 10288 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 480) I0330 22:53:24.006314 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 480) I0330 22:53:24.159318 12768 efficientnet_model.py:414] Project shape: (21, 32, 32, 112) I0330 22:53:24.192318 12768 efficientnet_model.py:374] Block blocks_9 input shape: (21, 32, 32, 112) I0330 22:53:24.278319 14952 efficientnet_model.py:414] Project shape: (21, 32, 32, 112) I0330 22:53:24.333328 14952 efficientnet_model.py:374] Block blocks_9 input shape: (21, 32, 32, 112) I0330 22:53:24.388384 10288 efficientnet_model.py:414] Project shape: (21, 32, 32, 112) I0330 22:53:24.420334 10288 efficientnet_model.py:374] Block blocks_9 input shape: (21, 32, 32, 112) I0330 22:53:24.549335 12768 efficientnet_model.py:390] Expand shape: (21, 32, 32, 672) I0330 22:53:24.621345 14952 efficientnet_model.py:390] Expand shape: (21, 32, 32, 672) I0330 22:53:24.667352 10288 efficientnet_model.py:390] Expand shape: (21, 32, 32, 672) I0330 22:53:24.788353 12768 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 672) I0330 22:53:24.842355 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 672) I0330 22:53:24.932356 14952 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 672) I0330 22:53:24.947360 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 672) I0330 22:53:24.973205 10288 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 672) I0330 22:53:24.988206 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 672) I0330 22:53:25.123209 12768 efficientnet_model.py:414] Project shape: (21, 32, 32, 112) I0330 22:53:25.181217 12768 efficientnet_model.py:374] Block blocks_10 input shape: (21, 32, 32, 112) I0330 22:53:25.266222 14952 efficientnet_model.py:414] Project shape: (21, 32, 32, 112) I0330 22:53:25.301224 14952 efficientnet_model.py:374] Block blocks_10 input shape: (21, 32, 32, 112) I0330 22:53:25.324225 10288 efficientnet_model.py:414] Project shape: (21, 32, 32, 112) I0330 22:53:25.370228 10288 efficientnet_model.py:374] Block blocks_10 input shape: (21, 32, 32, 112) I0330 22:53:25.538777 12768 efficientnet_model.py:390] Expand shape: (21, 32, 32, 672) I0330 22:53:25.628782 14952 efficientnet_model.py:390] Expand shape: (21, 32, 32, 672) I0330 22:53:25.656785 10288 efficientnet_model.py:390] Expand shape: (21, 32, 32, 672) I0330 22:53:25.800791 12768 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 672) I0330 22:53:25.853793 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 672) I0330 22:53:25.944797 14952 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 672) I0330 22:53:25.958797 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 672) I0330 22:53:26.011653 10288 efficientnet_model.py:393] DWConv shape: (21, 32, 32, 672) I0330 22:53:26.031654 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 672) I0330 22:53:26.323671 12768 efficientnet_model.py:414] Project shape: (21, 32, 32, 112) I0330 22:53:26.359673 12768 efficientnet_model.py:374] Block blocks_11 input shape: (21, 32, 32, 112) I0330 22:53:26.444676 14952 efficientnet_model.py:414] Project shape: (21, 32, 32, 112) I0330 22:53:26.481680 14952 efficientnet_model.py:374] Block blocks_11 input shape: (21, 32, 32, 112) I0330 22:53:26.504256 10288 efficientnet_model.py:414] Project shape: (21, 32, 32, 112) I0330 22:53:26.537256 10288 efficientnet_model.py:374] Block blocks_11 input shape: (21, 32, 32, 112) I0330 22:53:26.696186 12768 efficientnet_model.py:390] Expand shape: (21, 32, 32, 672) I0330 22:53:26.786188 14952 efficientnet_model.py:390] Expand shape: (21, 32, 32, 672) I0330 22:53:26.813191 10288 efficientnet_model.py:390] Expand shape: (21, 32, 32, 672) I0330 22:53:26.944175 12768 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 672) I0330 22:53:26.998183 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 672) I0330 22:53:27.090195 14952 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 672) I0330 22:53:27.107190 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 672) I0330 22:53:27.142193 10288 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 672) I0330 22:53:27.157192 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 672) I0330 22:53:27.290195 12768 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:27.327203 12768 efficientnet_model.py:374] Block blocks_12 input shape: (21, 16, 16, 192) I0330 22:53:27.438209 14952 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:27.476210 14952 efficientnet_model.py:374] Block blocks_12 input shape: (21, 16, 16, 192) I0330 22:53:27.520356 10288 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:27.581309 10288 efficientnet_model.py:374] Block blocks_12 input shape: (21, 16, 16, 192) I0330 22:53:27.749318 12768 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:27.843326 14952 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:27.891473 10288 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:28.027485 12768 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:28.087486 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:28.187493 14952 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:28.209493 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:28.273491 10288 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:28.302494 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:28.437596 12768 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:28.476602 12768 efficientnet_model.py:374] Block blocks_13 input shape: (21, 16, 16, 192) I0330 22:53:28.571609 14952 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:28.616606 14952 efficientnet_model.py:374] Block blocks_13 input shape: (21, 16, 16, 192) I0330 22:53:28.658606 10288 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:28.697615 10288 efficientnet_model.py:374] Block blocks_13 input shape: (21, 16, 16, 192) I0330 22:53:28.849864 12768 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:28.944865 14952 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:28.993874 10288 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:29.136881 12768 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:29.216886 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:29.284890 14952 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:29.304893 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:29.347893 10288 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:29.373895 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:29.541905 12768 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:29.592901 12768 efficientnet_model.py:374] Block blocks_14 input shape: (21, 16, 16, 192) I0330 22:53:29.676913 14952 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:29.717915 14952 efficientnet_model.py:374] Block blocks_14 input shape: (21, 16, 16, 192) I0330 22:53:29.763918 10288 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:29.806921 10288 efficientnet_model.py:374] Block blocks_14 input shape: (21, 16, 16, 192) I0330 22:53:29.962723 12768 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:30.060722 14952 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:30.089733 10288 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:30.228337 12768 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:30.293339 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:30.365345 14952 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:30.380344 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:30.427347 10288 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:30.442348 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:30.604356 12768 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:30.647359 12768 efficientnet_model.py:374] Block blocks_15 input shape: (21, 16, 16, 192) I0330 22:53:30.738365 14952 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:30.781364 14952 efficientnet_model.py:374] Block blocks_15 input shape: (21, 16, 16, 192) I0330 22:53:30.805366 10288 efficientnet_model.py:414] Project shape: (21, 16, 16, 192) I0330 22:53:30.868366 10288 efficientnet_model.py:374] Block blocks_15 input shape: (21, 16, 16, 192) I0330 22:53:30.998380 12768 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:31.117390 14952 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:31.155385 10288 efficientnet_model.py:390] Expand shape: (21, 16, 16, 1152) I0330 22:53:31.310390 12768 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:31.366398 12768 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:31.458404 14952 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:31.475404 14952 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:31.504407 10288 efficientnet_model.py:393] DWConv shape: (21, 16, 16, 1152) I0330 22:53:31.519408 10288 efficientnet_model.py:195] Built SE se : (21, 1, 1, 1152) I0330 22:53:31.679256 12768 efficientnet_model.py:414] Project shape: (21, 16, 16, 320) I0330 22:53:31.859261 14952 efficientnet_model.py:414] Project shape: (21, 16, 16, 320) I0330 22:53:32.054278 10288 efficientnet_model.py:414] Project shape: (21, 16, 16, 320) I0330 22:54:00.696263 12768 det_model_fn.py:81] LR schedule method: cosine I0330 22:54:00.960279 12768 utils.py:373] Adding scale summary ('lrn_rate', <tf.Tensor 'Select:0' shape=() dtype=float32>) I0330 22:54:00.963281 12768 utils.py:373] Adding scale summary ('trainloss/cls_loss', <tf.Tensor 'AddN:0' shape=() dtype=float32>) I0330 22:54:00.966282 12768 utils.py:373] Adding scale summary ('trainloss/box_loss', <tf.Tensor 'AddN_1:0' shape=() dtype=float32>) I0330 22:54:00.970283 12768 utils.py:373] Adding scale summary ('trainloss/det_loss', <tf.Tensor 'add_3:0' shape=() dtype=float32>) I0330 22:54:00.972282 12768 utils.py:373] Adding scale summary ('trainloss/reg_l2_loss', <tf.Tensor 'mul_14:0' shape=() dtype=float32>) I0330 22:54:00.975283 12768 utils.py:373] Adding scale summary ('trainloss/loss', <tf.Tensor 'add_4:0' shape=() dtype=float32>) I0330 22:54:00.980283 12768 utils.py:373] Adding scale summary ('train_epochs', <tf.Tensor 'truediv_7:0' shape=() dtype=float32>) I0330 22:54:01.017280 12768 det_model_fn.py:397] clip gradients norm by 10.000000 I0330 22:54:02.205347 14952 det_model_fn.py:81] LR schedule method: cosine I0330 22:54:02.488367 14952 utils.py:373] Adding scale summary ('lrn_rate', <tf.Tensor 'replica_1/Select:0' shape=() dtype=float32>) I0330 22:54:02.493367 14952 utils.py:373] Adding scale summary ('trainloss/cls_loss', <tf.Tensor 'replica_1/AddN:0' shape=() dtype=float32>) I0330 22:54:02.496366 14952 utils.py:373] Adding scale summary ('trainloss/box_loss', <tf.Tensor 'replica_1/AddN_1:0' shape=() dtype=float32>) I0330 22:54:02.498368 14952 utils.py:373] Adding scale summary ('trainloss/det_loss', <tf.Tensor 'replica_1/add_3:0' shape=() dtype=float32>) I0330 22:54:02.501369 14952 utils.py:373] Adding scale summary ('trainloss/reg_l2_loss', <tf.Tensor 'replica_1/mul_14:0' shape=() dtype=float32>) I0330 22:54:02.503366 14952 utils.py:373] Adding scale summary ('trainloss/loss', <tf.Tensor 'replica_1/add_4:0' shape=() dtype=float32>) I0330 22:54:02.508368 14952 utils.py:373] Adding scale summary ('train_epochs', <tf.Tensor 'replica_1/truediv_7:0' shape=() dtype=float32>) I0330 22:54:02.525367 14952 det_model_fn.py:397] clip gradients norm by 10.000000 I0330 22:54:03.448876 10288 det_model_fn.py:81] LR schedule method: cosine I0330 22:54:03.750894 10288 utils.py:373] Adding scale summary ('lrn_rate', <tf.Tensor 'replica_2/Select:0' shape=() dtype=float32>) I0330 22:54:03.753894 10288 utils.py:373] Adding scale summary ('trainloss/cls_loss', <tf.Tensor 'replica_2/AddN:0' shape=() dtype=float32>) I0330 22:54:03.756893 10288 utils.py:373] Adding scale summary ('trainloss/box_loss', <tf.Tensor 'replica_2/AddN_1:0' shape=() dtype=float32>) I0330 22:54:03.758894 10288 utils.py:373] Adding scale summary ('trainloss/det_loss', <tf.Tensor 'replica_2/add_3:0' shape=() dtype=float32>) I0330 22:54:03.761894 10288 utils.py:373] Adding scale summary ('trainloss/reg_l2_loss', <tf.Tensor 'replica_2/mul_14:0' shape=() dtype=float32>) I0330 22:54:03.763894 10288 utils.py:373] Adding scale summary ('trainloss/loss', <tf.Tensor 'replica_2/add_4:0' shape=() dtype=float32>) I0330 22:54:03.767894 10288 utils.py:373] Adding scale summary ('train_epochs', <tf.Tensor 'replica_2/truediv_7:0' shape=() dtype=float32>) I0330 22:54:03.785893 10288 det_model_fn.py:397] clip gradients norm by 10.000000 I0330 22:59:16.394097 12768 utils.py:373] Adding scale summary ('gradient_norm', <tf.Tensor 'clip/global_norm_1/global_norm:0' shape=() dtype=float32>) I0330 22:59:20.326254 14952 utils.py:373] Adding scale summary ('gradient_norm', <tf.Tensor 'replica_1/clip/global_norm_1/global_norm:0' shape=() dtype=float32>) I0330 22:59:24.380414 10288 utils.py:373] Adding scale summary ('gradient_norm', <tf.Tensor 'replica_2/clip/global_norm_1/global_norm:0' shape=() dtype=float32>) WARNING:tensorflow:From C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\values.py:580: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. W0330 22:59:31.729707 12768 deprecation.py:339] From C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\distribute\values.py:580: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:52.921545 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:52.995548 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:53.131436 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:53.176455 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:53.224457 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:53.461467 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:53.694476 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:53.753478 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:53.936995 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 I0330 22:59:54.078046 8636 cross_device_ops.py:847] batch_all_reduce: 1 all-reduces with algorithm = nccl, num_packs = 1 INFO:tensorflow:Done calling model_fn. I0330 23:00:44.568503 12768 api.py:479] Done calling model_fn. INFO:tensorflow:Done calling model_fn. I0330 23:00:44.603502 14952 api.py:479] Done calling model_fn. INFO:tensorflow:Done calling model_fn. I0330 23:00:44.638509 10288 api.py:479] Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. I0330 23:00:47.731379 8636 basic_session_run_hooks.py:546] Create CheckpointSaverHook. WARNING:tensorflow:From C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\util.py:96: DistributedIteratorV1.initialize (from tensorflow.python.distribute.input_lib) is deprecated and will be removed in a future version. Instructions for updating: Use the iterator's
initializerproperty instead. W0330 23:00:47.732379 8636 deprecation.py:339] From C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow_estimator\python\estimator\util.py:96: DistributedIteratorV1.initialize (from tensorflow.python.distribute.input_lib) is deprecated and will be removed in a future version. Instructions for updating: Use the iterator'sinitializerproperty instead. INFO:tensorflow:Graph was finalized. I0330 23:01:01.847940 8636 monitored_session.py:246] Graph was finalized. 2021-03-30 23:01:01.848660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 23:01:01.848938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 23:01:01.849190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:b3:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s 2021-03-30 23:01:01.849438: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-03-30 23:01:01.849562: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-03-30 23:01:01.849684: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-03-30 23:01:01.849805: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-03-30 23:01:01.849922: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-03-30 23:01:01.850041: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-03-30 23:01:01.850159: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-03-30 23:01:01.850275: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-03-30 23:01:01.850453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2 2021-03-30 23:01:01.850738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-30 23:01:01.850848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2 2021-03-30 23:01:01.850920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N N N 2021-03-30 23:01:01.850995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: N N N 2021-03-30 23:01:01.851066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: N N N 2021-03-30 23:01:01.851349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9417 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:17:00.0, compute capability: 7.5) 2021-03-30 23:01:01.851587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9417 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5) 2021-03-30 23:01:01.851817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9417 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5) 2021-03-30 23:01:01.852031: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-30 23:01:07.850951: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes) Traceback (most recent call last): File "C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call return fn(*args) File "C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\client\session.py", line 1358, in _run_fn self._extend_graph() File "C:\Users\ha485.conda\envs\'Official3'\lib\site-packages\tensorflow\python\client\session.py", line 1398, in _extend_graph tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'NcclAllReduce' used by {{node efficientnet-b0/stem/tpu_batch_normalization/NcclAllReduce}} with these attrs: [reduction="sum", shared_name="c0", T=DT_FLOAT, num_devices=3] Registered devices: [CPU, GPU] Registered kernels: