search

google / automl

Google Brain AutoML
Apache License 2.0
6.23k stars 1.45k forks source link

AttributeError: 'PrefetchDataset' object has no attribute 'shape' #893

Closed takusaitoh closed 3 years ago

takusaitoh commented 3 years ago

Hi I have a problem when I tried to run "main.py".(7. Eval on COCO 2017 val or test-dev.) It seems that x has no attribute of "shape".

Error is follows, please could you help me to solve this error.

export MODEL=efficientdet-d0
export CKPT_PATH=efficientdet-d0
python main.py --mode=eval      --model_name=${MODEL}  --model_dir=${CKPT_PATH}      --validation_file_pattern=tfrecord/val*      --val_json_file=annotations/instances_val2017.json
2020-12-01 15:11:11.234897: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-12-01 15:11:11.234939: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
I1201 15:11:12.240603 140229059118848 main.py:230] {'name': 'efficientdet-d0', 'act_type': 'swish', 'image_size': (512, 512), 'target_size': None, 'input_rand_hflip': True, 'jitter_min': 0.1, 'jitter_max': 2.0, 'autoaugment_policy': None, 'grid_mask': False, 'sample_image': None, 'map_freq': 5, 'num_classes': 90, 'seg_num_classes': 3, 'heads': ['object_detection'], 'skip_crowd_during_training': True, 'label_map': None, 'max_instances_per_image': 100, 'regenerate_source_id': False, 'min_level': 3, 'max_level': 7, 'num_scales': 3, 'aspect_ratios': [1.0, 2.0, 0.5], 'anchor_scale': 4.0, 'is_training_bn': True, 'momentum': 0.9, 'optimizer': 'sgd', 'learning_rate': 0.08, 'lr_warmup_init': 0.008, 'lr_warmup_epoch': 1.0, 'first_lr_drop_epoch': 200.0, 'second_lr_drop_epoch': 250.0, 'poly_lr_power': 0.9, 'clip_gradients_norm': 10.0, 'num_epochs': 300, 'data_format': 'channels_last', 'label_smoothing': 0.0, 'alpha': 0.25, 'gamma': 1.5, 'delta': 0.1, 'box_loss_weight': 50.0, 'iou_loss_type': None, 'iou_loss_weight': 1.0, 'weight_decay': 4e-05, 'strategy': None, 'mixed_precision': False, 'loss_scale': 1024, 'model_optimizations': {}, 'box_class_repeats': 3, 'fpn_cell_repeats': 3, 'fpn_num_filters': 64, 'separable_conv': True, 'apply_bn_for_resampling': True, 'conv_after_downsample': False, 'conv_bn_act_pattern': False, 'drop_remainder': True, 'nms_configs': {'method': 'gaussian', 'iou_thresh': None, 'score_thresh': 0.0, 'sigma': None, 'pyfunc': False, 'max_nms_inputs': 0, 'max_output_size': 100}, 'fpn_name': None, 'fpn_weight_method': None, 'fpn_config': None, 'survival_prob': None, 'img_summary_steps': None, 'lr_decay_method': 'cosine', 'moving_average_decay': 0.9998, 'ckpt_var_scope': None, 'skip_mismatch': True, 'backbone_name': 'efficientnet-b0', 'backbone_config': None, 'var_freeze_expr': None, 'use_keras_model': True, 'dataset_type': None, 'positives_momentum': None, 'grad_checkpoint': False, 'model_name': 'efficientdet-d0', 'iterations_per_loop': 100, 'model_dir': 'efficientdet-d0', 'num_shards': 8, 'num_examples_per_epoch': 120000, 'backbone_ckpt': '', 'ckpt': None, 'val_json_file': 'annotations/instances_val2017.json', 'testdev_dir': None, 'profile': False, 'mode': 'eval'}
I1201 15:11:12.258552 140229059118848 estimator.py:201] Using config: {'_model_dir': 'efficientdet-d0', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I1201 15:11:12.259219 140229059118848 estimator.py:201] Using config: {'_model_dir': 'efficientdet-d0', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I1201 15:11:12.259562 140229059118848 checkpoint_utils.py:125] Waiting for new checkpoint at efficientdet-d0
I1201 15:11:12.261218 140229059118848 checkpoint_utils.py:134] Found new checkpoint at efficientdet-d0/model
I1201 15:11:12.261281 140229059118848 main.py:310] Starting to evaluate.
2020-12-01 15:11:12.374120: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-12-01 15:11:12.390421: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-01 15:11:12.390978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:05:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.635GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2020-12-01 15:11:12.391040: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-12-01 15:11:12.391079: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-12-01 15:11:12.391114: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-12-01 15:11:12.391149: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-12-01 15:11:12.391183: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-12-01 15:11:12.391219: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2020-12-01 15:11:12.393070: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-12-01 15:11:12.393083: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
I1201 15:11:12.689212 140229059118848 estimator.py:1111] Calling model_fn.
I1201 15:11:12.693496 140229059118848 efficientnet_builder.py:215] global_params= GlobalParams(batch_norm_momentum=0.99, batch_norm_epsilon=0.001, dropout_rate=0.2, data_format='channels_last', num_classes=1000, width_coefficient=1.0, depth_coefficient=1.0, depth_divisor=8, min_depth=None, survival_prob=0.0, relu_fn=functools.partial(<function activation_fn at 0x7f8940ceff28>, act_type='swish'), batch_norm=<class 'utils.BatchNormalization'>, use_se=True, local_pooling=None, condconv_num_experts=None, clip_projection_output=False, blocks_args=['r1_k3_s11_e1_i32_o16_se0.25', 'r2_k3_s22_e6_i16_o24_se0.25', 'r2_k5_s22_e6_i24_o40_se0.25', 'r3_k3_s22_e6_i40_o80_se0.25', 'r3_k5_s11_e6_i80_o112_se0.25', 'r4_k5_s22_e6_i112_o192_se0.25', 'r1_k3_s11_e6_i192_o320_se0.25'], fix_head_stem=None, grad_checkpoint=False)
I1201 15:11:12.893575 140229059118848 efficientdet_keras.py:725] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I1201 15:11:12.894428 140229059118848 efficientdet_keras.py:725] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I1201 15:11:12.895064 140229059118848 efficientdet_keras.py:725] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I1201 15:11:12.895764 140229059118848 efficientdet_keras.py:725] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I1201 15:11:12.896456 140229059118848 efficientdet_keras.py:725] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I1201 15:11:12.897186 140229059118848 efficientdet_keras.py:725] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I1201 15:11:12.898146 140229059118848 efficientdet_keras.py:725] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I1201 15:11:12.898826 140229059118848 efficientdet_keras.py:725] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I1201 15:11:12.899772 140229059118848 efficientdet_keras.py:725] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I1201 15:11:12.900394 140229059118848 efficientdet_keras.py:725] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I1201 15:11:12.901095 140229059118848 efficientdet_keras.py:725] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I1201 15:11:12.901755 140229059118848 efficientdet_keras.py:725] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I1201 15:11:12.902386 140229059118848 efficientdet_keras.py:725] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I1201 15:11:12.903052 140229059118848 efficientdet_keras.py:725] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I1201 15:11:12.903717 140229059118848 efficientdet_keras.py:725] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I1201 15:11:12.904389 140229059118848 efficientdet_keras.py:725] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
I1201 15:11:12.905378 140229059118848 efficientdet_keras.py:725] fnode 0 : {'feat_level': 6, 'inputs_offsets': [3, 4]}
I1201 15:11:12.906073 140229059118848 efficientdet_keras.py:725] fnode 1 : {'feat_level': 5, 'inputs_offsets': [2, 5]}
I1201 15:11:12.906700 140229059118848 efficientdet_keras.py:725] fnode 2 : {'feat_level': 4, 'inputs_offsets': [1, 6]}
I1201 15:11:12.907427 140229059118848 efficientdet_keras.py:725] fnode 3 : {'feat_level': 3, 'inputs_offsets': [0, 7]}
I1201 15:11:12.908078 140229059118848 efficientdet_keras.py:725] fnode 4 : {'feat_level': 4, 'inputs_offsets': [1, 7, 8]}
I1201 15:11:12.908779 140229059118848 efficientdet_keras.py:725] fnode 5 : {'feat_level': 5, 'inputs_offsets': [2, 6, 9]}
I1201 15:11:12.909471 140229059118848 efficientdet_keras.py:725] fnode 6 : {'feat_level': 6, 'inputs_offsets': [3, 5, 10]}
I1201 15:11:12.910131 140229059118848 efficientdet_keras.py:725] fnode 7 : {'feat_level': 7, 'inputs_offsets': [4, 11]}
Traceback (most recent call last):
  File "main.py", line 367, in <module>
    app.run(main)
  File "/opt/conda/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/opt/conda/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "main.py", line 312, in main
    eval_results = eval_est.evaluate(eval_input_fn, steps=eval_steps)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 469, in evaluate
    name=name)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 511, in _actual_eval
    return _evaluate()
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 493, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1424, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1460, in _call_model_fn_eval
    features, labels, model_fn_lib.ModeKeys.EVAL, config)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1112, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/work/shared/efficientdet/det_model_fn.py", line 621, in efficientdet_model_fn
    variable_filter_fn=variable_filter_fn)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 302, in wrapper
    return func(*args, **kwargs)
  File "/work/shared/efficientdet/det_model_fn.py", line 344, in _model_fn
    precision, model_fn, features, params['is_training_bn'])
  File "/work/shared/efficientdet/utils.py", line 630, in build_model_with_precision
    outputs = mm(ii, *args, **kwargs)
  File "/work/shared/efficientdet/det_model_fn.py", line 333, in model_fn
    cls_out_list, box_out_list = model(inputs, params['is_training_bn'])
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 302, in wrapper
    return func(*args, **kwargs)
  File "/work/shared/efficientdet/keras/efficientdet_keras.py", line 856, in call
    all_feats = self.backbone(inputs, training=training, features_only=True)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 302, in wrapper
    return func(*args, **kwargs)
  File "/work/shared/efficientdet/backbone/efficientnet_model.py", line 734, in call
    outputs = self._stem(inputs, training)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 302, in wrapper
    return func(*args, **kwargs)
  File "/work/shared/efficientdet/backbone/efficientnet_model.py", line 528, in call
    return self._relu_fn(self._bn(self._conv_stem(inputs), training=training))
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 752, in __call__
    self.name)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/keras/engine/input_spec.py", line 166, in assert_input_compatibility
    if x.shape.ndims is None:
AttributeError: 'PrefetchDataset' object has no attribute 'shape'
tPiffrader commented 3 years ago

https://stackoverflow.com/questions/62436302/extract-target-from-tensorflow-prefetchdataset if you convert it, you can get the shape.