esparig
commented
3 years ago Similarly for EfficientNet:
root@745616268d5f:/usr/local/lib/python3.6/dist-packages/official/vision/image_classification# python3 classifier_trainer.py --mode=train_and_eval --model_type=efficientnet --dataset=imagenet --model_dir=$MODEL_DIR --data_dir=$DATA_DIR --config_file=configs/examples/efficientnet/imagenet/efficientnet-b0-gpu.yaml --params_override='runtime.num_gpus=1'
2021-09-15 06:42:04.619103: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-15 06:42:04.629605: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-15 06:42:04.631699: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I0915 06:42:04.638748 140551238154048 classifier_trainer.py:181] Base params: {'evaluation': {'epochs_between_evals': 1, 'skip_eval': False, 'steps': None},
'export': {'checkpoint': None, 'destination': None},
'mode': None,
'model': {'learning_rate': {'boundaries': None,
'decay_epochs': 2.4,
'decay_rate': 0.97,
'examples_per_epoch': None,
'initial_lr': 0.008,
'multipliers': None,
'name': 'exponential',
'scale_by_batch_size': 0.0078125,
'staircase': True,
'warmup_epochs': 5},
'loss': {'label_smoothing': 0.1, 'name': 'categorical_crossentropy'},
'model_params': {'model_name': 'efficientnet-b0',
'model_weights_path': '',
'overrides': {'activation': 'swish',
'batch_norm': 'default',
'dtype': 'float32',
'num_classes': 1000,
'rescale_input': True},
'weights_format': 'saved_model'},
'name': 'EfficientNet',
'num_classes': 1000,
'optimizer': {'beta_1': None,
'beta_2': None,
'decay': 0.9,
'epsilon': 0.001,
'lookahead': None,
'momentum': 0.9,
'moving_average_decay': None,
'name': 'rmsprop',
'nesterov': None}},
'model_dir': None,
'model_name': None,
'runtime': {'all_reduce_alg': None,
'batchnorm_spatial_persistent': False,
'dataset_num_private_threads': None,
'default_shard_dim': -1,
'distribution_strategy': 'mirrored',
'enable_xla': False,
'gpu_thread_mode': None,
'loss_scale': None,
'mixed_precision_dtype': None,
'num_cores_per_replica': 1,
'num_gpus': 0,
'num_packs': 1,
'per_gpu_thread_count': 0,
'run_eagerly': False,
'task_index': -1,
'tpu': None,
'tpu_enable_xla_dynamic_padder': None,
'worker_hosts': None},
'train': {'callbacks': {'enable_backup_and_restore': False,
'enable_checkpoint_and_export': True,
'enable_tensorboard': True,
'enable_time_history': True},
'epochs': 500,
'metrics': ['accuracy', 'top_5'],
'resume_checkpoint': True,
'set_epoch_loop': False,
'steps': None,
'tensorboard': {'track_lr': True, 'write_model_weights': False},
'time_history': {'log_steps': 100}},
'train_dataset': {'augmenter': {'name': None, 'params': None},
'batch_size': 128,
'builder': 'records',
'cache': False,
'data_dir': None,
'download': False,
'dtype': 'float32',
'file_shuffle_buffer_size': 1024,
'filenames': None,
'image_size': 224,
'mean_subtract': False,
'name': 'imagenet2012',
'num_channels': 3,
'num_classes': 1000,
'num_devices': 1,
'num_examples': 1281167,
'one_hot': True,
'shuffle_buffer_size': 10000,
'skip_decoding': True,
'split': 'train',
'standardize': False,
'tf_data_service': None,
'use_per_replica_batch_size': True},
'validation_dataset': {'augmenter': {'name': None, 'params': None},
'batch_size': 128,
'builder': 'records',
'cache': False,
'data_dir': None,
'download': False,
'dtype': 'float32',
'file_shuffle_buffer_size': 1024,
'filenames': None,
'image_size': 224,
'mean_subtract': False,
'name': 'imagenet2012',
'num_channels': 3,
'num_classes': 1000,
'num_devices': 1,
'num_examples': 1281167,
'one_hot': True,
'shuffle_buffer_size': 10000,
'skip_decoding': True,
'split': 'validation',
'standardize': False,
'tf_data_service': None,
'use_per_replica_batch_size': True}}
I0915 06:42:04.642026 140551238154048 classifier_trainer.py:184] Overriding params: configs/examples/efficientnet/imagenet/efficientnet-b0-gpu.yaml
I0915 06:42:04.653718 140551238154048 classifier_trainer.py:184] Overriding params: runtime.num_gpus=1
I0915 06:42:04.654549 140551238154048 classifier_trainer.py:184] Overriding params: {'model_dir': '', 'mode': 'train_and_eval', 'model': {'name': 'efficientnet'}, 'runtime': {'run_eagerly': None, 'tpu': None}, 'train_dataset': {'data_dir': ''}, 'validation_dataset': {'data_dir': ''}, 'train': {'time_history': {'log_steps': 100}}}
I0915 06:42:04.656747 140551238154048 classifier_trainer.py:190] Final model parameters: {'evaluation': {'epochs_between_evals': 1, 'skip_eval': False, 'steps': None},
'export': {'checkpoint': None, 'destination': None},
'mode': 'train_and_eval',
'model': {'learning_rate': {'boundaries': None,
'decay_epochs': 2.4,
'decay_rate': 0.97,
'examples_per_epoch': None,
'initial_lr': 0.008,
'multipliers': None,
'name': 'exponential',
'scale_by_batch_size': 0.0078125,
'staircase': True,
'warmup_epochs': 5},
'loss': {'label_smoothing': 0.1, 'name': 'categorical_crossentropy'},
'model_params': {'model_name': 'efficientnet-b0',
'model_weights_path': '',
'overrides': {'activation': 'swish',
'batch_norm': 'default',
'dtype': 'float32',
'num_classes': 1000,
'rescale_input': True},
'weights_format': 'saved_model'},
'name': 'efficientnet',
'num_classes': 1000,
'optimizer': {'beta_1': None,
'beta_2': None,
'decay': 0.9,
'epsilon': 0.001,
'lookahead': False,
'momentum': 0.9,
'moving_average_decay': 0.0,
'name': 'rmsprop',
'nesterov': None}},
'model_dir': '',
'model_name': None,
'runtime': {'all_reduce_alg': None,
'batchnorm_spatial_persistent': False,
'dataset_num_private_threads': None,
'default_shard_dim': -1,
'distribution_strategy': 'mirrored',
'enable_xla': False,
'gpu_thread_mode': None,
'loss_scale': None,
'mixed_precision_dtype': None,
'num_cores_per_replica': 1,
'num_gpus': 1,
'num_packs': 1,
'per_gpu_thread_count': 0,
'run_eagerly': None,
'task_index': -1,
'tpu': None,
'tpu_enable_xla_dynamic_padder': None,
'worker_hosts': None},
'train': {'callbacks': {'enable_backup_and_restore': False,
'enable_checkpoint_and_export': True,
'enable_tensorboard': True,
'enable_time_history': True},
'epochs': 500,
'metrics': ['accuracy', 'top_5'],
'resume_checkpoint': True,
'set_epoch_loop': False,
'steps': None,
'tensorboard': {'track_lr': True, 'write_model_weights': False},
'time_history': {'log_steps': 100}},
'train_dataset': {'augmenter': {'name': 'autoaugment', 'params': None},
'batch_size': 32,
'builder': 'records',
'cache': False,
'data_dir': '',
'download': False,
'dtype': 'float32',
'file_shuffle_buffer_size': 1024,
'filenames': None,
'image_size': 224,
'mean_subtract': False,
'name': 'imagenet2012',
'num_channels': 3,
'num_classes': 1000,
'num_devices': 1,
'num_examples': 1281167,
'one_hot': True,
'shuffle_buffer_size': 10000,
'skip_decoding': True,
'split': 'train',
'standardize': False,
'tf_data_service': None,
'use_per_replica_batch_size': True},
'validation_dataset': {'augmenter': {'name': None, 'params': None},
'batch_size': 32,
'builder': 'records',
'cache': False,
'data_dir': '',
'download': False,
'dtype': 'float32',
'file_shuffle_buffer_size': 1024,
'filenames': None,
'image_size': 224,
'mean_subtract': False,
'name': 'imagenet2012',
'num_channels': 3,
'num_classes': 1000,
'num_devices': 1,
'num_examples': 50000,
'one_hot': True,
'shuffle_buffer_size': 10000,
'skip_decoding': True,
'split': 'validation',
'standardize': False,
'tf_data_service': None,
'use_per_replica_batch_size': True}}
I0915 06:42:04.657643 140551238154048 classifier_trainer.py:290] Running train and eval.
2021-09-15 06:42:04.658841: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-09-15 06:42:04.659902: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-15 06:42:04.661132: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-15 06:42:04.662254: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-15 06:42:05.669043: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-15 06:42:05.670517: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-15 06:42:05.671733: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-09-15 06:42:05.672929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 30995 MB memory: -> device: 0, name: Tesla V100-PCIE-32GB, pci bus id: 0000:00:07.0, compute capability: 7.0
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0915 06:42:06.720428 140551238154048 mirrored_strategy.py:369] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0915 06:42:06.722068 140551238154048 classifier_trainer.py:305] Detected 1 devices.
W0915 06:42:06.722257 140551238154048 classifier_trainer.py:105] label_smoothing > 0, so datasets will be one hot encoded.
I0915 06:42:06.722772 140551238154048 dataset_factory.py:176] Using augmentation: autoaugment
I0915 06:42:06.723030 140551238154048 dataset_factory.py:176] Using augmentation: None
I0915 06:42:06.723595 140551238154048 dataset_factory.py:369] Using TFRecords to load data.
Traceback (most recent call last):
File "classifier_trainer.py", line 456, in <module>
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "classifier_trainer.py", line 443, in main
stats = run(flags.FLAGS)
File "classifier_trainer.py", line 435, in run
return train_and_eval(params, strategy_override)
File "classifier_trainer.py", line 312, in train_and_eval
builder.build(strategy) if builder else None for builder in builders
File "classifier_trainer.py", line 312, in <listcomp>
builder.build(strategy) if builder else None for builder in builders
File "/usr/local/lib/python3.6/dist-packages/official/vision/image_classification/dataset_factory.py", line 302, in build
dataset = strategy.distribute_datasets_from_function(self._build)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py", line 1161, in distribute_datasets_from_function
dataset_fn, options)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/mirrored_strategy.py", line 589, in _distribute_datasets_from_function
options)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/input_lib.py", line 169, in get_distributed_datasets_from_function
input_contexts, dataset_fn, options)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/input_lib.py", line 1579, in __init__
input_contexts, self._input_workers, dataset_fn))
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/input_lib.py", line 2327, in _create_datasets_from_function_with_input_context
dataset = dataset_fn(ctx)
File "/usr/local/lib/python3.6/dist-packages/official/vision/image_classification/dataset_factory.py", line 333, in _build
dataset = builder()
File "/usr/local/lib/python3.6/dist-packages/official/vision/image_classification/dataset_factory.py", line 376, in load_records
dataset = tf.data.Dataset.list_files(file_pattern, shuffle=False)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 1230, in list_files
condition, [message], summarize=1, name="assert_not_empty")
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
return target(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/tf_should_use.py", line 247, in wrapped
return _add_should_use_warning(fn(*args, **kwargs),
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 164, in Assert
(condition, "\n".join(data_str)))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Expected 'tf.Tensor(False, shape=(), dtype=bool)' to be true. Summarized data: b'No files matched pattern: train*'
saberkun
arashwan
jackd
husenzhang
laxmareddyp
google-ml-butler[bot]
Prerequisites
Please answer the following question for yourself before submitting an issue.
1. The entire URL of the documentation with the issue
https://github.com/tensorflow/models/tree/master/official/vision/image_classification
2. Describe the issue
I can't reproduce the examples provided in the documentation. These are the steps I'm following:
a) sudo docker run --gpus all -it --rm tensorflow/tensorflow:latest-gpu /bin/bash b) python3 -m pip install --upgrade pip c) pip install tf-models-official d) download config files using curl (configs/examples/resnet/imagenet/gpu.yaml and configs/examples/efficientnet/imagenet/efficientnet-b0-gpu.yaml) e) execute the code provided:
As a result I'm getting:
Changing the command to:
I'm getting:
I don't know how to fix this, I think documentation is not very clear, please help. I wouldn't mind to use the new code base in beta, but there is even less documentation.