search

JiahuiYu / generative_inpainting

DeepFill v1/v2 with Contextual Attention and Gated Convolution, CVPR 2018, and ICCV 2019 Oral
http://jiahuiyu.com/deepfill/
Other
3.23k stars 781 forks source link

error while trying to train. #524

Open khalidpandit opened 2 years ago

khalidpandit commented 2 years ago

Traceback (most recent call last): File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Need minval < maxval, got 0 >= -384 [[{{node random_uniform_4}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train.py", line 102, in trainer.train() File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/neuralgym/train/multigpu_trainer.py", line 59, in train super().train() File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/neuralgym/train/trainer.py", line 130, in train cb.run(sess, step) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/neuralgym/callbacks/secondary_multigpu_trainer.py", line 24, in run self.train() File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/neuralgym/train/multigpu_trainer.py", line 59, in train super().train() File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/neuralgym/train/trainer.py", line 138, in train feed_dict=self.context['feed_dict']) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Need minval < maxval, got 0 >= -384 [[node random_uniform_4 (defined at /home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'random_uniform_4': File "train.py", line 76, in 'model': model, 'FLAGS': FLAGS, 'data': data, 'loss_type': 'd'}, File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/neuralgym/callbacks/secondary_multigpu_trainer.py", line 20, in init MultiGPUTrainer.init(self, primary=False, context) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/neuralgym/train/multigpu_trainer.py", line 25, in init super().init(self.context) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/neuralgym/train/trainer.py", line 38, in init self._train_op, self._loss = self.train_ops_and_losses() File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/neuralgym/train/multigpu_trainer.py", line 75, in train_ops_and_losses gpu_id=gpu, *graph_def_kwargs) File "train.py", line 18, in multigpu_graph_def FLAGS, images, FLAGS, reuse=True) File "/DATA/scratch/khalid/gen_inpainting/inpaint_model.py", line 140, in build_graph_with_losses bbox = random_bbox(FLAGS) File "/DATA/scratch/khalid/gen_inpainting/inpaint_ops.py", line 118, in random_bbox [], minval=FLAGS.vertical_margin, maxval=maxt, dtype=tf.int32) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/ops/random_ops.py", line 243, in random_uniform shape, minval, maxval, seed=seed1, seed2=seed2, name=name) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_random_ops.py", line 922, in random_uniform_int seed=seed, seed2=seed2, name=name) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(args, **kwargs) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/home/khalid/miniconda3/envs/research/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init self._traceback = tf_stack.extract_stack()