search

stefan-it / turkish-bert

Turkish BERT/DistilBERT, ELECTRA and ConvBERT models
482 stars 42 forks source link

ValueError: Tensor conversion requested dtype string for Tensor with dtype float32: <tf.Tensor 'args_0:0' shape=() dtype=float32> #20

Closed etetteh closed 3 years ago

etetteh commented 3 years ago

I have built my pretraining data and stored it, the vocab file and config file in my GCS bucket. But when I run the pretraining step:

  --data-dir $DATA_DIR \
  --model-name $MODEL_NAME \
  --hparams $HPARAMS_FILE

I keep getting the following

Running training
================================================================================
2020-11-17 08:09:31.456003: W tensorflow/core/distributed_runtime/rpc/grpc_session.cc:370] GrpcSession::ListDevices will initialize the session with an empty graph and other defaults because the session has not yet been created.
ERROR:tensorflow:Error recorded from training_loop: in converted code:
    relative to /home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python:

    data/ops/readers.py:336 __init__
        filenames, compression_type, buffer_size, num_parallel_reads)
    data/ops/readers.py:296 __init__
        filenames = _create_or_validate_filenames_dataset(filenames)
    data/ops/readers.py:56 _create_or_validate_filenames_dataset
        filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
    framework/ops.py:1184 convert_to_tensor
        return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
    framework/ops.py:1242 convert_to_tensor_v2
        as_ref=False)
    framework/ops.py:1273 internal_convert_to_tensor
        (dtype.name, value.dtype.name, value))

    ValueError: Tensor conversion requested dtype string for Tensor with dtype float32: <tf.Tensor 'args_0:0' shape=() dtype=float32>

Traceback (most recent call last):
  File "electra/run_pretraining.py", line 385, in <module>
    main()
  File "electra/run_pretraining.py", line 381, in main
    args.model_name, args.data_dir, **hparams))
  File "electra/run_pretraining.py", line 344, in train_or_eval
    max_steps=config.num_train_steps)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3035, in train
    rendezvous.raise_errors()
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 136, in raise_errors
    six.reraise(typ, value, traceback)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3030, in train
    saving_listeners=saving_listeners)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1191, in _train_model_default
    features, labels, ModeKeys.TRAIN, self.config)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2857, in _call_model_fn
    config)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3148, in _model_fn
    input_holders.generate_infeed_enqueue_ops_and_dequeue_fn())
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 1428, in generate_infeed_enqueue_ops_and_dequeue_fn
    self._invoke_input_fn_and_record_structure())
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 1525, in _invoke_input_fn_and_record_structure
    host_device, host_id))
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 899, in generate_per_host_enqueue_ops_fn_for_host
    inputs = _Inputs.from_input_fn(input_fn(user_context))
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3001, in _input_fn
    return input_fn(**kwargs)
  File "/home/etetteh/electra/pretrain/pretrain_data.py", line 63, in input_fn
    cycle_length=cycle_length))
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 1990, in apply
    return DatasetV1Adapter(super(DatasetV1, self).apply(transformation_func))
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 1378, in apply
    dataset = transformation_func(self)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/data/experimental/ops/interleave_ops.py", line 94, in _apply_fn
    buffer_output_elements, prefetch_input_elements)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/data/ops/readers.py", line 226, in __init__
    map_func, self._transformation_name(), dataset=input_dataset)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2713, in __init__
    self._function = wrapper_fn._get_concrete_function_internal()
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1853, in _get_concrete_function_internal
    *args, **kwargs)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1847, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2147, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2038, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2707, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2652, in _wrapper_helper
    ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
  File "/home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in converted code:
    relative to /home/etetteh/anaconda3/lib/python3.6/site-packages/tensorflow_core/python:

    data/ops/readers.py:336 __init__
        filenames, compression_type, buffer_size, num_parallel_reads)
    data/ops/readers.py:296 __init__
        filenames = _create_or_validate_filenames_dataset(filenames)
    data/ops/readers.py:56 _create_or_validate_filenames_dataset
        filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
    framework/ops.py:1184 convert_to_tensor
        return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
    framework/ops.py:1242 convert_to_tensor_v2
        as_ref=False)
    framework/ops.py:1273 internal_convert_to_tensor
        (dtype.name, value.dtype.name, value))

    ValueError: Tensor conversion requested dtype string for Tensor with dtype float32: <tf.Tensor 'args_0:0' shape=() dtype=float32>
stefan-it commented 3 years ago

Hi @etetteh ,

could you provide the value for DATA_DIR :thinking:

E.g. when DATA_DIR is set to gs://tr-electra then your .tfrecords must be located under gs://tr-electra/output-512/ - this depends on the value pretrain_tfrecords that you've specified in configure_pretraining.py.

I used the following configuration:

https://github.com/stefan-it/turkish-bert/blob/3cc154da3cc05f3a6db1812d4e00e45077cc4a31/electra/configure_pretraining_base.py#L92-L93

stefan-it commented 3 years ago

Also make sure, that you're using a TensorFlow 1.xx version (I think I trained the models with 1.15).

etetteh commented 3 years ago

Yes that's true. Thanks so much. So the problem was in the configure_pre-training I used a max sequence length of 128 but in the run_pretraining I was using 512 that was what was giving me the problem.

Yes. Also I'm using Tensorflow version 1.15.4, but it's working fine. Thanks a lot.

On Tue, Nov 17, 2020, 16:45 Stefan Schweter notifications@github.com wrote:

Also make sure, that you're using a TensorFlow 1.xx version (I think I trained the models with 1.15).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stefan-it/turkish-bert/issues/20#issuecomment-729232625, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGZQ72H7WMT753WI3W4R6Q3SQLVGXANCNFSM4TYIAPZA .