search

bytedance / neurst

Neural end-to-end Speech Translation Toolkit
Other
298 stars 45 forks source link

speech transformer example error #42

Open ParkGiBum opened 2 years ago

ParkGiBum commented 2 years ago

I am trying to follow example on speech transfomer with must-c 1.0 en-de dataset

after preprocessing with command

./examples/speech_transformer/must-c/02-audio_feature_extraction.sh ../../data/dataset de --untar

./examples/speech_transformer/must-c/03-preprocess.sh ../mosesdecoder/ ../../data/dataset de

I tried

python3 -m neurst.cli.run_exp \ --config_paths /home/sylee/data/dataset/asr_st/de/asr_training_args.yml,/home/sylee/data/dataset/asr_st/de/asr_validation_args.yml \ --hparams_set speech_transformer_s \ --model_dir /home/sylee/data/dataset/asr_st/de/asr_benchmark

but it keeps making tensor conversion error as follow

sorry for elementary question

TypeError: Failed to convert object of type <class 'neurst.training.revised_dynamic_loss_scale.RevisedDynamicLossScale'> to Tensor. Contents: <neurst.training.revised_dynamic_loss_scale.RevisedDynamicLossScale object at 0x7f6b141da970>. Consider casting elements to a supported type.
Traceback (most recent call last):
  File "/home/sylee/.conda/envs/test2/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/sylee/.conda/envs/test2/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/sylee/neurst/neurst/neurst/cli/run_exp.py", line 127, in <module>
    cli_main()
  File "/home/sylee/neurst/neurst/neurst/cli/run_exp.py", line 123, in cli_main
    app.run(_main, argv=["pseudo.py"])
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/home/sylee/neurst/neurst/neurst/cli/run_exp.py", line 118, in _main
    run_experiment(args, remaining_argv)
  File "/home/sylee/neurst/neurst/neurst/cli/run_exp.py", line 107, in run_experiment
    entry.run()
  File "/home/sylee/neurst/neurst/neurst/exps/trainer.py", line 294, in run
    history = keras_model.fit(
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/engine/training.py", line 1184, in fit
    tmp_logs = self.train_function(iterator)
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 933, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 759, in _initialize
    self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3066, in _get_concrete_function_internal_garbage_collected
    graph_function, _ = self._maybe_define_function(args, kwargs)
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3463, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3298, in _create_graph_function
    func_graph_module.func_graph_from_py_func(
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1007, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 668, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 994, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/engine/training.py:853 train_function  *
        return step_function(self, iterator)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/engine/training.py:842 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/distribute/distribute_lib.py:1286 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/distribute/distribute_lib.py:2849 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/distribute/mirrored_strategy.py:670 _call_for_each_replica
        return mirrored_run.call_for_each_replica(
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/distribute/mirrored_run.py:104 call_for_each_replica
        return _call_for_each_replica(strategy, fn, args, kwargs)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/distribute/mirrored_run.py:246 _call_for_each_replica
        coord.join(threads)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/training/coordinator.py:389 join
        six.reraise(*self._exc_info_to_raise)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/six.py:703 reraise
        raise value
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/training/coordinator.py:297 stop_on_exception
        yield
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/distribute/mirrored_run.py:346 run
        self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/engine/training.py:835 run_step  **
        outputs = model.train_step(data)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/engine/training.py:791 train_step
        self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py:520 minimize
        grads_and_vars = self._compute_gradients(
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/mixed_precision/loss_scale_optimizer.py:676 _compute_gradients
        loss = self.get_scaled_loss(loss)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/mixed_precision/loss_scale_optimizer.py:644 get_scaled_loss
        return loss * tf.cast(self.loss_scale, loss.dtype)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/mixed_precision/loss_scale_optimizer.py:906 __getattribute__
        return object.__getattribute__(self, name)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/keras/mixed_precision/loss_scale_optimizer.py:568 loss_scale
        return tf.convert_to_tensor(self._loss_scale)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py:206 wrapper
        return target(*args, **kwargs)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:1430 convert_to_tensor_v2_with_dispatch
        return convert_to_tensor_v2(
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:1436 convert_to_tensor_v2
        return convert_to_tensor(
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/profiler/trace.py:163 wrapped
        return func(*args, **kwargs)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:1566 convert_to_tensor
        ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/framework/constant_op.py:346 _constant_tensor_conversion_function
        return constant(v, dtype=dtype, name=name)
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/framework/constant_op.py:271 constant
        return _constant_impl(value, dtype, shape, name, verify_shape=False,
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/framework/constant_op.py:288 _constant_impl
        tensor_util.make_tensor_proto(
    /home/sylee/.conda/envs/test2/lib/python3.9/site-packages/tensorflow/python/framework/tensor_util.py:551 make_tensor_proto
        raise TypeError("Failed to convert object of type %s to Tensor. "

    TypeError: Failed to convert object of type <class 'neurst.training.revised_dynamic_loss_scale.RevisedDynamicLossScale'> to Tensor. Contents: <neurst.training.revised_dynamic_loss_scale.RevisedDynamicLossScale object at 0x7f6b141da970>. Consider casting elements to a supported type.
zhaocq-nlp commented 2 years ago

@ParkGiBum Hi, could you please provide the version of your TensorFlow? The NeurST now works well on tensorflow==2.4.1

zhaocq-nlp commented 2 years ago

@ParkGiBum I inspected the installed packages of my environment and found maybe numpy<=1.19.5 should work.