Add many2many implementation of the RNN

[kouroshD]

Previously, I have added one2many implementation of the RNN, now, I would like to have the possibility to add many2many RNN. x2y stands for x time series input and y time series output

[kouroshD]

I was working in this issue in the last two days, and I have implemented the following architecture options:

I have to mention that, in the last one, i.e., many-to-many, and Tx!=Ty, in the second part of it, the input to the next time is the output of the previous time step, so: x^<Tx+i> = yhat^<i> ; s.t., 1<i<Ty

The commits are #157145 and #4e20884d4.

However, I have tested them with low value parameters, i.e., low n_a, Tx, Ty, m_train, m_validation, m_test, since each time running them needs some time.

[kouroshD]

However, while implementing I had an error, which I managed to understand what is the source of it and fix it after some time, but I do not still understand exactly, why there was this error. So, in the following lines of the code:

                for tx in range(self.Tx):
                    x = Lambda(lambda z: z[:, tx, :])(X)
                    x = self.reshapor(x)
                    # Step 2: Perform one step of LSTM_cell
                    a, _, c = self.LSTM_cell(inputs=x, initial_state=[a, c])

                out = self.densor(a)
                x = Lambda(lambda z: z)(out)
                for ty in range(self.Ty):
                    x = self.reshapor(x)
                    # Step 3: Perform one step of LSTM_cell (≈1 line)
                    a, _, c = self.LSTM_cell(inputs=x, initial_state=[a, c])

                    # Step 3: Apply Dense layer to the hidden state output of the LSTM_cell (≈1 line)
                    out = self.densor(a)

                    # Step 4: Append the prediction "out" to "outputs". out.shape = (None, 78) (≈1 line)
                    outputs = outputs + [out]

In the first and second for loops, I was using same variable t to iterate instead of current values of tx and ty. Then I had the following error:

Traceback (most recent call last):
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1619, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 9 of dimension 1 out of bounds. for 'model/lambda/strided_slice' (op: 'StridedSlice') with input shapes: [?,7,2], [3], [3], [3] and with computed input tensors: input[1] = <0 9 0>, input[2] = <0 10 0>, input[3] = <1 1 1>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/.../code/src/main.py", line 90, in <module>
    epochs=epochs, plot_loss_value_obj=plot_losses, verbosity=verbosity)
  File "/.../code/src/RnnKeras.py", line 61, in fit_model
    verbose=verbosity) #, callbacks=[plot_loss_value_obj])
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
    use_multiprocessing=use_multiprocessing)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 342, in fit
    total_epochs=epochs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 128, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 98, in execution_function
    distributed_function(input_fn))
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
    result = self._call(*args, **kwds)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 615, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 497, in _initialize
    *args, **kwds))
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2389, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 85, in distributed_function
    per_replica_function, args=args)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 763, in experimental_run_v2
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1819, in call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 2164, in _call_for_each_replica
    return fn(*args, **kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py", line 292, in wrapper
    return func(*args, **kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 433, in train_on_batch
    output_loss_metrics=model._output_loss_metrics)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 312, in train_on_batch
    output_loss_metrics=output_loss_metrics))
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 253, in _process_single_batch
    training=training))
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 127, in _model_loss
    outs = model(inputs, **kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 778, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 717, in call
    convert_kwargs_to_constants=base_layer_utils.call_context().saving)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 891, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 778, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/keras/layers/core.py", line 846, in call
    result = self.function(inputs, **kwargs)
  File "/.../code/src/RnnKeras.py", line 173, in <lambda>
    x = Lambda(lambda z: z[:, t, :])(X)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 898, in _slice_helper
    name=name)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 1064, in strided_slice
    shrink_axis_mask=shrink_axis_mask)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 9535, in strided_slice
    shrink_axis_mask=shrink_axis_mask, name=name)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 742, in _apply_op_helper
    attrs=attr_protos, op_def=op_def)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 595, in _create_op_internal
    compute_device)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3322, in _create_op_internal
    op_def=op_def)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1786, in __init__
    control_input_ops)
  File "/.../code/venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1622, in _create_c_op
    raise ValueError(str(e))
ValueError: slice index 9 of dimension 1 out of bounds. for 'model/lambda/strided_slice' (op: 'StridedSlice') with input shapes: [?,7,2], [3], [3], [3] and with computed input tensors: input[1] = <0 9 0>, input[2] = <0 10 0>, input[3] = <1 1 1>.

And the error was happening in line x = Lambda(lambda z: z[:, t, :])(X) complaining that it has the value t that is not expected (high value). What I do not still understand is that, t is assigned in the line before using for t in range(self.Tx):, therefore we should not have had problem with the passing by reference or scope (which in python is different from C++).

@traversaro @raffaello-camoriano Do you have any hint in that? I may open an issue eventually in Keras-team repo to understand it better, if they reply!

[kouroshD]

Here is the result of the prediction for many2many architecture prediction, with the following Tx = 40 and Ty=60:

[kouroshD]

I have opened an issue in https://github.com/keras-team/keras/issues/13914 , to ask the Keras community.

[raffaello-camoriano]

@traversaro @raffaello-camoriano Do you have any hint in that?

Hi @kouroshD, I took a look at the previous version of the code which yields problems: https://github.com/dic-iit/element_human-action-intention-recognition/blame/4e20884d47ba3c22c444a7c74833250807d316a3/code/src/RnnKeras.py#L171-L187

I didn't run it, but at a first glance I couldn't see a reason justifying the error you mentioned.

The fact that it was possible to fix it by using 2 different index variables for the independent for loops leaves me puzzled, I don't know.

[kouroshD]

We have done the issue as discussed in the previous comments. However, there was an open question which I opened an issue in https://github.com/keras-team/keras/issues/13914 . I would say, we can close this issue, since the opened issue is not blocking, and we reached the goal.

[traversaro]

Sorry @kouroshD , I had missed my mention. I never used Keras, so unfortunately I can't help you.

[kouroshD]

Sorry @kouroshD , I had missed my mention. I never used Keras, so unfortunately I can't help you.

No worries, thank you in any case @traversaro :) .