Open mimxrt opened 3 years ago
@mimxrt not sure exactly why it does not work. This should work in theory:
m_in = K.Input(shape=(None, 2), batch_size=batch_size, ragged=True)
I'm not sure why but it doesn't work for me. I also made sure to get the latest version of TensorFlow this time! Can you try and run this example?
import sys
import tcn
import numpy as np
import tensorflow as tf
import tensorflow.keras as K
batch_size=2
print(f"sys.version: {sys.version}")
print(f"tf.__version__: {tf.__version__}")
X = [
np.array([[1, 1], [2, 2], [3, 3], [4, 4], [5, 5]]),
np.array([[1, 1], [2, 2], [3, 3]]),
np.array([[1, 1]]),
np.array([[1, 1], [2, 2], [3, 3], [4, 4], [5, 5], [6, 6]]),
]
Y = np.array([6, 1, 2, 4])
ds_raw_X = tf.data.Dataset.from_tensor_slices(tf.ragged.constant(X, inner_shape=(2,)))
ds_raw_Y = tf.data.Dataset.from_tensor_slices(Y)
ds_raw = tf.data.Dataset.zip((ds_raw_X, ds_raw_Y))
print("Raw dataset:")
for e in ds_raw.take(2):
print(f"{(e[0].shape, e[1].shape)}")
print("...\n")
ds = ds_raw.batch(batch_size)
print("Final dataset:")
for e in ds.take(2):
print(f"{(e[0].shape, e[1].shape)}")
print("...\n")
m_in = K.Input(shape=(None, 2), batch_size=batch_size, ragged=True) # note the added ragged=True parameter
m_out = tcn.TCN(
nb_filters=32,
return_sequences=False
)(m_in)
m_out = K.layers.Dense(1)(m_out)
model = K.Model([m_in], m_out)
print(model.summary(), end="\n\n")
model.compile(optimizer=K.optimizers.Adam(), loss=K.losses.MeanSquaredError())
model.fit(ds, epochs=1)
Output for me:
sys.version: 3.7.10 (default, Feb 26 2021, 13:06:18) [MSC v.1916 64 bit (AMD64)]
tf.__version__: 2.4.1
Raw dataset:
(TensorShape([5, 2]), TensorShape([]))
(TensorShape([3, 2]), TensorShape([]))
...
Final dataset:
(TensorShape([2, None, 2]), TensorShape([2]))
(TensorShape([2, None, 2]), TensorShape([2]))
...
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-2-c63b87df74e5> in <module>
39 nb_filters=32,
40 return_sequences=False
---> 41 )(m_in)
42 m_out = K.layers.Dense(1)(m_out)
43
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in __call__(self, *args, **kwargs)
950 if _in_functional_construction_mode(self, inputs, args, kwargs, input_list):
951 return self._functional_construction_call(inputs, args, kwargs,
--> 952 input_list)
953
954 # Maintains info about the `Layer.call` stack.
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in _functional_construction_call(self, inputs, args, kwargs, input_list)
1089 # Check input assumptions set after layer building, e.g. input shape.
1090 outputs = self._keras_tensor_symbolic_call(
-> 1091 inputs, input_masks, args, kwargs)
1092
1093 if outputs is None:
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in _keras_tensor_symbolic_call(self, inputs, input_masks, args, kwargs)
820 return nest.map_structure(keras_tensor.KerasTensor, output_signature)
821 else:
--> 822 return self._infer_output_signature(inputs, args, kwargs, input_masks)
823
824 def _infer_output_signature(self, inputs, args, kwargs, input_masks):
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in _infer_output_signature(self, inputs, args, kwargs, input_masks)
861 # TODO(kaftan): do we maybe_build here, or have we already done it?
862 self._maybe_build(inputs)
--> 863 outputs = call_fn(inputs, *args, **kwargs)
864
865 self._handle_activity_regularization(inputs, outputs)
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\autograph\impl\api.py in wrapper(*args, **kwargs)
668 except Exception as e: # pylint:disable=broad-except
669 if hasattr(e, 'ag_error_metadata'):
--> 670 raise e.ag_error_metadata.to_exception(e)
671 else:
672 raise
TypeError: in user code:
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tcn\tcn.py:319 call *
x, skip_out = layer(K.cast(x, 'float32'), training=training)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tcn\tcn.py:155 call *
x = layer(x, training=training) if training_flag else layer(x)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:1012 __call__ **
outputs = call_fn(inputs, *args, **kwargs)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\layers\convolutional.py:246 call
inputs = array_ops.pad(inputs, self._compute_causal_padding(inputs))
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\util\dispatch.py:201 wrapper
return target(*args, **kwargs)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\ops\array_ops.py:3422 pad
result = gen_array_ops.pad(tensor, paddings, name=name)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\ops\gen_array_ops.py:6484 pad
"Pad", input=input, paddings=paddings, name=name)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\op_def_library.py:525 _apply_op_helper
raise err
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\op_def_library.py:522 _apply_op_helper
preferred_dtype=default_dtype)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\profiler\trace.py:163 wrapped
return func(*args, **kwargs)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\ops.py:1540 convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\constant_op.py:339 _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\constant_op.py:265 constant
allow_broadcast=True)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\constant_op.py:283 _constant_impl
allow_broadcast=allow_broadcast))
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\tensor_util.py:553 make_tensor_proto
"supported type." % (type(values), values))
TypeError: Failed to convert object of type <class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'> to Tensor. Contents: tf.RaggedTensor(values=Tensor("Placeholder:0", shape=(None, 2), dtype=float32), row_splits=Tensor("Placeholder_1:0", shape=(3,), dtype=int64)). Consider casting elements to a supported type.
@mimxrt yes it does not work for me too. But seems like it's a Keras issue here.
Thanks for testing! I created another script that shows that it works by simply using tf.keras.layers.LSTM
instead of TCN
(only one line difference):
import sys
import tcn
import numpy as np
import tensorflow as tf
import tensorflow.keras as K
batch_size=2
print(f"sys.version: {sys.version}")
print(f"tf.__version__: {tf.__version__}")
X = [
np.array([[1, 1], [2, 2], [3, 3], [4, 4], [5, 5]]),
np.array([[1, 1], [2, 2], [3, 3]]),
np.array([[1, 1]]),
np.array([[1, 1], [2, 2], [3, 3], [4, 4], [5, 5], [6, 6]]),
]
Y = np.array([6, 1, 2, 4])
ds_raw_X = tf.data.Dataset.from_tensors(tf.ragged.constant(X, inner_shape=(2,)))
ds_raw_Y = tf.data.Dataset.from_tensors(tf.constant(Y))
ds_raw = tf.data.Dataset.zip((ds_raw_X, ds_raw_Y))
ds = ds_raw
ds = ds.unbatch().batch(batch_size)
print("\n\n===========================")
print("===== Works with LSTM =====")
print("===========================\n")
m_in = K.Input(shape=[None, 2], batch_size=batch_size, ragged=True)
m_out = K.layers.LSTM(32)(m_in)
m_out = K.layers.Dense(1)(m_out)
model = K.Model([m_in], m_out)
print(model.summary(), end="\n\n")
model.compile(optimizer=K.optimizers.Adam(), loss=K.losses.MeanSquaredError())
model.fit(ds, epochs=5)
print("\n\n===========================")
print("===== Fails with TCN ======")
print("===========================\n")
m_in = K.Input(shape=[None, 2], batch_size=batch_size, ragged=True)
m_out = tcn.TCN(32)(m_in)
m_out = K.layers.Dense(1)(m_out)
model = K.Model([m_in], m_out)
print(model.summary(), end="\n\n")
model.compile(optimizer=K.optimizers.Adam(), loss=K.losses.MeanSquaredError())
model.fit(ds, epochs=5)
Output:
sys.version: 3.7.10 (default, Feb 26 2021, 13:06:18) [MSC v.1916 64 bit (AMD64)]
tf.__version__: 2.4.1
===========================
===== Works with LSTM =====
===========================
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(2, None, 2)] 0
_________________________________________________________________
lstm (LSTM) (2, 32) 4480
_________________________________________________________________
dense (Dense) (2, 1) 33
=================================================================
Total params: 4,513
Trainable params: 4,513
Non-trainable params: 0
_________________________________________________________________
None
Epoch 1/5
2/2 [==============================] - 1s 5ms/step - loss: 15.8364
Epoch 2/5
2/2 [==============================] - 0s 4ms/step - loss: 15.1551
Epoch 3/5
2/2 [==============================] - 0s 5ms/step - loss: 14.4901
Epoch 4/5
2/2 [==============================] - 0s 5ms/step - loss: 13.8392
Epoch 5/5
2/2 [==============================] - 0s 5ms/step - loss: 13.2030
===========================
===== Fails with TCN ======
===========================
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-1-aef4100541a3> in <module>
42
43 m_in = K.Input(shape=[None, 2], batch_size=batch_size, ragged=True)
---> 44 m_out = tcn.TCN(32)(m_in)
45 m_out = K.layers.Dense(1)(m_out)
46 model = K.Model([m_in], m_out)
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in __call__(self, *args, **kwargs)
950 if _in_functional_construction_mode(self, inputs, args, kwargs, input_list):
951 return self._functional_construction_call(inputs, args, kwargs,
--> 952 input_list)
953
954 # Maintains info about the `Layer.call` stack.
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in _functional_construction_call(self, inputs, args, kwargs, input_list)
1089 # Check input assumptions set after layer building, e.g. input shape.
1090 outputs = self._keras_tensor_symbolic_call(
-> 1091 inputs, input_masks, args, kwargs)
1092
1093 if outputs is None:
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in _keras_tensor_symbolic_call(self, inputs, input_masks, args, kwargs)
820 return nest.map_structure(keras_tensor.KerasTensor, output_signature)
821 else:
--> 822 return self._infer_output_signature(inputs, args, kwargs, input_masks)
823
824 def _infer_output_signature(self, inputs, args, kwargs, input_masks):
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in _infer_output_signature(self, inputs, args, kwargs, input_masks)
861 # TODO(kaftan): do we maybe_build here, or have we already done it?
862 self._maybe_build(inputs)
--> 863 outputs = call_fn(inputs, *args, **kwargs)
864
865 self._handle_activity_regularization(inputs, outputs)
~\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\autograph\impl\api.py in wrapper(*args, **kwargs)
668 except Exception as e: # pylint:disable=broad-except
669 if hasattr(e, 'ag_error_metadata'):
--> 670 raise e.ag_error_metadata.to_exception(e)
671 else:
672 raise
TypeError: in user code:
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tcn\tcn.py:319 call *
x, skip_out = layer(K.cast(x, 'float32'), training=training)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tcn\tcn.py:155 call *
x = layer(x, training=training) if training_flag else layer(x)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:1012 __call__ **
outputs = call_fn(inputs, *args, **kwargs)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\layers\convolutional.py:246 call
inputs = array_ops.pad(inputs, self._compute_causal_padding(inputs))
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\util\dispatch.py:201 wrapper
return target(*args, **kwargs)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\ops\array_ops.py:3422 pad
result = gen_array_ops.pad(tensor, paddings, name=name)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\ops\gen_array_ops.py:6484 pad
"Pad", input=input, paddings=paddings, name=name)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\op_def_library.py:525 _apply_op_helper
raise err
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\op_def_library.py:522 _apply_op_helper
preferred_dtype=default_dtype)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\profiler\trace.py:163 wrapped
return func(*args, **kwargs)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\ops.py:1540 convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\constant_op.py:339 _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\constant_op.py:265 constant
allow_broadcast=True)
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\constant_op.py:283 _constant_impl
allow_broadcast=allow_broadcast))
C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\tensor_util.py:553 make_tensor_proto
"supported type." % (type(values), values))
TypeError: Failed to convert object of type <class 'tensorflow.python.ops.ragged.ragged_tensor.RaggedTensor'> to Tensor. Contents: tf.RaggedTensor(values=Tensor("Placeholder:0", shape=(None, 2), dtype=float32), row_splits=Tensor("Placeholder_1:0", shape=(3,), dtype=int64)). Consider casting elements to a supported type.
Is there anything that I need to configure properly to make it work with TCNs?
I'd say it's because we have a custom function to build() the network. The support for RaggedTensor should be about updating this function. That's my gut feeling.
Maybe it also has something todo with this issue: https://github.com/tensorflow/tensorflow/issues/42417?
However, I also don't really know what's the technical difficulty as in my understanding everything that supports tf.keras.layers.Masking
, should also be able to support tf.RaggedTensor
. In the worst case you would have to manually pad the ragged tensor data to the maximum length and mask the padded values (granted, you would have to know / compute the length in advance).
Yes, I think you are right here. It's related. I'll see what I can do ;)
However, I also don't really know what's the technical difficulty as in my understanding everything that supports
tf.keras.layers.Masking
, should also be able to supporttf.RaggedTensor
. In the worst case you would have to manually pad the ragged tensor data to the maximum length and mask the padded values (granted, you would have to know / compute the length in advance).
@mimxrt Unfortunately I do not think this is true since in the issue that you mentioned it appears that convolutions cannot be applied to tensors with variable length. I've experienced this with ConvLSTM2D as well here: https://github.com/tensorflow/tensorflow/issues/48678
@maximgeller I think you might have misunderstood my statement: I meant if masking is supported (i.e., everything that supports tf.keras.layers.Masking
) then it should technically be possible to support ragged tensors. The latter still means that someone has to implement it.
The correct way to do that is here: https://www.tensorflow.org/guide/keras/masking_and_padding
Doc: https://stackoverflow.com/questions/55176818/how-to-support-masking-in-custom-tf-keras-layers-layer
Currentrly, Keras TCN does not support
tf.RaggedTensor
input. It would be very useful for speeding up training for variying length time series inputs. ~As I understand it, there is no way to batch sequences of different lengths except when usingtf.RaggedTensor
~*. See the following minimal example of the current state:*EDIT: Masking is another option, but still...
Output:
I think this is expected to fail in all cases because I did not add the
ragged=True
parameter. When adding this parameter, the error is as expected (Layer tcn_1 does not support RaggedTensors as input
):Output: