Complex data type error with TensorFlow Functional API

[jeffreyegan]

@NEGU93 first & foremost, thank you for a contribution like this. Development and support for complex-value neural networks is long overdue and much-needed!

I've been starting to explore use of your CVNN library and am encountering an issue out of the gate using your ReadMe as a starting point. Just curious, am I missing something while interacting with tf or did something break on the functional API side? (I'm able to build, train, and test a sequential model based on the ReadMe without issue).

My environment is python 3.8.10, TensorFlow 2.8.0, and CVNN 1.2.13

I've provided my console out and the code example below as well.

Console Out

2022-03-23 16:29:16.489773: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-23 16:29:16.518763: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2022-03-23 16:29:16.518785: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-03-23 16:29:16.519056: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Model: "model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 input_2 (InputLayer)           [(None, 32, 32, 3)]  0           []                               

 complex_conv2d (ComplexConv2D)  (None, 30, 30, 32)  1792        ['input_2[0][0]']                

 complex_conv2d_1 (ComplexConv2  (None, 28, 28, 32)  18496       ['complex_conv2d[1][0]']         
 D)                                                                                               

 complex_max_pooling2d (Complex  (None, 14, 14, 32)  0           ['complex_conv2d_1[1][0]']       
 MaxPooling2D)                                                                                    

 complex_conv2d_transpose (Comp  (None, 28, 28, 5)   1290        ['complex_max_pooling2d[1][0]']  
 lexConv2DTranspose)                                                                              

 concatenate (Concatenate)      (None, 28, 28, 37)   0           ['complex_conv2d_transpose[1][0]'
                                                                 , 'complex_conv2d_1[1][0]']      

 complex_conv2d_2 (ComplexConv2  (None, 26, 26, 4)   2672        ['concatenate[1][0]']            
 D)                                                                                               

 complex_conv2d_3 (ComplexConv2  (None, 24, 24, 4)   296         ['complex_conv2d_2[1][0]']       
 D)                                                                                               

==================================================================================================
Total params: 24,546
Trainable params: 24,546
Non-trainable params: 0
__________________________________________________________________________________________________
2022-03-23 16:29:17.839433: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1228800000 exceeds 10% of free system memory.
Epoch 1/4
Traceback (most recent call last):
  File "example_functional.py", line 37, in <module>
    history = model.fit(train_images, train_labels, epochs=epochs, validation_data=(test_images, test_labels), verbose=1)
  File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py", line 1147, in autograph_handler
    raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/engine/training.py", line 1021, in train_function  *
        return step_function(self, iterator)
    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/engine/training.py", line 1010, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/engine/training.py", line 1000, in run_step  **
        outputs = model.train_step(data)
    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/engine/training.py", line 860, in train_step
        loss = self.compute_loss(x, y, y_pred, sample_weight)
    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/engine/training.py", line 918, in compute_loss
        return self.compiled_loss(
    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/engine/compile_utils.py", line 201, in __call__
        loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/losses.py", line 141, in __call__
        losses = call_fn(y_true, y_pred)
    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/losses.py", line 245, in call  **
        return ag_fn(y_true, y_pred, **self._fn_kwargs)
    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/losses.py", line 1862, in sparse_categorical_crossentropy
        return backend.sparse_categorical_crossentropy(
    File "/home/jeffrey/repos/cvnn/venv/lib/python3.8/site-packages/keras/backend.py", line 5202, in sparse_categorical_crossentropy
        res = tf.nn.sparse_softmax_cross_entropy_with_logits(

    TypeError: Value passed to parameter 'features' has DataType complex64 not in list of allowed values: float16, bfloat16, float32, float64

Example Test Script

import numpy as np
import tensorflow as tf
import cvnn.layers as complex_layers

def get_dataset():
        (train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.cifar10.load_data()
        train_images = train_images.astype(dtype=np.complex64) / 255.0
        test_images = test_images.astype(dtype=np.complex64) / 255.0
        return (train_images, train_labels), (test_images, test_labels)

def get_model():
    inputs = complex_layers.complex_input(shape=(32, 32, 3))  #adjusted input dims to work with available data set
    c0 = complex_layers.ComplexConv2D(32, activation='cart_relu', kernel_size=3)(inputs)
    c1 = complex_layers.ComplexConv2D(32, activation='cart_relu', kernel_size=3)(c0)
    c2 = complex_layers.ComplexMaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid')(c1)
    t01 = complex_layers.ComplexConv2DTranspose(5, kernel_size=2, strides=(2, 2), activation='cart_relu')(c2)
    concat01 = tf.keras.layers.concatenate([t01, c1], axis=-1)

    c3 = complex_layers.ComplexConv2D(4, activation='cart_relu', kernel_size=3)(concat01)
    out = complex_layers.ComplexConv2D(4, activation='cart_relu', kernel_size=3)(c3)
    return tf.keras.Model(inputs, out)

# Assume you already have complex data... example numpy arrays of dtype np.complex64
(train_images, train_labels), (test_images, test_labels) = get_dataset()        # to be done by each user

model = get_model()   # Get your model

# Compile as any TensorFlow model
model.compile(optimizer='adam', metrics=['accuracy'], loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True))
model.summary()

# Train and evaluate
history = model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels), verbose=1)
test_loss, test_acc = model.evaluate(test_images,  test_labels, verbose=2)

[NEGU93]

Hello,

The problem is that keras losses does not work with complex numbers. For this you have 2 options:

• Use the activation functions that although having complex input it has real output (documented here)
• Use my CVNN loss definitions (documented here)

On another note, the example depicted is a small U-NET shaped network designed for segmentation (the output is actually another image, in this case, size 24x24x4) but your labels are for classification tasks, I recommend you implement your own neural network for the task at hand or at least add a flatten and a dense layer at the end.

[jeffreyegan]

@NEGU93 Thank you for the quick and useful response! Admittedly I was a little quick with piecing together your ReadMe file's example just to test out the package, my environment, and get the equivalent of a Hello World function running before I started tailoring it to my application. The pointers and info about handling complex values at the activation and loss functions were what I needed. Thank you again so much for the help!!