UnimplementedError: Fusion is not implemented: [BiasAdd,Tanh] (only when tanh activation functions are used)

[saddamhijazi]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): YES
• OS Platform and Distribution: Linux Ubuntu 18.04.6 LTS)
• TensorFlow version (use command below): 2.4.1
• Python version: Python 3.8.5

I am trying to build a keras model to solve a problem in which I define a custom loss function, the model has tanh activation functions and in the custom loss function I generate random input values and then I do some manipulations to prepare the input of the model that takes 3 inputs and outputs just one output. The loss function is then defined as a function of the output and some of its derivatives with respect to the inputs which are computed using tf.gradients.

The code is shown below

import sys
import os.path
import tensorflow as tf
import math as m
import numpy as np
import scipy.io
from scipy.interpolate import griddata
import time
from itertools import product, combinations
from sklearn.utils import shuffle
from sklearn.preprocessing import MinMaxScaler
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Layer, Activation, Dense, BatchNormalization
from tensorflow.keras.optimizers import Adam

inputs = tf.keras.Input(shape=(3,),dtype='float32')
layer1 = Dense(units=200, activation='tanh',dtype='float32')(inputs)
layer2 = Dense(units=200, activation='tanh',dtype='float32')(layer1)
layer3 = Dense(units=200, activation='tanh',dtype='float32')(layer2)
layer4 = Dense(units=200, activation='tanh',dtype='float32')(layer3)
layer5 = Dense(units=200, activation='tanh',dtype='float32')(layer4)
layer6 = Dense(units=200, activation='tanh',dtype='float32')(layer5)
layer7 = Dense(units=200, activation='tanh',dtype='float32')(layer6)
layer8 = Dense(units=200, activation='tanh',dtype='float32')(layer7)
layer9 = Dense(units=200, activation='tanh',dtype='float32')(layer8)
predictions = Dense(units=1, activation='linear',dtype='float32')(layer9)
model = tf.keras.Model(inputs=inputs, outputs=predictions)

# Define custom loss
def custom_loss():

    # Create a loss function
    def loss(y_true,y_pred):

        # Write here the loss function

        x0 = tf.random.uniform(shape=[2,100],maxval = 1)
        y0 = tf.random.uniform(shape=[2,100],maxval = 1)
        t0 = tf.random.uniform(shape=[100,2],maxval = 1)
        pY0 = tf.transpose(tf.concat([tf.ones([1,y0.shape[1]]) * y0[0,:], tf.zeros([1,y0.shape[1]], dtype='float32'), tf.ones([1,y0.shape[1]]) * y0[1,:]], 0))
        pY1 = tf.transpose(tf.concat([tf.ones([1,y0.shape[1]]) * y0[0,:], tf.ones([1,y0.shape[1]], dtype='float32'), tf.ones([1,y0.shape[1]]) * y0[1,:]], 0))
        pX0 = tf.transpose(tf.concat([tf.zeros([1,x0.shape[1]], dtype='float32'), tf.ones([1,x0.shape[1]]) * x0[0,:], tf.ones([1,x0.shape[1]]) * x0[1,:]], 0))
        pX1 = tf.transpose(tf.concat([tf.ones([1,x0.shape[1]], dtype='float32'), tf.ones([1,x0.shape[1]]) * x0[0,:], tf.ones([1,x0.shape[1]]) * x0[1,:]], 0))
        pIC = tf.concat([t0, tf.zeros([t0.shape[0],1], dtype='float32')], 1)
        pF = tf.random.uniform(shape=[1000,3],maxval = 1)
        uX0 = model(pX0, training=True)
        uX1 = model(pX1, training=True)
        uY0 = model(pY0, training=True)
        uY1 = model(pY1, training=True)
        uIC = model(pIC, training=True)
        u = model(pF, training=True)

        # compute the derivatives of u
        u_grad = tf.gradients(u, pF)[0]
        u_Dot = tf.gradients(u, pF)[0]
        u_t = u_grad[:,2]
        u_x = u_grad[:,0]
        u_y = u_grad[:,1]
        u_xx = (tf.gradients(u_x, pF))[0][:,0]
        u_yy = (tf.gradients(u_y, pF))[0][:,1]

        return tf.reduce_mean(tf.square(uX0)) + tf.reduce_mean(tf.square(uX1)) + \
        tf.reduce_mean(tf.square(uY0)) + tf.reduce_mean(tf.square(uY1)) + \
        tf.reduce_mean(tf.square(uIC - tf.transpose(tf.math.sin(pi*pIC[:,0])*tf.math.sin(pi*pIC[:,1]) * tf.ones([1,1], dtype='float32')) )) + \
        tf.reduce_mean(tf.square(u_t-u_xx-u_yy))

    # Return a function
    return loss

# Compile the model
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4) ,
              loss=custom_loss(),
              # Call the loss function with the selected layer
              metrics=['accuracy'])

input_T = tf.linspace([0.0, 0.0, 0.0], [1.0, 1.0, 1.0], 1000) # We define a dummy input in order to use the fit method
output_T = tf.linspace([0.0], [1.0], 1000) # We define a dummy output
model.fit(input_T, output_T,epochs=10, batch_size = input_T.shape[0])

I tried to run the code on my PC using the CPU and I got the following error

`File "NewTest_HeatEq.py", line 169, in <module>
    model.fit(input_T, output_T,epochs=10, batch_size = input_T.shape[0])
  File "/home/saddam/anaconda3/envs/tf2/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "/home/saddam/anaconda3/envs/tf2/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/home/saddam/anaconda3/envs/tf2/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 888, in _call
    return self._stateless_fn(*args, **kwds)
  File "/home/saddam/anaconda3/envs/tf2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2942, in __call__
    return graph_function._call_flat(
  File "/home/saddam/anaconda3/envs/tf2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/home/saddam/anaconda3/envs/tf2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 555, in call
    outputs = execute.execute(
  File "/home/saddam/anaconda3/envs/tf2/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError:  Fusion is not implemented: [BiasAdd,Tanh]
     [[node loss/model/dense/Tanh_3 (defined at NewTest_HeatEq.py:100) ]] [Op:__inference_train_function_4519]

Function call stack:
train_function

However, the code worked when I ran it on a GPU machine, and also the code works when I run it on my PC changing only the activation functions to sigmoid ones. I find the behavior not normal, do you have any idea on what has caused that and how to fix the issue.

[tilakrayal]

@saddamhijazi , I ran the code shared and face a different error, please find the gist here and share all dependencies to replicate the issue or share a colab gist with the reported error.

[tilakrayal]

Also please try to test your code in latest tf v2.7 and let us know if the issue still persists.Thanks!

[saddamhijazi]

@saddamhijazi , I ran the code shared and face a different error, please find the gist here and share all dependencies to replicate the issue or share a colab gist with the reported error.

Sorry for the error, I forgot to include this line

pi = tf.constant(m.pi)

Now the code runs in a normal way without error, I don't know if the problem is caused by the tensorflow version on my laptop. May I ask you if it is possible to install the latest version using anaconda ?

In the following repository the latest available version is 2.6 not 2.7

https://repo.anaconda.com/pkgs/main/linux-64/

[tilakrayal]

@saddamhijazi , Installation issues within the Anaconda environment are tracked in the Anaconda repo.Thanks!

[saddamhijazi]

I have managed to run the code without problems on TensorFlow 2.7 but it still does not work on 2.4, for me it is not a problem I can use the 2.7 version. The issue could be closed.

Thank you very much.

[tilakrayal]

@saddamhijazi , There is a high possibility that issue was fixed with later TF versions.Glad the suggestion worked for you.Please feel feel to move this issue to closed status as issue got resolved in latest stable version.Thanks!

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No