Metrics don't work with categorical / one-hot outputs

[lnicola]

Apologies, this might actually be my egregious lack of experience in this domain.

I've set up a little network to play with:

# xs_training.shape is (255722, 6, 7)
# y_validation.shape is (694001,)
model = Sequential([
  Conv1D(128, 3, activation='relu', padding='same', input_shape=(xs_training.shape[1], xs_training.shape[2])),
  MaxPooling1D(2, padding='same'),
  Conv1D(256, 3, activation='relu', padding='same'),
  MaxPooling1D(2, padding='same'),
  Flatten(),
  Dense(256, activation='relu'),
  Dropout(0.2),
  Dense(256, activation='relu'),
  Dropout(0.2),
  Dense(len(d), activation='softmax'),
])
model.summary()
model.compile(
  optimizer=Nadam(),
  loss='categorical_crossentropy',
  metrics=['accuracy', CohenKappa(len(d))],
)
model.fit(
  xs_training,
  to_categorical(y_training),
  epochs=10,
  batch_size=64,
  validation_data=(xs_validation, to_categorical(y_validation))
)

This works if I only use accuracy as a metric, but if I add CohenKappa I get:

NotImplementedError: in user code:

    /usr/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py:805 train_function  *
        return step_function(self, iterator)
    /home/grayshade/.local/lib/python3.9/site-packages/tensorflow_addons/metrics/cohens_kappa.py:205 result  *
        weight_mtx = tf.ones([nb_ratings, nb_ratings], dtype=tf.float32)
    /usr/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py:201 wrapper  **
        return target(*args, **kwargs)
    /usr/lib/python3.9/site-packages/tensorflow/python/ops/array_ops.py:3120 ones
        output = _constant_if_small(one, shape, dtype, name)
    /usr/lib/python3.9/site-packages/tensorflow/python/ops/array_ops.py:2804 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:5 prod

    /usr/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3030 prod
        return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out,
    /usr/lib/python3.9/site-packages/numpy/core/fromnumeric.py:87 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    /usr/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:852 __array__
        raise NotImplementedError(

    NotImplementedError: Cannot convert a symbolic Tensor (strided_slice_1:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

I suppose the problem is that CohenKappa doesn't work with one-hot vectors (Accuracy somehow detects them and calls CategoricalAccuracy, doesn't it?).

[AakashKumarNain]

I suppose the problem is that CohenKappa doesn't work with one-hot vectors

It actually does. The only case where the current codebase doesn't work with OHE is when you convert OHE in case of a binary classification problem. Can you please confirm the number of labels in your data?

[lnicola]

I have 49 labels, from 0 to 48.

Would a smaller example you could try help here?

[lnicola]

Here's an example you can run:

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.utils import to_categorical
import tensorflow_addons
from tensorflow_addons.metrics import CohenKappa

x_training = np.array([[1, 1], [2, 2], [3, 3]])
x_validation = np.array([[4, 4], [5, 5], [6, 6]])
y_training = np.array([0, 1, 2])
y_validation = np.array([2, 2, 2])

model = Sequential([
  Dense(5, input_shape=(2, )),
  Dense(3, activation='softmax'),
])
model.compile(
  loss='categorical_crossentropy',
  #metrics=['categorical_accuracy'],
  metrics=['categorical_accuracy', CohenKappa(3)],
)
model.fit(
  x_training,
  to_categorical(y_training),
  epochs=10,
  batch_size=64,
  validation_data=(x_validation, to_categorical(y_validation))
)

[DragonPG2000]

I tried the code example and it works albeit with Cohen Kappa score as 0.

[lnicola]

Maybe I'm running a different version? tensorflow-addons 0.13.0.dev0 from pip. EDIT: It's not from pip, for some reason that one didn't build for me. I built it from source.