[BUG] REDCOMETS crashes on UWaveZ

[TonyBagnall]

Describe the bug

REDCOMET classifier crashes on the one UWave dataset, but not on others. Some form of sampling issue maybe? @zy18811 any ideas?

Steps/Code to reproduce the bug

from aeon.classification.dictionary_based import REDCOMETS
from aeon.datasets import load_classification
trainX, trainy = load_classification("UWaveGestureLibraryZ", split="TRAIN")
testX, testy = load_classification("UWaveGestureLibraryZ", split="TEST")

red = REDCOMETS()
red.fit(trainX, trainy)
acc = red.score(testX,testy)
print("REDCOMETS accuracy: ", acc)

Expected results

prints accuracy

Actual results

2024-01-15 14:38:54.269164: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1/1 [==============================] - 0s 361ms/step
1/1 [==============================] - 0s 332ms/step
1/1 [==============================] - 0s 330ms/step
1/1 [==============================] - 0s 340ms/step
WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x00000252D72ADE50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
1/1 [==============================] - 0s 327ms/step
Traceback (most recent call last):
  File "C:\Code\aeon\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3460, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-dde97c65104f>", line 1, in <module>
    runfile('C:\\Code\\aeon\\aeon\\local\\classification_debug.py', wdir='C:\\Code\\aeon\\aeon\\local')
  File "C:\Program Files\JetBrains\PyCharm 2020.3\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2020.3\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:\Code\aeon\aeon\local\classification_debug.py", line 16, in <module>
    acc = red.score(testX,testy)
  File "C:\Code\aeon\aeon\classification\base.py", line 251, in score
    return accuracy_score(y, self.predict(X), normalize=True)
  File "C:\Code\aeon\aeon\classification\base.py", line 190, in predict
    return self._predict(X)
  File "C:\Code\aeon\aeon\classification\dictionary_based\_redcomets.py", line 370, in _predict
    [self.classes_[i] for i in self._predict_proba(X).argmax(axis=1)]
  File "C:\Code\aeon\aeon\classification\dictionary_based\_redcomets.py", line 387, in _predict_proba
    return self._predict_proba_unvivariate(np.squeeze(X))
  File "C:\Code\aeon\aeon\classification\dictionary_based\_redcomets.py", line 425, in _predict_proba_unvivariate
    self._parallel_sax(self.sax_transforms, X), self.sax_clfs
  File "C:\Code\aeon\aeon\classification\dictionary_based\_redcomets.py", line 590, in _parallel_sax
    sax_parallel_res = Parallel(n_jobs=self.n_jobs, backend=self.parallel_backend)(
  File "C:\Code\aeon\venv\lib\site-packages\joblib\parallel.py", line 1085, in __call__
    if self.dispatch_one_batch(iterator):
  File "C:\Code\aeon\venv\lib\site-packages\joblib\parallel.py", line 901, in dispatch_one_batch
    self._dispatch(tasks)
  File "C:\Code\aeon\venv\lib\site-packages\joblib\parallel.py", line 819, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "C:\Code\aeon\venv\lib\site-packages\joblib\_parallel_backends.py", line 208, in apply_async
    result = ImmediateResult(func)
  File "C:\Code\aeon\venv\lib\site-packages\joblib\_parallel_backends.py", line 597, in __init__
    self.results = batch()
  File "C:\Code\aeon\venv\lib\site-packages\joblib\parallel.py", line 288, in __call__
    return [func(*args, **kwargs)
  File "C:\Code\aeon\venv\lib\site-packages\joblib\parallel.py", line 288, in <listcomp>
    return [func(*args, **kwargs)
  File "C:\Code\aeon\aeon\classification\dictionary_based\_redcomets.py", line 588, in _sax_wrapper
    return np.squeeze(sax.fit_transform(X))
  File "C:\Code\aeon\aeon\transformations\collection\base.py", line 162, in fit_transform
    Xt = self._fit_transform(X=X_inner, y=y_inner)
  File "C:\Code\aeon\aeon\transformations\collection\base.py", line 329, in _fit_transform
    return self._transform(X, y)

Versions

No response

[zy18811]

@TonyBagnall hmmm, strange. I'll have a look!

[TonyBagnall]

its only this dataset, ran on the other 111 fine, so some weird edge case. Looks like an empty resample or perhaps a single class resample, not sure how it works internally tbh, there is something parallel going on with SFA :) Reproduced the above on windows and linux though

[zy18811]

@TonyBagnall

Found the problem! Happy to say its not my code 😛 (I think...)

Line 117 of the code for SAX calls:

X = scipy.stats.zscore(X, axis=-1)

Every single value is identical in the 931th sample of the test split for UWaveZ, -0.99841144, which makes scipy.stats.zscore(X) return an array of NaNs which then breaks things during SAX. This is presumably due to dividing by the standard deviation which is zero.

This can be minimally verified like so:

import scipy
import numpy as np

from aeon.datasets import load_classification

testX, _ = load_classification("UWaveGestureLibraryZ", split="TEST")

testX_931 = testX[930]
print(f"All values the same: {np.all(testX_931 == -0.99841144)}")

z_scores = scipy.stats.zscore(testX[930], axis=-1)
print(f"All values NaN: {np.all(np.isnan(z_scores))}")

Other z-norm implementations, e.g. sklearn.preprocessing.scale(), avoid this problem by returning an all zero array in this case, so something like that is possibly the most obvious solution. scale() isn't a drop in replacement sadly, as it wouldn't be able to handle multivariate input (can only handle 2 axes so won't play nice with numpy 3D arrays).

While looking into this I have also found a small issue in the REDCOMETS implementation which is that SAX is z-normed while SFA is not (both should be) so I'll make a quick fix for that and submit a PR.

[TonyBagnall]

good find, we had a similar issue with shapelets, I think we have our own z-norm that protects against this and another weird numab feature @baraline? I'll look later

[zy18811]

@TonyBagnall think this is what you were looking for?

[TonyBagnall]

I think it was a numba function. In any case, red comets runs now on uwave. thanks for the fix, can close this now