Conv1D Conversion Normalization Issue

[otcathatsya]

Using the development version, I encounter an issue when trying to convert a Conv1D network. From what I can tell all layers used are supported, but when using normalization, a KeyError is encountered:

Normalizing parameters...
Traceback (most recent call last):
  File "C:\Users\Catherine\anaconda3\envs\msc-pc-rc\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Catherine\anaconda3\envs\msc-pc-rc\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Catherine\anaconda3\envs\msc-pc-rc\Scripts\snntoolbox.exe\__main__.py", line 7, in <module>
  File "C:\Users\Catherine\anaconda3\envs\msc-pc-rc\lib\site-packages\snntoolbox\bin\run.py", line 49, in main
    run_pipeline(config)
  File "C:\Users\Catherine\anaconda3\envs\msc-pc-rc\lib\site-packages\snntoolbox\bin\utils.py", line 94, in run_pipeline
    normalize_parameters(parsed_model, config, **normset)
  File "C:\Users\Catherine\anaconda3\envs\msc-pc-rc\lib\site-packages\snntoolbox\conversion\utils.py", line 153, in normalize_parameters
    scale_fac = scale_facs[layer.name]
KeyError: '0Conv1D_12x16'

When normalization is instead turned off and x/y test are provided in normalized form it does proceed but ends up with with an optimization loop failure in backend:

Current accuracy of batch:
100.00%_2022-08-18 17:18:09.710720: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.715268: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.719344: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.723677: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.727767: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_100.00%_2022-08-18 17:18:09.738498: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.742617: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.746918: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_100.00%_2022-08-18 17:18:09.755846: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.761203: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.768469: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_100.00%_2022-08-18 17:18:09.776898: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.782365: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.786573: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.791726: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_2022-08-18 17:18:09.797365: W tensorflow/core/data/root_dataset.cc:247] Optimization loop failed: CANCELLED: Operation was cancelled
100.00%_
Batch 1 of 1 completed (100.0%)
Moving accuracy of SNN (top-1, top-1): 100.00%, 100.00%.
Moving accuracy of ANN (top-1, top-1): 0.00%, 100.00%.

The configuration file and model used are:

[paths]
dataset_path = %(path_wd)s/climate-ts
filename_ann = lstm-cnn

[simulation]
simulator: INI
duration: 20
dt: 1
num_to_test: 3
batch_size = 16

[output]
log_vars = {'neuron_operations_b_t', 'synaptic_operations_b_t'}
plot_vars = {}

[tools]
evaluate_ann = True
parse = True
normalize = False
convert = True
simulate = True
serialise_only = False

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 input (InputLayer)          [(16, 14, 4)]             0

 0Conv1D_12x16 (Conv1D)      (16, 12, 16)              208

 1Conv1D_10x16 (Conv1D)      (16, 10, 16)              784

 2Flatten_160 (Flatten)      (16, 160)                 0

 3Dense_1 (Dense)            (16, 1)                   161

=================================================================

I experimented with normalization off as the input data is created using keras' timeseries_dataset_from_array after applying a scaler. When leaving it on scaling was moved after dataset creation for all input data but x_norm.

[otcathatsya]

Update; the KeyError was caused by me forgetting to clean the log directory. It does however still run into an Optimization loop failed: CANCELLED: Operation was cancelled, giving a wrong accuracy of 100% no matter the model after the first simulation step. Data shape is (16, 14, 4) for X and (16, 1) on Y for time series forecasting.

[rbodo]

As for the Optimization loop message: It is only a Warning, not an error, and concerns the optimization, which shouldn't matter at this point (the conversion assumes training is completed). Did you perhaps add a "fit" call somewhere? You should be able to get rid of the warning by either setting model.trainable = True (source) or making sure your batch size (16) is not larger than your entire dataset, i.e. the 3 test samples specified in your config (source).

As for the accuracy, how do you know it is wrong? Is it possible that with only three test samples you just might have gotten lucky? Perhaps test on a larger number of samples. Also, you probably need to modify the accuracy calculation somehow. By default the toolbox assumes a classification task with one-hot labels, and I don't know if your time series forecasting works that way.