Combine fireTS library with neupy library for NARX network based on Levenberg Marquardt

[TobiasEl]

Hi. I want to create a NARX (Nonlinear Autoregressive with exogenous variables) model based on LM (Levenberg Marquardt) method.

Since this two method are not implemented in keras, I search for the library fireTs (for NARX) and neupy (for LM).

I'm using the sample code for both libraries:

fireTS (NARX): fireTS neupy (LM): Neupy

and I combine them:

from fireTS.models import NARX
from sklearn.ensemble import RandomForestRegressor
import numpy as np
from neupy import algorithms
from neupy.layers import *

x = np.array([[1, 2], [3, 4]])
y = np.array([[1], [0]])

#y = np.ravel(y) <-just to avoid a warning (the error is the same without comment)

network = Input(2) >> Sigmoid(3) >> Sigmoid(1)
optimizer = algorithms.LevenbergMarquardt(network)

mdl1 = NARX(
    optimizer, #I change random forest for LevenbergMarquardt
    auto_order=2,
    exog_order=[2, 2],
    exog_delay=[1, 1])

mdl1.fit(x, y)

ypred1 = mdl1.predict(x, y)

ypred1

But I'm having this error in .fit method:

ZeroDivisionError Traceback (most recent call last)

in () 18 exog_delay=[1, 1]) 19 ---> 20 mdl1.fit(x, y) 21 22 ypred1 = mdl1.predict(x, y) /usr/local/lib/python3.7/dist-packages/neupy/utils/iters.py in count_minibatches(inputs, batch_size) 22 23 def count_minibatches(inputs, batch_size): ---> 24 return int(math.ceil(count_samples(inputs) / batch_size)) 25 26 ZeroDivisionError: division by zero

Any solution?

[jxx123]

This seems to be the issue of neupy. batch_size was not set to 0 by default in neupy's fit method causing the division by zero issue. After setting batch_size, mdl1.fit(x, y, batch_size=1). The fit method is able to pass, but predict method still needs to set the batch_size correctly to get the results, but for now, fireTS does not pass along the parameters to the internal estimator (i.e. your optimizer). I tried to modify fireTS code (https://github.com/jxx123/fireTS/blob/3d1abb051003a1327b31e721acea796cbbd62c39/fireTS/core.py#L133) by setting batch_size=1, but still somehow errors out in some other places in neupy:

File "/home/jinyu/miniconda3/envs/firets-neupy/lib/python3.7/site-packages/neupy/layers/graph.py", line 532, in predict
    return np.concatenate(outputs, axis=0)
  File "<__array_function__ internals>", line 6, in concatenate
ValueError: need at least one array to concatenate

I would suggest figure out how neupy with LM method works, then try integrate it with fireTS.

[TobiasEl]

Thanks for answer my question. Do you know an alternative way to create a NARX method based on LM? I found this implementation of LM in https://github.com/fabiodimarco/tf-levenberg-marquardt

And here is the code I tried:

import tensorflow as tf
import numpy as np
import levenberg_marquardt as lm
from fireTS.models import NARX
import numpy as np

input_size = 20000
batch_size = 1000

x_train = np.linspace(-1, 1, input_size, dtype=np.float64)
y_train = np.sinc(10 * x_train)

x_train = tf.expand_dims(tf.cast(x_train, tf.float32), axis=-1)
y_train = tf.expand_dims(tf.cast(y_train, tf.float32), axis=-1)

train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_dataset = train_dataset.shuffle(input_size)
train_dataset = train_dataset.batch(batch_size).cache()
train_dataset = train_dataset.prefetch(tf.data.experimental.AUTOTUNE)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(20, activation='tanh', input_shape=(1,)),
    tf.keras.layers.Dense(1, activation='linear')])

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
    loss=tf.keras.losses.MeanSquaredError())

model_wrapper = lm.ModelWrapper(model)

model_wrapper.compile(
    optimizer=tf.keras.optimizers.SGD(learning_rate=1.0),
    loss=lm.MeanSquaredError())

mdl1 = NARX(
    model_wrapper,
    auto_order=2,
    exog_order=[2, 2],
    exog_delay=[1, 1])

mdl1.fit(train_dataset,epoch=10)
ypred1 = mdl1.predict(x=x_test, y=y_test)

ypred1

But I have this error:

AttributeError                            Traceback (most recent call last)

<ipython-input-15-221d19103ad1> in <module>()
     37     auto_order=2,
     38     exog_order=[2, 2],
---> 39     exog_delay=[1, 1])
     40 
     41 mdl1.fit(train_dataset,epoch=10)

2 frames

/usr/local/lib/python3.7/dist-packages/fireTS/core.py in __init__(self, base_estimator, **base_params)
     14 
     15     def __init__(self, base_estimator, **base_params):
---> 16         self.base_estimator = base_estimator.set_params(**base_params)
     17 
     18     def set_params(self, **params):

AttributeError: 'ModelWrapper' object has no attribute 'set_params'

I tried to delete the params exog_order, exog_delay, but NARX expected them.

[jxx123]

The error shows your model_wrapper does not implement the sklearn BaseEstimator interface (hence no method called 'set_params'). Your model_wrapper is based on Keras API, which is not supported by fireTS. To make it work, you want to wrap your model_wrapper again with the sklearn BaseEstimator API.