How to handle conditional LSTM with input conv1d layers?

[GuitarML]

I would like to use a conditional LSTM for a guitar amplifier model, where the conditional part would be user controls such as gain/drive on an amplifier (reference https://github.com/GuitarML/GuitarLSTM repo). Would the cond_rnn layer work if my model has two conv1d layers prior to the conditional LSTM? For example:

    model = Sequential()
    model.add(Conv1D(conv1d_filters, 12, strides=conv1d_strides, activation=None, padding='same',input_shape=(input_size,1)))
    model.add(Conv1D(conv1d_filters, 12, strides=conv1d_strides, activation=None, padding='same'))
    model.add(LSTM(hidden_units))
    model.add(Dense(1, activation=None))

Where "input_size" is something like 100, for an input shape of (100,1) to the first conv1d layer. Would it work to replace the LSTM here with your conditional rnn? Or is a different approach required where the condition is introduced at the first layer.

Thanks!

[philipperemy]

@GuitarML A conditional LSTM takes the same input X as the LSTM. So if it works for an LSTM it will work for a conditional LSTM (in terms of shape). The conditional LSTM takes an extra input Y that has shape (batch_size, cond_size). For example, if you have 10 user controls, you can encode them as one-hot vectors. Continuous variables also work for conditioning. So long story short, the answer is yes.

[GuitarML]

@philipperemy thanks for the response! My concern was more about the first two conv1d layers in my model, if I used the conditional LSTM does it have to be the first layer? If not, how would I introduce the conditional variables, or would I need a functional model instead of a sequential model to accomplish that.

[philipperemy]

You can use a functional model. Sequential will probably work too.

[philipperemy]

I correct what I have just said. Yes use a functional model if CondLSTM is not the first layer of your model ;)

[GuitarML]

Thanks for the clarification!

[saadsaifse]

@GuitarML have you tried this? got any sample code? I am having trouble specifying the inputs

[saadsaifse]

Ok, the following did work

input1 = tf.keras.Input(shape=(X_train.shape[1:]))
input2 = tf.keras.Input(shape=X_train_aux.shape[1:])

conv = tf.keras.layers.Conv1D(filters=12, kernel_size=4, padding='causal',
                               activation='linear',
                               input_shape=input1.shape[1:],
                               groups=1)(input1)
cond = cond_rnn.ConditionalRNN(units=512, cell='LSTM', return_sequences=True)([conv, input2])
dense1 = tf.keras.layers.Dense(50, activation="tanh")(cond)
dense2 = tf.keras.layers.Dense(25, activation="tanh")(dense1)
output = tf.keras.layers.Dense(1, activation="linear")(dense2)

model = tf.keras.Model(inputs=[input1, input2], outputs=output)

model.compile(loss="mse", optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
              metrics=[tf.keras.metrics.RootMeanSquaredError()])

history = model.fit([X_train, X_train_aux], Y_train,
                batch_size=64, epochs=10000, shuffle=False, callbacks=[callback, tensorboard_callback])