Keras LSTM 1st Layer Requires 3d Input

[CirdanCapital]

Hi Again,

I want to discuss my implementation. I have a Keras LSTM learner in the first layer. It requires that the X_test input is 3d. This is fine but it breaks my second layer as these (scikit-learn) learners require 2d input. I have consequently modified backend.py as follows:

def _propagate_features(self, task):
        “”"Propagate features from input array to output array.“”"
        p_out, p_in = self.job.predict_out, self.job.predict_in

        # Check for loss of obs between layers (i.e. with blendindex)
        n_in, n_out = p_in.shape[0], p_out.shape[0]
        r = int(n_in - n_out)

        if not issparse(p_in):
            # Simple item setting
            if len(p_in.shape) == 3:
                print(‘p_in.shape before:\n’, p_in.shape)
                print(‘p_out.shape before:\n’, p_out.shape)
                p_in = p_in[:,0,:]
                print(‘p_in.shape after:\n’, p_in.shape)
                p_out = np.concatenate((p_out, p_in), axis=1)
                print(‘p_out.shape after:\n’, p_out.shape)
            else:
                p_out[:, :task.n_feature_prop] = p_in[r:, task.propagate_features]
        else: ....

And this works well i.e. everything works and I see the following output:

p_in.shape before:
(4052, 1, 247)
p_out.shape before:
(4052, 2)
p_in.shape after:
(4052, 247)
p_out.shape after:
(4052, 249)

The problem now is that I go back to my previous problem in that I am looking at the learners in both:

for fitted_estimator in ensemble.layers[i].learners[j].learner:

for fitted_estimator in ensemble.layers[i].learners[j].sub_learner:

for the second layer and 3rd and 4th learners (linear regressions) and I see that their coef_ length is only 2 and not 249 like I expected given that I am trying (see above) to propogate the 247 input to the second layer. I am trying to train the scikit-learn models on the second layer because with a 3d input it seems the only model that I can have in the first layer is the Keras model.

Please try and help me. What I really want is an ensemble with

1. LSTM; and other scikit-learn models (this is working but I don't understand the second layer)
2. My own cost functions (these work)
3. My own simple linear regression where I set intercept to 0 and have a coef_ of 1 to one column in the input data i.e. one of the 247 columns (I think I can do this but see below)

So far, everything is working but I need to work out why these fitted 2nd layer linear regression objects have a coef array of length 2 (when I expect 249) but I know how to change them now (thanks to you) and then I think I am done. I am in London and I am happy to share my code with you. Please help!!! I really need to get this finished.

[CirdanCapital]

False alarm. I changed the code from:

    ensemble.add(ests_1,
            folds= 10,
            propagate_features=[247],
            proba=False)

to

    ensemble.add(ests_1,
            folds= 10,
            propagate_features=[i for i in range(247)],
            proba=False)

And now it works! I just need to change the linear regressions and I am done! Thanks again for your help yesterday and thank you for writing this library. You might want to fix the 3d input issue for Keras' LSTMs when you find time. You can close this issue. Let us know if you would like to come to our office in Knightsbridge and meet the team and I'll buy you a coffee. This is Christian from Cirdan Capital.

[flennerhag]

Hi,

I'm glad it works. A few notes:

• The safe way to manipulate input data is to add a preprocessing instance to you meta-learner layer that flattens the data. In your case, overriding _propagate_features is fine, but in general there is no way of knowing how to merge dimensions.

• As stated in the documentation, the propagate_features argument expects an iterable, such as range(247), or more generally a list of column ids.

• You are using an LSTM, which is a sequential model, but the SuperLearner instance uses standard cross-validation. You may want to use the temporal ensemble class.

Best of luck!