could support Dropout layer?

[xinsuinizhuan]

It's easy to overfit, so add some dropout layer could solved this problem?

[josephjaspers]

Hi! How are you doing?

I haven't worked on this in quite some bit. (Though I have thought about splitting apart the Vector library and the neural-network library)!

I could look into adding drop out. (It should just be adding a masking layer which actually shouldn't be too hard) though I am not sure when I will start working on it!

I imagine Tensoflow/Pytorche would be an easier way to tackle your problem though :)

[xinsuinizhuan]

It is a good news for your reply! I am fine, too. This fordcast item need to continues，I write this item with C++, but the Tensoflow/Pytorche is python. The overfit problem need to solve, I got some fordcast struct, as:

1、 model = Sequential() model.add(LSTM(units=50, return_sequences=True, input_shape=(forecast_features_set.shape[1], 1))) model.add(Dropout(0.2)) model.add(LSTM(units=50, return_sequences=True)) model.add(Dropout(0.2)) model.add(LSTM(units=50, return_sequences=True)) model.add(Dropout(0.2)) model.add(LSTM(units=50)) model.add(Dropout(0.2)) model.add(Dense(units = 1)) model.compile(optimizer = 'adam', loss = 'mean_squared_error')

2、 model = Sequential()

layer 1: LSTM

model.add(LSTM( input_dim=1, output_dim=150, return_sequences=True)) model.add(Dropout(0.2))

layer 2: LSTM

model.add(LSTM(output_dim=200, return_sequences=False)) model.add(Dropout(0.2))

layer 3: dense

linear activation: a(x) = x

model.add(Dense(output_dim=1, activation='linear'))

show model

model.summary()

compile the model

sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)

model.compile(loss='mse', optimizer="rmsprop")

train the model

model.fit(X_train, y_train, batch_size=512, nb_epoch=100, validation_split=0.05, verbose=2)

save model

model.save('../model/dwtlstm'+type+'.h5')

3、

design network

model = Sequential() model.add(LSTM(64, input_shape=(train_X.shape[1], train_X.shape[2]))) model.add(Dense(1)) model.compile(loss='mae', optimizer='adam')

fit network

history = model.fit(train_X, train_y, epochs=50, batch_size=72, validation_data=(test_X, test_y), verbose=2, shuffle=False)

plot history

pyplot.plot(history.history['loss'], label='train') pyplot.plot(history.history['val_loss'], label='test') pyplot.legend() pyplot.show()

4、 model = Sequential() layers = [1, 75, 100, prediction_steps] model.add(LSTM(layers[1], input_shape=(None, layers[0]), return_sequences=True)) # add first layer model.add(Dropout(0.2)) # add dropout for first layer model.add(LSTM(layers[2], return_sequences=False)) # add second layer model.add(Dropout(0.2)) # add dropout for second layer model.add(Dense(layers[3])) # add output layer model.add(Activation('linear')) # output layer with linear activation start = time.time() model.compile(loss="mse", optimizer="rmsprop") print('Compilation Time : ', time.time() - start) return model

5、

build the model

model = Sequential()

layer 1: LSTM

model.add(LSTM( input_dim=1, output_dim=50, return_sequences=True)) model.add(Dropout(0.2))

layer 2: LSTM

model.add(LSTM(output_dim=100, return_sequences=False)) model.add(Dropout(0.2))

layer 3: dense

linear activation: a(x) = x

model.add(Dense(output_dim=1, activation='linear'))

compile the model

model.compile(loss="mse", optimizer="rmsprop")

train the model

model.fit(X_train, y_train, batch_size=512, nb_epoch=50, validation_split=0.05, verbose=1)

to some extent, the Dropout layer could solved the overfit, so i try to turn to you.

[xinsuinizhuan]

and how about train with gpu？ I use gpu ，so many compile errors, cuda 11.0

[xinsuinizhuan]

how about Dropout and gpu？