Open DavidGriscti opened 5 years ago
Hi,
I haven't done any comparison with other deep learning frameworks. That would be interesting to do (especially to detect possible errors in ConvNetSharp) but I currently haven't got much time to do that. Of course I expect mainstream frameworks to be more correct/accurate/performant.
Hi. When I use a small BatchSize ,For example, 100,200... the ConvNetSharp neural network will not converge. But when I use the frameworks like Keras, and use the same small BatchSize(even 1) and network parameters, the Keras neural network will converge well. I can't find the reason...
double[][] traindata = ReadFromCsvFile(@"C:\LD\train111.csv", true);
double[][] trainout = ReadFromCsvFile(@"C:\LD\out111.csv", true);
var featurenum = traindata[0].Length;
var outputnum = trainout[0].Length;
Net<double> net = new ConvNetSharp.Core.Net<double>();
net.AddLayer(new InputLayer(1, 1, featurenum));
net.AddLayer(new FullyConnLayer(128));
net.AddLayer(new ReluLayer());
//net.AddLayer(new DropoutLayer(0.1));
net.AddLayer(new FullyConnLayer(128));
net.AddLayer(new ReluLayer());
// net.AddLayer(new DropoutLayer(0.1));
net.AddLayer(new FullyConnLayer(outputnum));
net.AddLayer(new RegressionLayer());
// Training
int maxEpochs = 150;
int nBatchSize = 100;//traindata.Length;
//train input
var featuresPerRow = traindata[0].Length;
var flattenedInput = new double[traindata.Length * featuresPerRow];
for (int i = 0; i < traindata.Length; i++)
{
var row = traindata[i];
for (int j = 0; j < row.Length; j++)
{
flattenedInput[(i * featuresPerRow) + j] = row[j];
}
}
var volumetrain = BuilderInstance.Volume.From(flattenedInput, new Shape(1, 1, featuresPerRow, traindata.Length));
//train output
var featuresPerRowOut = trainout[0].Length;
var flattenedOut = new double[trainout.Length * featuresPerRowOut];
for (int i = 0; i < trainout.Length; i++)
{
var row = trainout[i];
for (int j = 0; j < row.Length; j++)
{
flattenedOut[(i * featuresPerRowOut) + j] = row[j];
}
}
var volumeout = BuilderInstance.Volume.From(flattenedOut, new Shape(1, 1, featuresPerRowOut, trainout.Length));
var trainer = new SgdTrainer(net) { LearningRate = 0.01, Momentum = 0.9, BatchSize = nBatchSize , L2Decay = 0.001 };
for (int i = 0; i < maxEpochs; i++)
{
trainer.Train(volumetrain, volumeout);
Console.WriteLine("Train Loss:" + trainer.Loss);
}
Console.ReadKey();
Hi,
That's interesting. Can you show your keras code as well? Could you try to set L2Decay to 0 in Keras and in ConvNetSharp? It's not currently used in ConvNetSharp.
Hi, There is keras code ,it is using R: set.seed(1234) library(keras)
root_mean_squared_error= function(y_true, y_pred) { k_sqrt(k_mean(k_square(y_pred - y_true),keepdims = T)) }
metric_rmse <- custom_metric("rmse", function(y_true, y_pred) { 1/(1+k_sqrt(k_mean(k_square(y_pred - y_true), keepdims = )))*100 })
trainf <- read.csv('c:/LD/train111.csv', header = T)
yf <- read.csv('c:/LD/out111.csv', header = T)
train.y=yf$LS train.x=data.matrix((trainf))
gc() filepath="c:/LD/best.hdf5" checkpoint = callback_model_checkpoint(filepath, monitor='val_loss', verbose=1, save_best_only=T, mode='min')
earlyStopping=callback_early_stopping(monitor="val_loss", patience=30, verbose=1, mode='min') callbacks_list = list(checkpoint,earlyStopping)
model <- keras_model_sequential()
r<-ncol(train.x)
model %>% layer_dense(units =128, activation = 'relu', input_shape = c(r)) %>% layer_dropout(rate = 0.0) %>% layer_dense(units = 128,activation = 'relu') %>% layer_dropout(rate = 0.0) %>% layer_dense(units = 1)
model %>% compile( loss = root_mean_squared_error, optimizer = optimizer_sgd(lr = 0.01, momentum = 0.9, decay = 0.01), metrics = metric_rmse )
history <- model %>% fit( train.x, train.y, callbacks=callbacks_list, epochs = 150, batch_size = 8 )
I noticed something in your C# code: the 4th dimension of your input/output data should be the batch size. But I see it's the whole dataset size. You need to adapt the code to call the train method on each batch (if batch size = 1, once for each entry in the dataset).
I should emove the batch size parameter of the trainer because it's confusing: it should deduce it from the shape of the input.
Thank you for pointing out the mistake. Could you write an example code or pseudocode for using the batch and train?
Hi, It's great work you made. How does ConvNetSharp compare to other Deep Learning frameworks like Keras in the accuracy of the prediction of DNN regression? Have you done any tests? Thanks.