Loop to train multiple small NN models

[AlexSiormpas]

I want to train ~400 small Neural Networks (3Layer, 5 neurons each) over a data set. I've constructed a loop to achieve this and inserted a save function at the end of the loop to save R data at the end of each step. At the early steps of the loop, things run smoothly but as loop progress, RAM allocation is increasing and the speed of my laptop is starting to slow down. Currently, it takes ~18 hours to finish the full process end-to-end. Is there anything I can do to clean up memory so I can make the process run smoother?

for(i in 1:400){
... filter data dynamically per i...
model <- keras_model_sequential() %>%
            layer_dense(units = 5, activation = "relu", input_shape = dim(Predictors)[2]) %>%
            layer_dense(units = 5, activation = "relu") %>%
            layer_dense(units = 5, activation = "relu") %>%
            layer_dense(units = 1)
        model %>% compile(loss = "mean_absolute_percentage_error",
                          optimizer = optimizer_rmsprop(),
                          metrics = list("mean_absolute_percentage_error"))
        model}
    model <- build_model()
    model %>% summary()
  #Train the model parameters and stop early once validation is not improving further
    print_dot_callback <- callback_lambda(on_epoch_end = function(epoch, logs) {
                                          if (epoch %% 80 == 0) cat("\n")
                                          cat(".")})
    epochs <- 500
    early_stop <- callback_early_stopping(monitor = "val_loss", patience = 20)
    history <- model %>% fit(
        Predictors,
        Predict,
        epochs = epochs,
        validation_split = 0.2,
        verbose = 0,
        callbacks = list(early_stop, print_dot_callback))
    plot(history, metrics = "mean_absolute_percentage_error", smooth = FALSE) + coord_cartesian(xlim = c(0, 500), ylim = c(0, 100))
    MAPE<-history$metrics$val_mean_absolute_percentage_error%>%tail(20)%>%mean()

#---- Summarize MAPE and generate a prediction ----
    Summary[i,]<-c( tickers[i],
                    as.numeric(Predict[length(Predict)]),
                        model%>%predict(PredictorsT)%>%as.numeric(),
                        MAPE)
#---- End of the loop. Remove temporary objects and save teh data locally ----
rm(early_stop,history,Predictors,PredictorsT,print_dot_callback,epochs,MAPE,maxvec,minvec,model,Predict,build_model,DataML)
    save.image(file='.RData')
}

[dfalbel]

I would run each model in a different process using callr

[AlexSiormpas]

@dfalbel Thanks for the pointer. So in terms of implementation steps would it be like :

• build a custom process of the full model fitting
• loop over and call the custom process within each iteration

Would parallel computation help me here if I also replace the basic for(i in 1:400){...} with foreach(i = 1:400) %dopar% {...}