Specify number of cores for cross-validation

ansonrel commented 4 years ago

Hi,

First of all, thanks a lot for your package and the documentation. I'm trying to run the main function saverx on a shared Linux server (18.04.2 LTS) but I'm having troubles limiting the number of cores that are used during the [1] "Cross-validation round: 1" step. I have tried to use the ncore argument and to set a tensorflow session but it doesn't seem to work. The function still uses all cores of the server.

library(tensorflow)
library(keras)
library(SAVERX)

config <- tf$ConfigProto(intra_op_parallelism_threads = 6L,
                         inter_op_parallelism_threads = 6L)
session = tf$Session(config = config)
k_set_session(session)

res <- saverx(input.file.name = "data.rds", data.species = "Human", 
              ncores = 6, clearup.python.session = TRUE)

Do you know how I could limit the CPU usage ?

Best, Anthony

jingshuw commented 4 years ago

The ncores argument controls the number of cores used in the R computation of the shrinkage step of SAVER-X. The tensorflow backend of Keras automatically uses all available CPU cores. We will add an argument to control the number of cores used in Tensorflow in next version of SAVER-X

ansonrel commented 4 years ago

Perfect, thanks for your response and the future update. Just fyi or if anyone wants to control the CPU utilization in the meantime, the following chunk

library(tensorflow)
library(keras)

config <- tf$ConfigProto(intra_op_parallelism_threads = 6L,
                         inter_op_parallelism_threads = 6L)
session = tf$Session(config = config)
k_set_session(session)

does effectively control the number of cores used by saverx. Just make sure that you define your conda environment correctly with reticulate, if you use one (that's why it did not work for me in the first place).

jingshuw / SAVERX

Specify number of cores for cross-validation #7