Does not log inside parLapply while on Centos or Ubuntu, works fine on Mac

[Naeemkh]

I use the logger package inside a package. One of the functions is being called by parLapply. While using it on Centos (7.6), Red Hat (6.10), or Ubuntu (20.04), the logging commands from that function are not showing up in the .log file. On the Mac system, it works fine and logs as expected. Am I missing something? Here is the file:

https://github.com/fasrc/GPSmatching/blob/develop/R/matching_l1.R

Thanks!

[daroczig]

Please share a minimal reproducible example of that parLapply call to be able to better investigate this, but if I would need to guess, it depends on how you set up the cluster (eg forking or starting new processes).

[Naeemkh]

Thanks, here is a minimal reproducible example.

library(logger)
library(parallel)

# initializing logger
flogger <- layout_glue_generator(format =
                                           paste('{time} {node} {pid} ',
                                                 '{namespace} {fn} ',
                                                 '{level}:  {msg}',
                                                 sep = ""))
log_appender(appender_file("mylog.log"))
log_threshold(logger::DEBUG)
log_layout(flogger)

# functions
myfunction <- function(n){
  logger::log_debug("logging in the function. Started processing n = {n}")
  return(n^3)
}

log_debug("logging with lapply:  ")

# Using function with lapply (which logs)
res_1 <-  lapply(input_list, myfunction)

log_debug("----------------------------------")
log_debug("logging with parLapply:  ")

# Using function with parLapply (which does not log)
nthread <-  4
input_list <- list(seq(10,100,10))
cl <- makeCluster(nthread, type="PSOCK")
res_2 <-  parLapply(cl,input_list,myfunction)
stopCluster(cl)

print(res_1)
print(res_2)

[Naeemkh]

Is there any way to extract the current appender_file (path/name) inside the package?

[daroczig]

Sorry for getting back so late on this :disappointed:

It seems to me that the above setup uses multiple processes, so starting 4 new R session and passing the input_list along with myfunction, but not setting up logger, e.g. the log_appender etc calls.

If you replace makeCluster with makeForkCluster, then it will fork the current process, so all log_appender settings will be preserved on Linux. Alternative solution is setting up the logging environment somehow in the new processes, not (only) in the parent process.

I hope this helps. Closing the ticket, as not something we can solve within logger.