brianstock / MixSIAR

A framework for Bayesian mixing models in R:
http://brianstock.github.io/MixSIAR/
94 stars 75 forks source link

Computer crashes in middle of loop how do you start up again? #283

Open naalipalo opened 3 years ago

naalipalo commented 3 years ago

Ive been running 13 models since 10am 2 days ago. I've got 1 left and my work computer is going to auto-restart in 1 hour...I don't think my last model will finish. Which means, though I have generated the output for at least 12 of them, I have not "saved" any jags.mod in the looped iterative process to which I can load that data back in. I can easily add and remove models to an already created jags.mod, but have no idea if we can use the generated output files to put back in enough to run the LOO table analysis. When you stop it in the middle of running, nothing is saved and output_JAGS doesn't exist.

jags.mod <- parLapply(cl, seq_len(n.mod), function(mod) {

  source[[mod]] <- load_source_data(filename=source.filename,
                                    source_factors=NULL,
                                    conc_dep=FALSE,
                                    data_type="means",
                                    mix[[mod]])
  discr[[mod]] <- load_discr_data(filename=discr.filename, mix[[mod]])

 plot_data(filename=paste0("isospace_plot_", mod),
            plot_save_pdf=FALSE,
            plot_save_png=TRUE,
            mix[[mod]], source[[mod]], discr[[mod]], 
            return_obj = TRUE)

  model_filename <- paste0("MixSIAR_model_", mod, ".txt")
  resid_err <- TRUE
  process_err <- TRUE
  write_JAGS_model(model_filename, resid_err, process_err, mix[[mod]], source[[mod]])

   # Can also set custom MCMC parameters
  run <- list(chainLength=700000, burn=300000, thin=500, chains=3, calcDIC=TRUE)

  jags.mod <- run_model(run=run, mix[[mod]], source[[mod]], discr[[mod]], model_filename, alpha.prior=1)

  output_JAGS(jags.mod, mix[[mod]], source[[mod]], output_options=list(
    summary_save = TRUE,                 # Save the summary statistics as a txt file?
    ))
  graphics.off()
  return(jags.mod)
})
brianstock commented 3 years ago

Hello,

If your models take longer than ~1 hour to run, I'd be saving the jags.mod objects individually within the loop as .rds files so you don't lose anything. I probably wouldn't run output_JAGS to reduce the possibility for errors to stop the loop. When they all finish you can load them into R for diagnostics and output.