Question about simChef

[rcannood]

Hi @tiffanymtang @jpdunc23 @PhilBoileau !

I'm still in the process of reviewing the JOSS submission in https://github.com/openjournals/joss-reviews/issues/6156. One of the things I'm struggling to understand a little bit is what simChef's core functionality is.

---

Is simChef designed to be able to run benchmarking experiments using simulation models, methods and metrics? The core functionality of the method is then not the DGPs, Methods and Evaluators themselves, but rather simChef is the glue that allows users to perform and visualise this benchmark?

---

Just so I can understand it better; would dyngen be what you consider a DGP. If you look at Supp. Fig. 3 (see below), would AUROC and AUPR then be the Evaluators and SSN, LIONESS and PySCENIC be considered methods?

Supplementary Figure 3: dyngen allows benchmarking Cell-specific Network Inference (CSNI) methods. A: A cell is simulated using the global gene regulatory network (GRN, top left). However, at any particular state in the simulation, only a fraction of the gene regulatory interactions are active. B: CSNI methods were executed to predict the regulatory interactions that are active in each cell specif- ically. Using the ground-truth cell-specific GRN, the performance of each method was quantified on 42 dyngen datasets. Box plots denote the 𝑄0 to 𝑄4 quartile values. Significance of differences between methods was tested with pairwise non-parametric paired two-sided Durbin-Conover tests, Holm corrected for multiple testing. Test statistics and p-values for all combinations are reported in the Source Data file.

---

If this is indeed the case, what could make it easier for me to understand the aim of simChef is to refer to a study where simChef is being used, or showcase a real-life use case for simChef. I know the manuscript already has R code to showcase what simChef can do, but based on this example I find it hard to understand everything.

---

Is it only possible to evaluate all of the executions with future? Do I understand correctly that this framework only allows executing code locally -- as opposed to on an HPC or other cloud infrastructure?

What worries me a little bit is that a benchmark of trajectory inference methods I did had so many methods and evaluators that it was not possible to run this on a single computer in a decent amount of time.

[rcannood]

I found this case study and the corresponding documentation in the "Get started" page of simChef. This confirms my previous suspicions of what simChef is and does.

I also found a response to my question on the vignette related to computing experimental replicates in parallel. I'll create a separate issue to follow up on this.

Closing this issue since I found answers to my own questions :)

[jpdunc23]

@rcannood we'd love to see simChef used in conjunction with cool packages like dyngen!