results of the simulation experiments

[WhirlFirst]

Hi,

Thank you for this impressive work! I am pretty interested in your simulation experiments. I found the processed PBMC data from the figshare. But I didn't find the results such as acr_design.h5ad for reproducing the results in pbmc_design_comparison.ipynb Could you please provide the final output results of the simulation experiments?

[emdann]

Hi @WhirlFirst, thanks for your interest in our work!

I didn't share all the intermediate .h5ad objects for each of the simulations because these would be > 100 large objects with very similar data. However, you should be able to regenerate the results by running this:

python run_oor_design.py PBMC_merged.normal.subsample500cells.clean_celltypes.h5ad ${name-of-oor-cell-population} ${design ID}

I am in the process of cleaning and refactoring this code for the publication (the latest much expanded version is on branch revision-1.0), which will expand substantially the documentation, but please let me know if there's anything you think is missing that would be helpful here. e.g. would it be useful if we shared the output tables used to generate figures (i.e. nhood level milo DA predictions + fraction of OOR cells in csv)? Or other intermediate output (maybe only latent dimensions + cell level metadata for each simulation)?

[WhirlFirst]

Thank you for your timely reply! It will be very helpful for me if you could share the output tables that are used for reproducing the figures. Could you also share the classifier_outs_all data frame used in the OOR classification section?

[emdann]

I've added the main milo results from the simulations to Figshare (see README for info). The classifier_outs_all dataframe is now in metadata.

Hope this helps, thanks for the feedback!

[WhirlFirst]

Thanks! I tried the OOR_simulations_remove.AUPRC_results_all.csv file you provided. I found it seems to achieve a higher AUPRC of the "AR" and "CR" settings, compared with the results reported in Figure 2D of your bioRxiv paper. Did I compare with the wrong figure or you got new results about simulation experiments? Here is my result:

[emdann]

Yes, we've re-run this analysis including small fixes, and different simulation scenarios. The revised version will be published soon (hopefully in 3-4 weeks). If you want to reproduce the results from the biorxiv I would use the old version of the code.

[WhirlFirst]

Cool! I just noticed that you updated the new analysis code. I will try it later. Thanks.