Can we extract the performance metrics from a customized model?

[lxhuang7]

Thanks for this awesome tool! We are using it with our own data to create a customized model. My question is how we can obtain the model performance info from the customized model? Is there any utility function in celltypist we can use or we have to write our own code to evaluate how well our model performs? Thanks!

[ChuanXu1]

@lxhuang7, you need to write your own code to assess the performance of the model, such as 1) testing it on an independent new dataset (with ground-truth cell types available) to obtain the precision, recall, F1, AUROC, etc, or 2) train-test splitting your training dataset to obtain these metrics (and to find the best parameter combo).

[niehu2018]

@ChuanXu1 , Should I train a model using all my annotated data to get the final model? or use cv method to train model on training data, then validate on validating data, then test the model on testing data, then pick up a model with the best performance as the final model?