biolog benchmarks custom models

[arianccbasile]

Dear Daniel, I would like to ask for your support regarding the use of Biolog benchmarks.ipynb. I tried to use it to check whether or not the metrics (Specificity, F1-score, Precision, Accuracy, Sensitivity) of a model change after the use of a putative gap fill approach and in which measure. It worked well when I used Biolog for the gap-filled models of the organisms you also used for your benchmarks, using fbc2 flavor options. However, when I used Biolog with a new model reconstructed with CarveMe, False positive and True positive result to be null both before and after the gap fill.

[{'Accuracy': 0.5163934426229508, 'F1-score': 0.0, 'FN': 59, 'FP': 0, 'Precision': 0, 'Sensitivity': 0.0, 'Specificity': 1.0, 'TN': 63, 'TP': 0}, {'Accuracy': 0.5163934426229508, 'F1-score': 0.0, 'FN': 59, 'FP': 0, 'Precision': 0, 'Sensitivity': 0.0, 'Specificity': 1.0, 'TN': 63, 'TP': 0}]

I'm pretty sure it cannot be possible because it also happens in the model created with CarveMe without attempting gap filling. Do you think the problem could be in the medium definition, in the handling of models, or what else?

Sincerely, Arianna

[cdanielmachado]

Hi Arianna,

It seems that all simulations fail. I suspect it has to do with the SBML flavor.

You used the --fbc2 flag in CarveMe, right?

Did you also change the notebook script to replace cobra with fbc2 (or bigg) in this part of the code?

    model_list = [
        ('iML1515', '../models/original/iML1515.xml', 'bigg'),
        ('CarveMe', '../models/CarveMe/Ecoli_K12_MG1655.xml', 'cobra'),
        ('modelSEED', '../models/modelSEED/ecoli.xml', 'seed')
    ]```