High ipTM, low pLDDT

[stianale]

Congrats with an impressive work

I wonder a bit about my results for my protein-dsDNA complex, which are of very high ipTM, but suprisingly low pLDDT. How does this even make sense? I would think that they are at least somewhat related measures.

{
    "plddt": 41.25,
    "gpde": 0.2570812702178955,
    "ptm": 0.9742947816848755,
    "iptm": 0.9466376304626465,
    "chain_ptm": [
        0.9805907011032104,
        0.231978639960289,
        0.22907303273677826
    ],
    "chain_iptm": [
        0.9428915977478027,
        0.8514121770858765,
        0.8502060174942017
    ],
    "chain_pair_iptm": [
        [
            0.0,
            0.9440976977348328,
            0.9416854977607727
        ],
        [
            0.9440976977348328,
            0.0,
            0.7587265968322754
        ],
        [
            0.9416854977607727,
            0.7587265968322754,
            0.0
        ]
    ],
    "chain_pair_iptm_global": [
        [
            0.0,
            0.8971518874168396,
            0.8965488076210022
        ],
        [
            0.8971518874168396,
            0.0,
            0.8508090972900391
        ],
        [
            0.8965488076210022,
            0.8508090972900391,
            0.0
        ]
    ],
    "chain_plddt": [
        51.25,
        23.0,
        22.625
    ],
    "chain_pair_plddt": [
        [
            0.0,
            45.25,
            45.25
        ],
        [
            45.25,
            0.0,
            22.75
        ],
        [
            45.25,
            22.75,
            0.0
        ]
    ],
    "has_clash": false,
    "disorder": 0.0,
    "ranking_score": 0.9521690607070923,
    "num_recycles": 10

[xinshi-chen]

Hi @stianale ,

Thank you for raising this concern, and we appreciate your feedback!

You're absolutely right to point out the apparent discrepancy between high ipTM and low pLDDT. We've observed similar behavior in our model and believe it stems from the fact that the pTM (and ipTM) predictions tend to be "over-confident." As shown in Figure 8 of our technical report, the majority of the "chain pair ipTM" values are clustered around or above 0.4, with very few falling in the 0-0.4 range.

Figure 8: N represents the number of samples.

We hypothesize that this issue arises from the difference between the training stages of the PAE head and the other confidence heads: the PAE head, which computes pTM, is trained in the final stage only. We're actively working on refining the training process to mitigate this over-confidence issue.

In light of this, we consider pTM as a more reliable "relative" measure for comparing different samples rather than an "absolute" accuracy measure. We can use pTM when selecting a sample from many generated ones, but the exact scores are not entirely accurate.

Thanks again for your thoughtful comment!