Stranger Strings

[Sum02dean]

Talk with Damian on strange predictive behavior for a minority of incorrect predictions.

• for each observation vector 1s or 0s , if has a score element, vector gets a 1
• more ones, model should give a higher score
• the more zeros, model should decrease the score
• neg = decrease in diff(string-xgboost)
• pos = increase in diff(string-xgboost)
• xgboost is knocking down the probability heavily for scores which have a lot of evidence for a positive interaction (many 1's in binary vector)
• Furthermore, the difference in probability is knocked don by a consistent value across many samples.

Solution steps:

• Given an example list of the protein names, demonstrating the strange behavior, measure the class entropy and predictive variance.

• Note, the proteins are not necessarily encoded closely in genome space

• Sanity check, first check if these observation have any pathway membership, if so, check to see if the labels coincide with the high scores (y==1)