alizat
commented
3 years ago Dear @mislam5285 , Thanks for reaching out to me.
I sending this message to let you know that I am currently looking at your queries so I could reply to them. Please note that I am quite busy these days, so I am unable to reply right away and will need at least a few days to investigate the matter stated in your first query and to check your code in the second query.
By the way, a quick run of "EnsemKRR with SVD and CV setting S1" gave me much better results than what you got (AUC for FOLD 1 was 0.937), so I am not exactly sure what went wrong on your side. I guess, try re-downloading the code from GitHub and try again (in start.m, modify line 27 to "predictionMethods = {'ensemkrr'};" AND line 33 to "dimRedTechniques = 1;", and press F5)
As for the second query, optimizing the threshold (for converting to 0/1 class labels) using the f-score is fine and commonplace. The G-mean way seems to make sense as well. Note that I haven't checked your code yet. By the way, have you considered an evaluation metric called the MCC (Matthew Correlation Coefficient)?
In the upcoming days, I will have another look at your queries and try to reply as soon as I can.
Yours Sincerely, Ali Ezzat
mislam5285
Dear Sir, I have some queries regarding AUC score reported in your article titled “Drug-target interaction prediction using ensemble learning and dimensionality reduction”
Query 1: I have run your code provided in “https://github.com/alizat/EnsemDT-EnsemKRR”. The AUC score that is computed by “ensemkrr_S1_DimRed1 : predictionMethods is ensemkrr, CV setting is S1, and the dimensionality reduction method is SVD” is :
FOLD 1: 0.873 FOLD 2: 0.878 FOLD 3: 0.879 FOLD 4: 0.874 FOLD 5: 0.875
AUC: 0.876
In the published article on page no.05 and Table 5, the AUC score is reported as 0.942.
If you kindly specify for which hyperparameters setting in grid-search the AUC score 0.942 is obtained.
Sir, I need this for comparing the efficiency of your method with respect to other methods.
Query 2: In your article, only AUC score is given. I want to compute F1 score and G-mean score from your scoring matrix named “predScoresMatrix”. What I have done for computing F1 and Gmean score is given below. I want to know your valuable comments on whether the computed F1 score and Gmean score are correct or not. I would like to present this score in one article for comparison.
We have seen that every ensemble learning-based method is predicting a score between 0 and 1 instead of a 0/1 class label for each test sample. To compute F1 and Gmean scores from the predicted scores we have mapped it to a crisp class label using one threshold value which is optimally determined. Because the classification problem in this ensemble learning-based method has a severe class imbalance, the default threshold, say 0.5, may result in poor performance. The optimal threshold is determined by computing the ROC curve which provides three lists of values called false positive rate(fpr), true positive rate (tpr), and thresholds. These three lists have the same length i.e., for each threshold in the list called thresholds, there is one value in the list called fpr and tpr. We have computed the G-mean score for each threshold. Finally, we have selected the largest G-mean. For the F1 score, the optimal threshold is selected from the precision-recall curve. Corresponding to the optimal threshold value, the precision and recall value is taken for F1 score computation.
My Code in python:
ulr1='./ensemkrr_S1_DimRed1_Ytrue.txt' url2='./ensemkrr_S1_DimRed1_YPredProb.txt'
testy=pd.read_csv(ulr1,delimiter="\t",header=None) yhat=pd.read_csv(url2,delimiter="\t",header=None)
#########################Gmean Score########################################### ###############################################################################
calculate roc curves
fpr, tpr, thresholds = roc_curve(testy, yhat)
calculate the g-mean for each threshold
gmeans = sqrt(tpr * (1-fpr))
locate the index of the largest g-mean
ix = argmax(gmeans) print('Best Threshold=%f, G-Mean=%.3f' % (thresholds[ix], gmeans[ix])) ############################################################################## #########################F1 Score############################################# from sklearn.metrics import precision_recall_curve
calculate roc curves
precision, recall, thresholds = precision_recall_curve(testy, yhat)
convert to f score
fscore = (2 precision recall) / (precision + recall)
locate the index of the largest f score
ix = argmax(fscore) print('Best Threshold=%f, F-Score=%.3f' % (thresholds[ix], fscore[ix]))
Output: Best Threshold=0.074662, G-Mean=0.875 Best Threshold=2.907300, F-Score=0.342
Your valuable comments are important for me because before presenting these scores I want your guidance to know whether I have computed the scores correctly or not.