search

hkmztrk / DeepDTA

220 stars 108 forks source link

Some questions about metrics #9

Closed OilW closed 5 years ago

OilW commented 5 years ago

Hello，I run the code inside "source" on Windows and it works well. But now I want to get more metrics of this model. There is two questions and I hope to get some helps from you.

1) In "run_experiments.py" there is a funtion named "cindex_score" to calculate Concordance Index . And "emetrics.py" also have a function "get_cindex" to get CI. What's the differences between the two functions?

2) I noticed that there is another two metrics function in "emetrics.py", get_aupr and get_rm2. But it seems that they are not used in the code. I try to use them by changing the code in function "build_combined_categorical",from interactionModel.compile(optimizer='adam', loss='mean_squared_error', metrics=[cindex_score]) to interactionModel.compile(optimizer='adam', loss='mean_squared_error', metrics=[cindex_score , get_aupr]) But the result is image and if I add rm2 as a metric, code is interactionModel.compile(optimizer='adam', loss='mean_squared_error', metrics=[cindex_score , get_rm2]) the result is image I wonder how to use the two metric functions correctly.

Sorry for disturbing you. And looking Forward to your reply! Best wishes!

OilW commented 5 years ago

P.S. I try to get a new metric "auc-roc" with help of Stackoverflow.But they are all failed to run.

Is there something need to be pay more attention?

hkmztrk commented 5 years ago

@OilW thanks a lot for your interest in our work!

1- In "run_experiments.py" there is a funtion named "cindex_score" to calculate Concordance Index . And "emetrics.py" also have a function "get_cindex" to get CI. What's the differences between the two functions?

-- cindex_score is modified version so that in can be used as a Keras metric - it operates on tensors. get_cindex simply computes the metric on predicted vs true labels and it can be used after the model predicts the labels.

2- Therefore, it is not possible to directly use the metrics defined in "emetric.py" as you used:

interactionModel.compile(optimizer='adam', loss='mean_squared_error', metrics=[cindex_score , get_rm2]) Because here get_rm2 is not designed to work on tensors like cindex_score. If you want to use that metric in Keras during training, you should convert it in a way so that it works wilth Tensors as well.

I can however make an arrangement so that these other two metrics can be calculated after each prediction. It might take me a while for the update though.

OilW commented 5 years ago

Thank you for your advice! Your codes and your reply gives me a lot of help. Best wishes!