AUC & ROC

[shirancohen2016]

Btergm- Compare models with AUC.

Here is a problem description: Say I want to predict two networks (out-of-sample) after I have a model.

I get two AUC values (one per each network) and two ROC model values.

My question is: how to compare AUC between different models? as I have 2 AUC generated from each model.

if it was only one value of AUC I could always take the max value but here I have two.

here is an example: model1: AUC: 0.8, 0.65 model2: AUC: 0.7, 0.74

if I go by the AUC of the first network (0.8,0.7) I will choose model #1 but if I go by the AUC values of the second network (0.65,0.74) I will choose model #2

Do I need to do average of AUC ? Best, Shiran

[leifeld]

I have implemented a solution in commit 9f284e0bc5534fae13bdac92a4d6ef422beaed01. Please install the latest build from GitHub (e.g., using install_github("leifeld/btergm"), which requires the devtools R package), and try it out. The new version allows custom arguments in GOF statistics. They are passed over to the statistics by the respective gof method. The rocpr statistic has an argument called joint, which produces a joint ROC/PR prediction for all time steps rather than separate values per time step. For example: gof(model, statistics = c(esp, rocpr), joint = TRUE).

[shirancohen2016]

Thank you Philip, Can you please explain how does the joint prediction works for 2 or more time steps?

I guess it is not the average of both time steps

Best, Shiran

On Mar 14, 2017 02:24, "Philip Leifeld" notifications@github.com wrote:

I have implemented a solution in commit 9f284e0 https://github.com/leifeld/btergm/commit/9f284e0bc5534fae13bdac92a4d6ef422beaed01. Please install the latest build from GitHub (e.g., using install_github("leifeld/btergm"), which requires the devtools R package), and try it out. The new version allows custom arguments in GOF statistics. They are passed over to the statistics by the respective gof method. The rocpr statistic has an argument called joint, which produces a joint ROC/PR prediction for all time steps rather than separate values per time step. For example: gof(model, statistics = c(esp, rocpr), joint = TRUE).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/leifeld/btergm/issues/3#issuecomment-286284846, or mute the thread https://github.com/notifications/unsubscribe-auth/AY6SnZZwcO-vKz0UNyIEj0w4hSdUXn0fks5rld5DgaJpZM4MP90Z .

[leifeld]

To compute the ROC or PR curve, each cell in the network matrix is treated as an observation, and the sum of positive predictions for the respective cell is compared with its observed value. If joint = FALSE, this is done separately for each time step, thus producing t AUC values. If joint = TRUE, the observation and simulation vectors are collated such that the comparison is not done t times for n times n cells but rather once for a vector containing all t times n times n cells.

[shirancohen2016]

OK got it!

Thank you Philip.

On Mar 14, 2017 11:08 AM, "Philip Leifeld" notifications@github.com wrote:

To compute the ROC or PR curve, each cell in the network matrix is treated as an observation, and the sum of positive predictions for the respective cell is compared with its observed value. If joint = FALSE, this is done separately for each time step, thus producing t AUC values. If joint = TRUE, the observation and simulation vectors are collated such that the comparison is not done t times for n times n cells but rather once for a vector containing all t times n times n cells.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/leifeld/btergm/issues/3#issuecomment-286362144, or mute the thread https://github.com/notifications/unsubscribe-auth/AY6SnZ9oH7ns-92Bu7EpbtTG2EMJAAmZks5rllkmgaJpZM4MP90Z .