search

rimorob / netres

To support an ICML submission - anonymous for now
1 stars 1 forks source link

try assess on star incase it crashes. #10

Open fkgruber opened 1 year ago

rimorob commented 1 year ago

It definitely crashes. There are no safeguards for computing metrics that require there to be ground truth positives available. The true dag for the star cases has no examples of true edges that don't connect to the latent space

Separate question: how can we evaluate edges to the latent space? We'd have to match the inferred nodes to the latent nodes in some way.

fkgruber commented 1 year ago

the problem is if the true vs estimated latent dimensionality is different. Only if we treat any edge between any latent and an observable as 1 edge in both true or estimated network

fkgruber commented 1 year ago

It definitely crashes. There are no safeguards for computing metrics that require there to be ground truth positives available. The true dag for the star cases has no examples of true edges that don't connect to the latent space

Separate question: how can we evaluate edges to the latent space? We'd have to match the inferred nodes to the latent nodes in some way.

It definitely crashes. There are no safeguards for computing metrics that require there to be ground truth positives available. The true dag for the star cases has no examples of true edges that don't connect to the latent space

not sure how you want to define the performance when there are no positive cases. We could do something different in those cases.

rimorob commented 1 year ago

I think the solution is to only report metrics that don't crash, e.g. lift (for FNR, for example). This can be easily done by wrapping each metric inside try/catch

fkgruber commented 1 year ago

also skip if there is no true network provided. if there is true latent it should plot that only. maybe some plot mse vs iteration correplation plot vs iteration

rimorob commented 1 year ago

See separate issue that I've just closed - assess() will plot corrplot if true and inferred latent spaces are available, and will exit after that if the true network is not available.

fkgruber commented 1 year ago

I think the solution is to only report metrics that don't crash, e.g. lift (for FNR, for example). This can be easily done by wrapping each metric inside try/catch

lift is defined (in performance functions) as P(Yhat = + | Y = +)/P(Yhat = +). but the conditional is only defined if P(Y=+) >0. So in the star case it will be undefined. Same with TPR so AUC or prAUC can't be estimated either.

But I can show the other stuff instead.