search

FenTechSolutions / CausalDiscoveryToolbox

Package for causal inference in graphs and in the pairwise settings. Tools for graph structure recovery and dependencies are included.
https://fentechsolutions.github.io/CausalDiscoveryToolbox/html/index.html
MIT License
1.08k stars 198 forks source link

SID and SHD do not get the same results as the author of SID #137

Open Flopp88 opened 2 years ago

Flopp88 commented 2 years ago

Hello,

While experimenting on the SID metric, I tried to reproduce the example from the paper presenting the SID: https://www.researchgate.net/publication/237053920_Structural_Intervention_Distance_SID_for_Evaluating_Causal_Graphs (page 4)

The autor states that when comparing H1 and H2 to the ground truth graph G, we have SHD(G,H1)=SHD(G,H2)=1 , SID(G,H1)=0 and SID(G,H2)=8 However, when runing the code (SID.zip) on the equivalent adjacency matrices, it outputs: SHD(G,H1)=1 SHD(G,H2)=2 SID(G,H1)=1 SID(G,H2)=8

When trying with SID_CPDAG and SHD_CPDAG, it outputs: SHD_CPDAG,(G,H1)=1 SHD_CPDAG,(G,H2)=0 SID_CPDAG(G,H1)=[1, 1] SID_CPDAG(G,H2)=[8, 8]

What could be the issue ? Thank you for your help

diviyank commented 2 years ago

Hello, Could you try directly in R using the author's package?
https://cran.r-project.org/src/contrib/Archive/SID/ Best regards, Diviyan

Flopp88 commented 1 year ago

When running the author's package on R, i obtain the same results as in the paper: image image

diviyank commented 1 year ago

Thanks for the feedback! This is really troubling... There might be data transfer issues between the languages! I'll look into this, thanks!