HLR / DomiKnowS

37 stars 6 forks source link

CIFAR good and bad cases #359

Closed DariusNafar closed 1 year ago

DariusNafar commented 1 year ago

The first issue is that I don't see the "ifsatisfied" category. although I don't have exactL so it might be a bug or maybe not.

now for every graph, I have 5 examples for each of these cases: -tag is correct and ILP makes it False -category is correct and ILP makes it False -tag is Wrong and ILP makes it True -category is Wrong and ILP makes it True

I'll analyze an example here to show case the format:

for the graph exactL_ifLorLbottomup a case category was wrong and then it was corrected is as follows:

tag improved: original tag 99 new wrong ILP tag 61 , correct category 13 previous category 5 new ILP category 5, probabilities of correct tag and category are 0.37 and 0.26 and their SoftMax are 0.49 and 0.50 and their ILP probabilities are 0.37 and 0.50

this text describes a tag that the model predicted as 99 but ILP changes it to 61 and it becomes wrong. keep in mind that the category remains 5 and it doesn't change after ILP to the correct category of 13.

looking at these probabilities is not easy to know what is wrong. so what is going on is that the original probability of tag is 49 which is more than 37 but not of his children have a good probability. but category 5 has a probability of 50 so it chooses its parent.

the rest of the data is here attached after this.

DariusNafar commented 1 year ago

exactL_ifLorLbottomup

tag messed up: original tag 21 new wrong ILP tag 3 ,correct cathegory 11 previous cathegory 8 new ILP cathegory 8, probablites of correct tag and cathegory are 0.93 and 0.01 and their softmax are 0.93 and 0.99 and their ILP probablites are 0.04 and 0.99 cathegory improved: original cathegory 11 new wrong ILP cathegory 16 ,correct tag 21 previous tag 21 new ILP tag 21, probablites of correct tag and cathegory are 0.66 and 0.29 and their softmax are 0.66 and 0.59 and their ILP probablites are 0.66 and 0.29 cathegory messed up: original cathegory 2 new wrong ILP cathegory 1 ,correct tag 54 previous tag 1 new ILP tag 1, probablites of correct tag and cathegory are 0.02 and 0.52 and their softmax are 0.96 and 0.52 and their ILP probablites are 0.96 and 0.47 tag messed up: original tag 16 new wrong ILP tag 83 ,correct cathegory 3 previous cathegory 5 new ILP cathegory 5, probablites of correct tag and cathegory are 0.29 and 0.26 and their softmax are 0.29 and 0.59 and their ILP probablites are 0.19 and 0.59 tag improved: original tag 53 new wrong ILP tag 62 ,correct cathegory 4 previous cathegory 4 new ILP cathegory 4, probablites of correct tag and cathegory are 0.19 and 0.73 and their softmax are 0.64 and 0.73 and their ILP probablites are 0.19 and 0.73 cathegory improved: original cathegory 2 new wrong ILP cathegory 1 ,correct tag 62 previous tag 62 new ILP tag 62, probablites of correct tag and cathegory are 0.64 and 0.46 and their softmax are 0.64 and 0.47 and their ILP probablites are 0.64 and 0.46 tag improved: original tag 66 new wrong ILP tag 43 ,correct cathegory 12 previous cathegory 12 new ILP cathegory 12, probablites of correct tag and cathegory are 0.34 and 0.41 and their softmax are 0.37 and 0.41 and their ILP probablites are 0.34 and 0.41 tag messed up: original tag 16 new wrong ILP tag 87 ,correct cathegory 3 previous cathegory 5 new ILP cathegory 5, probablites of correct tag and cathegory are 0.42 and 0.04 and their softmax are 0.42 and 0.95 and their ILP probablites are 0.25 and 0.95 tag improved: original tag 98 new wrong ILP tag 65 ,correct cathegory 14 previous cathegory 1 new ILP cathegory 1, probablites of correct tag and cathegory are 0.29 and 0.07 and their softmax are 0.50 and 0.83 and their ILP probablites are 0.29 and 0.83 cathegory improved: original cathegory 3 new wrong ILP cathegory 5 ,correct tag 16 previous tag 16 new ILP tag 16, probablites of correct tag and cathegory are 0.92 and 0.05 and their softmax are 0.92 and 0.91 and their ILP probablites are 0.92 and 0.05 cathegory improved: original cathegory 16 new wrong ILP cathegory 12 ,correct tag 65 previous tag 65 new ILP tag 65, probablites of correct tag and cathegory are 0.53 and 0.50 and their softmax are 0.53 and 0.50 and their ILP probablites are 0.53 and 0.50 tag messed up: original tag 77 new wrong ILP tag 7 ,correct cathegory 13 previous cathegory 7 new ILP cathegory 7, probablites of correct tag and cathegory are 0.73 and 0.15 and their softmax are 0.73 and 0.85 and their ILP probablites are 0.20 and 0.85 cathegory improved: original cathegory 8 new wrong ILP cathegory 7 ,correct tag 3 previous tag 43 new ILP tag 43, probablites of correct tag and cathegory are 0.00 and 0.31 and their softmax are 0.70 and 0.68 and their ILP probablites are 0.70 and 0.31 tag messed up: original tag 78 new wrong ILP tag 7 ,correct cathegory 15 previous cathegory 7 new ILP cathegory 7, probablites of correct tag and cathegory are 0.31 and 0.19 and their softmax are 0.31 and 0.28 and their ILP probablites are 0.24 and 0.28 cathegory messed up: original cathegory 11 new wrong ILP cathegory 8 ,correct tag 15 previous tag 43 new ILP tag 43, probablites of correct tag and cathegory are 0.07 and 0.47 and their softmax are 0.43 and 0.47 and their ILP probablites are 0.43 and 0.17 tag improved: original tag 26 new wrong ILP tag 62 ,correct cathegory 13 previous cathegory 13 new ILP cathegory 13, probablites of correct tag and cathegory are 0.17 and 0.64 and their softmax are 0.19 and 0.64 and their ILP probablites are 0.17 and 0.64 cathegory messed up: original cathegory 10 new wrong ILP cathegory 9 ,correct tag 33 previous tag 68 new ILP tag 68, probablites of correct tag and cathegory are 0.12 and 0.39 and their softmax are 0.49 and 0.39 and their ILP probablites are 0.49 and 0.24 cathegory messed up: original cathegory 7 new wrong ILP cathegory 11 ,correct tag 7 previous tag 21 new ILP tag 21, probablites of correct tag and cathegory are 0.15 and 0.40 and their softmax are 0.33 and 0.40 and their ILP probablites are 0.33 and 0.34 tag improved: original tag 48 new wrong ILP tag 89 ,correct cathegory 18 previous cathegory 13 new ILP cathegory 13, probablites of correct tag and cathegory are 0.15 and 0.12 and their softmax are 0.40 and 0.62 and their ILP probablites are 0.15 and 0.62 cathegory messed up: original cathegory 16 new wrong ILP cathegory 11 ,correct tag 65 previous tag 21 new ILP tag 21, probablites of correct tag and cathegory are 0.00 and 0.60 and their softmax are 0.72 and 0.60 and their ILP probablites are 0.72 and 0.13

exactL_ifLorLbothways

tag messed up: original tag 71 new wrong ILP tag 33 ,correct cathegory 10 previous cathegory 10 new ILP cathegory 10, probablites of correct tag and cathegory are 1.00 and 1.00 and their softmax are 1.00 and 1.00 and their ILP probablites are 0.00 and 1.00 tag messed up: original tag 14 new wrong ILP tag 6 ,correct cathegory 7 previous cathegory 7 new ILP cathegory 7, probablites of correct tag and cathegory are 0.97 and 0.69 and their softmax are 0.97 and 0.69 and their ILP probablites are 0.00 and 0.69 tag messed up: original tag 69 new wrong ILP tag 41 ,correct cathegory 19 previous cathegory 19 new ILP cathegory 19, probablites of correct tag and cathegory are 1.00 and 1.00 and their softmax are 1.00 and 1.00 and their ILP probablites are 0.00 and 1.00 tag messed up: original tag 40 new wrong ILP tag 86 ,correct cathegory 5 previous cathegory 5 new ILP cathegory 5, probablites of correct tag and cathegory are 1.00 and 1.00 and their softmax are 1.00 and 1.00 and their ILP probablites are 0.00 and 1.00 tag messed up: original tag 92 new wrong ILP tag 62 ,correct cathegory 2 previous cathegory 2 new ILP cathegory 2, probablites of correct tag and cathegory are 0.95 and 1.00 and their softmax are 0.95 and 1.00 and their ILP probablites are 0.05 and 1.00 cathegory improved: original cathegory 2 new wrong ILP cathegory 5 ,correct tag 70 previous tag 82 new ILP tag 62, probablites of correct tag and cathegory are 0.00 and 0.40 and their softmax are 0.53 and 0.42 and their ILP probablites are 0.12 and 0.40 tag improved: original tag 61 new wrong ILP tag 10 ,correct cathegory 3 previous cathegory 3 new ILP cathegory 3, probablites of correct tag and cathegory are 0.20 and 0.99 and their softmax are 0.69 and 0.99 and their ILP probablites are 0.20 and 0.99 tag improved: original tag 41 new wrong ILP tag 45 ,correct cathegory 19 previous cathegory 13 new ILP cathegory 19, probablites of correct tag and cathegory are 0.29 and 0.29 and their softmax are 0.43 and 0.52 and their ILP probablites are 0.29 and 0.29 cathegory improved: original cathegory 19 new wrong ILP cathegory 13 ,correct tag 41 previous tag 45 new ILP tag 41, probablites of correct tag and cathegory are 0.29 and 0.29 and their softmax are 0.43 and 0.52 and their ILP probablites are 0.29 and 0.29 tag improved: original tag 6 new wrong ILP tag 7 ,correct cathegory 7 previous cathegory 7 new ILP cathegory 7, probablites of correct tag and cathegory are 0.48 and 0.99 and their softmax are 0.49 and 0.99 and their ILP probablites are 0.48 and 0.99 tag improved: original tag 67 new wrong ILP tag 73 ,correct cathegory 1 previous cathegory 1 new ILP cathegory 1, probablites of correct tag and cathegory are 0.00 and 1.00 and their softmax are 0.98 and 1.00 and their ILP probablites are 0.00 and 1.00 cathegory improved: original cathegory 11 new wrong ILP cathegory 16 ,correct tag 21 previous tag 21 new ILP tag 21, probablites of correct tag and cathegory are 0.66 and 0.29 and their softmax are 0.66 and 0.59 and their ILP probablites are 0.66 and 0.29 tag improved: original tag 37 new wrong ILP tag 81 ,correct cathegory 9 previous cathegory 19 new ILP cathegory 9, probablites of correct tag and cathegory are 0.18 and 0.41 and their softmax are 0.80 and 0.57 and their ILP probablites are 0.18 and 0.41 cathegory improved: original cathegory 9 new wrong ILP cathegory 19 ,correct tag 37 previous tag 81 new ILP tag 37, probablites of correct tag and cathegory are 0.18 and 0.41 and their softmax are 0.80 and 0.57 and their ILP probablites are 0.18 and 0.41 cathegory improved: original cathegory 12 new wrong ILP cathegory 18 ,correct tag 64 previous tag 66 new ILP tag 75, probablites of correct tag and cathegory are 0.00 and 0.46 and their softmax are 0.72 and 0.48 and their ILP probablites are 0.11 and 0.46 cathegory messed up: original cathegory 4 new wrong ILP cathegory 2 ,correct tag 53 previous tag 62 new ILP tag 62, probablites of correct tag and cathegory are 0.19 and 0.73 and their softmax are 0.64 and 0.73 and their ILP probablites are 0.64 and 0.27 cathegory messed up: original cathegory 12 new wrong ILP cathegory 8 ,correct tag 66 previous tag 43 new ILP tag 43, probablites of correct tag and cathegory are 0.34 and 0.41 and their softmax are 0.37 and 0.41 and their ILP probablites are 0.37 and 0.10 cathegory messed up: original cathegory 3 new wrong ILP cathegory 14 ,correct tag 9 previous tag 9 new ILP tag 98, probablites of correct tag and cathegory are 0.83 and 0.43 and their softmax are 0.83 and 0.43 and their ILP probablites are 0.04 and 0.42 cathegory messed up: original cathegory 5 new wrong ILP cathegory 3 ,correct tag 22 previous tag 22 new ILP tag 61, probablites of correct tag and cathegory are 0.88 and 0.44 and their softmax are 0.88 and 0.44 and their ILP probablites are 0.06 and 0.41 cathegory messed up: original cathegory 11 new wrong ILP cathegory 8 ,correct tag 38 previous tag 43 new ILP tag 43, probablites of correct tag and cathegory are 0.03 and 0.45 and their softmax are 0.39 and 0.45 and their ILP probablites are 0.39 and 0.27

exactL_nandL

tag messed up: original tag 21 new wrong ILP tag 3 ,correct cathegory 11 previous cathegory 8 new ILP cathegory 8, probablites of correct tag and cathegory are 0.93 and 0.01 and their softmax are 0.93 and 0.99 and their ILP probablites are 0.04 and 0.99 cathegory improved: original cathegory 2 new wrong ILP cathegory 5 ,correct tag 70 previous tag 82 new ILP tag 82, probablites of correct tag and cathegory are 0.00 and 0.40 and their softmax are 0.53 and 0.42 and their ILP probablites are 0.53 and 0.40 tag messed up: original tag 93 new wrong ILP tag 73 ,correct cathegory 15 previous cathegory 1 new ILP cathegory 1, probablites of correct tag and cathegory are 0.32 and 0.08 and their softmax are 0.32 and 0.76 and their ILP probablites are 0.14 and 0.76 tag messed up: original tag 58 new wrong ILP tag 5 ,correct cathegory 18 previous cathegory 6 new ILP cathegory 6, probablites of correct tag and cathegory are 0.41 and 0.08 and their softmax are 0.41 and 0.92 and their ILP probablites are 0.31 and 0.92 cathegory improved: original cathegory 4 new wrong ILP cathegory 5 ,correct tag 57 previous tag 57 new ILP tag 57, probablites of correct tag and cathegory are 0.74 and 0.38 and their softmax are 0.74 and 0.43 and their ILP probablites are 0.74 and 0.38 cathegory improved: original cathegory 11 new wrong ILP cathegory 16 ,correct tag 21 previous tag 21 new ILP tag 21, probablites of correct tag and cathegory are 0.66 and 0.29 and their softmax are 0.66 and 0.59 and their ILP probablites are 0.66 and 0.29 cathegory improved: original cathegory 9 new wrong ILP cathegory 13 ,correct tag 76 previous tag 76 new ILP tag 76, probablites of correct tag and cathegory are 0.96 and 0.06 and their softmax are 0.96 and 0.80 and their ILP probablites are 0.96 and 0.06 cathegory improved: original cathegory 11 new wrong ILP cathegory 19 ,correct tag 19 previous tag 19 new ILP tag 19, probablites of correct tag and cathegory are 0.55 and 0.15 and their softmax are 0.55 and 0.48 and their ILP probablites are 0.55 and 0.15 tag messed up: original tag 70 new wrong ILP tag 32 ,correct cathegory 2 previous cathegory 1 new ILP cathegory 1, probablites of correct tag and cathegory are 0.50 and 0.25 and their softmax are 0.50 and 0.74 and their ILP probablites are 0.07 and 0.74 tag improved: original tag 1 new wrong ILP tag 4 ,correct cathegory 1 previous cathegory 1 new ILP cathegory 1, probablites of correct tag and cathegory are 0.42 and 0.89 and their softmax are 0.44 and 0.89 and their ILP probablites are 0.42 and 0.89 tag improved: original tag 8 new wrong ILP tag 27 ,correct cathegory 18 previous cathegory 18 new ILP cathegory 18, probablites of correct tag and cathegory are 0.14 and 0.63 and their softmax are 0.47 and 0.63 and their ILP probablites are 0.14 and 0.63 cathegory messed up: original cathegory 2 new wrong ILP cathegory 1 ,correct tag 54 previous tag 1 new ILP tag 1, probablites of correct tag and cathegory are 0.02 and 0.52 and their softmax are 0.96 and 0.52 and their ILP probablites are 0.96 and 0.47 tag messed up: original tag 49 new wrong ILP tag 50 ,correct cathegory 10 previous cathegory 16 new ILP cathegory 16, probablites of correct tag and cathegory are 0.18 and 0.14 and their softmax are 0.18 and 0.24 and their ILP probablites are 0.10 and 0.24 tag improved: original tag 59 new wrong ILP tag 74 ,correct cathegory 17 previous cathegory 11 new ILP cathegory 17, probablites of correct tag and cathegory are 0.12 and 0.26 and their softmax are 0.15 and 0.26 and their ILP probablites are 0.12 and 0.26 tag improved: original tag 79 new wrong ILP tag 97 ,correct cathegory 13 previous cathegory 13 new ILP cathegory 13, probablites of correct tag and cathegory are 0.30 and 0.62 and their softmax are 0.65 and 0.62 and their ILP probablites are 0.30 and 0.62 cathegory messed up: original cathegory 12 new wrong ILP cathegory 15 ,correct tag 34 previous tag 29 new ILP tag 29, probablites of correct tag and cathegory are 0.08 and 0.71 and their softmax are 0.65 and 0.71 and their ILP probablites are 0.65 and 0.20 tag improved: original tag 53 new wrong ILP tag 62 ,correct cathegory 4 previous cathegory 4 new ILP cathegory 4, probablites of correct tag and cathegory are 0.19 and 0.73 and their softmax are 0.64 and 0.73 and their ILP probablites are 0.19 and 0.73 cathegory messed up: original cathegory 16 new wrong ILP cathegory 7 ,correct tag 80 previous tag 7 new ILP tag 7, probablites of correct tag and cathegory are 0.03 and 0.58 and their softmax are 0.75 and 0.58 and their ILP probablites are 0.75 and 0.39 cathegory messed up: original cathegory 1 new wrong ILP cathegory 0 ,correct tag 32 previous tag 72 new ILP tag 72, probablites of correct tag and cathegory are 0.05 and 0.47 and their softmax are 0.28 and 0.47 and their ILP probablites are 0.28 and 0.33 cathegory messed up: original cathegory 11 new wrong ILP cathegory 16 ,correct tag 21 previous tag 80 new ILP tag 80, probablites of correct tag and cathegory are 0.00 and 0.60 and their softmax are 0.96 and 0.60 and their ILP probablites are 0.96 and 0.38 exactL_ifLorLtopdown

tag messed up: original tag 33 new wrong ILP tag 49 ,correct cathegory 10 previous cathegory 10 new ILP cathegory 10, probablites of correct tag and cathegory are 0.91 and 0.96 and their softmax are 0.91 and 0.96 and their ILP probablites are 0.00 and 0.96 tag messed up: original tag 51 new wrong ILP tag 57 ,correct cathegory 4 previous cathegory 4 new ILP cathegory 4, probablites of correct tag and cathegory are 0.66 and 0.80 and their softmax are 0.66 and 0.80 and their ILP probablites are 0.00 and 0.80 tag messed up: original tag 71 new wrong ILP tag 49 ,correct cathegory 10 previous cathegory 10 new ILP cathegory 10, probablites of correct tag and cathegory are 1.00 and 1.00 and their softmax are 1.00 and 1.00 and their ILP probablites are 0.00 and 1.00 tag messed up: original tag 14 new wrong ILP tag 7 ,correct cathegory 7 previous cathegory 7 new ILP cathegory 7, probablites of correct tag and cathegory are 0.97 and 0.69 and their softmax are 0.97 and 0.69 and their ILP probablites are 0.01 and 0.69 tag messed up: original tag 40 new wrong ILP tag 87 ,correct cathegory 5 previous cathegory 5 new ILP cathegory 5, probablites of correct tag and cathegory are 1.00 and 1.00 and their softmax are 1.00 and 1.00 and their ILP probablites are 0.00 and 1.00 cathegory improved: original cathegory 4 new wrong ILP cathegory 5 ,correct tag 57 previous tag 57 new ILP tag 57, probablites of correct tag and cathegory are 0.74 and 0.38 and their softmax are 0.74 and 0.43 and their ILP probablites are 0.74 and 0.38 tag improved: original tag 2 new wrong ILP tag 35 ,correct cathegory 14 previous cathegory 14 new ILP cathegory 14, probablites of correct tag and cathegory are 0.00 and 1.00 and their softmax are 0.85 and 1.00 and their ILP probablites are 0.00 and 1.00 cathegory improved: original cathegory 2 new wrong ILP cathegory 1 ,correct tag 70 previous tag 70 new ILP tag 70, probablites of correct tag and cathegory are 0.50 and 0.25 and their softmax are 0.50 and 0.74 and their ILP probablites are 0.50 and 0.25 cathegory improved: original cathegory 13 new wrong ILP cathegory 18 ,correct tag 45 previous tag 45 new ILP tag 45, probablites of correct tag and cathegory are 0.43 and 0.19 and their softmax are 0.43 and 0.32 and their ILP probablites are 0.43 and 0.19 tag improved: original tag 58 new wrong ILP tag 13 ,correct cathegory 18 previous cathegory 18 new ILP cathegory 18, probablites of correct tag and cathegory are 0.19 and 1.00 and their softmax are 0.81 and 1.00 and their ILP probablites are 0.19 and 1.00 cathegory improved: original cathegory 12 new wrong ILP cathegory 18 ,correct tag 64 previous tag 66 new ILP tag 75, probablites of correct tag and cathegory are 0.00 and 0.46 and their softmax are 0.72 and 0.48 and their ILP probablites are 0.11 and 0.46 tag improved: original tag 7 new wrong ILP tag 14 ,correct cathegory 7 previous cathegory 7 new ILP cathegory 7, probablites of correct tag and cathegory are 0.13 and 1.00 and their softmax are 0.82 and 1.00 and their ILP probablites are 0.13 and 1.00 cathegory messed up: original cathegory 15 new wrong ILP cathegory 19 ,correct tag 44 previous tag 69 new ILP tag 69, probablites of correct tag and cathegory are 0.01 and 0.35 and their softmax are 0.45 and 0.35 and their ILP probablites are 0.45 and 0.23 cathegory improved: original cathegory 10 new wrong ILP cathegory 16 ,correct tag 49 previous tag 49 new ILP tag 49, probablites of correct tag and cathegory are 0.18 and 0.14 and their softmax are 0.18 and 0.24 and their ILP probablites are 0.18 and 0.14 tag improved: original tag 65 new wrong ILP tag 50 ,correct cathegory 16 previous cathegory 16 new ILP cathegory 16, probablites of correct tag and cathegory are 0.04 and 1.00 and their softmax are 0.61 and 1.00 and their ILP probablites are 0.04 and 1.00 tag improved: original tag 68 new wrong ILP tag 76 ,correct cathegory 9 previous cathegory 9 new ILP cathegory 9, probablites of correct tag and cathegory are 0.04 and 0.44 and their softmax are 0.80 and 0.44 and their ILP probablites are 0.04 and 0.44 cathegory messed up: original cathegory 16 new wrong ILP cathegory 7 ,correct tag 80 previous tag 7 new ILP tag 7, probablites of correct tag and cathegory are 0.03 and 0.58 and their softmax are 0.75 and 0.58 and their ILP probablites are 0.75 and 0.39 cathegory messed up: original cathegory 13 new wrong ILP cathegory 7 ,correct tag 99 previous tag 7 new ILP tag 7, probablites of correct tag and cathegory are 0.10 and 0.69 and their softmax are 0.48 and 0.69 and their ILP probablites are 0.48 and 0.29 cathegory messed up: original cathegory 7 new wrong ILP cathegory 13 ,correct tag 6 previous tag 45 new ILP tag 45, probablites of correct tag and cathegory are 0.26 and 0.50 and their softmax are 0.55 and 0.50 and their ILP probablites are 0.55 and 0.22 cathegory messed up: original cathegory 10 new wrong ILP cathegory 9 ,correct tag 33 previous tag 68 new ILP tag 68, probablites of correct tag and cathegory are 0.12 and 0.39 and their softmax are 0.49 and 0.39 and their ILP probablites are 0.49 and 0.24