search

emanjavacas / pie

A fully-fledge PyTorch package for Morphological Analysis, tailored to morphologically rich and historical languages.
MIT License
22 stars 10 forks source link

Regression with CNNs ? #69

Open PonteIneptique opened 4 years ago

PonteIneptique commented 4 years ago

Hey :) Weird thing: as I actually had equivalent scores between CNN and RNN embeddings, my experiments with Optuna for now have yielded consistently lower result (by a big margin) on CNNs (I assume this change is in cause https://github.com/emanjavacas/pie/commit/bca9516938a4a8bc5f4d385949eeff5a14a4b67b#diff-4853226f237bf1b0a17a302fbfa3e997 )

I cannot say if it's a regression before I actually retrain the same config as before but this might be something to look at in the next weeks.

PonteIneptique commented 4 years ago

Current scores:

Id Param Type (1=RNN, 0=CNN) Status
64 cemb_type 1.0 RUNNING
18 cemb_type 0.0 RUNNING
5 cemb_type 1.0 0.0 COMPLETE
9 cemb_type 1.0 0.0 COMPLETE
72 cemb_type 1.0 0.0 COMPLETE
7 cemb_type 0.0 0.7219 COMPLETE
9 cemb_type 0.0 0.842 PRUNED
50 cemb_type 0.0 0.8423 PRUNED
4 cemb_type 0.0 0.8585 PRUNED
13 cemb_type 0.0 0.8606 PRUNED
14 cemb_type 0.0 0.8635 PRUNED
15 cemb_type 0.0 0.8645 PRUNED
16 cemb_type 0.0 0.8668 PRUNED
10 cemb_type 0.0 0.8677 PRUNED
6 cemb_type 0.0 0.8698 PRUNED
6 cemb_type 0.0 0.881 PRUNED
38 cemb_type 0.0 0.8815 PRUNED
26 cemb_type 1.0 0.8883 PRUNED
8 cemb_type 0.0 0.8899 PRUNED
35 cemb_type 0.0 0.901 PRUNED
40 cemb_type 0.0 0.9029 PRUNED
81 cemb_type 0.0 0.9032 PRUNED
0 cemb_type 0.0 0.9127 COMPLETE
0 cemb_type 0.0 0.9193 COMPLETE
1 cemb_type 0.0 0.9196 COMPLETE
36 cemb_type 1.0 0.9343 PRUNED
1 cemb_type 0.0 0.9372 COMPLETE
34 cemb_type 1.0 0.947 PRUNED
3 cemb_type 0.0 0.9496 COMPLETE
12 cemb_type 1.0 0.9555 PRUNED
39 cemb_type 1.0 0.959 PRUNED
79 cemb_type 1.0 0.959 PRUNED
47 cemb_type 1.0 0.9606 PRUNED
30 cemb_type 1.0 0.9609 PRUNED
20 cemb_type 1.0 0.9619 PRUNED
29 cemb_type 1.0 0.9636 PRUNED
12 cemb_type 1.0 0.9642 PRUNED
22 cemb_type 1.0 0.9644 PRUNED
49 cemb_type 1.0 0.9646 PRUNED
63 cemb_type 1.0 0.9647 PRUNED
59 cemb_type 1.0 0.9652 PRUNED
66 cemb_type 1.0 0.9652 PRUNED
97 cemb_type 1.0 0.9653 PRUNED
37 cemb_type 1.0 0.9656 PRUNED
94 cemb_type 1.0 0.9656 PRUNED
48 cemb_type 1.0 0.966 PRUNED
75 cemb_type 1.0 0.9661 PRUNED
84 cemb_type 1.0 0.9661 PRUNED
33 cemb_type 1.0 0.9664 PRUNED
46 cemb_type 1.0 0.9665 PRUNED
82 cemb_type 1.0 0.967 PRUNED
78 cemb_type 1.0 0.9671 PRUNED
83 cemb_type 1.0 0.9671 PRUNED
98 cemb_type 1.0 0.9674 PRUNED
85 cemb_type 1.0 0.9676 PRUNED
44 cemb_type 1.0 0.9678 PRUNED
74 cemb_type 1.0 0.9678 PRUNED
62 cemb_type 1.0 0.968 PRUNED
68 cemb_type 1.0 0.9683 PRUNED
71 cemb_type 1.0 0.9683 PRUNED
92 cemb_type 1.0 0.9683 PRUNED
27 cemb_type 1.0 0.9684 PRUNED
99 cemb_type 1.0 0.9696 PRUNED
56 cemb_type 1.0 0.9706 PRUNED
3 cemb_type 1.0 0.9719 COMPLETE
16 cemb_type 1.0 0.973 COMPLETE
80 cemb_type 1.0 0.9733 PRUNED
2 cemb_type 1.0 0.9739 COMPLETE
13 cemb_type 1.0 0.9742 COMPLETE
19 cemb_type 1.0 0.9743 PRUNED
73 cemb_type 1.0 0.9744 COMPLETE
11 cemb_type 1.0 0.9745 COMPLETE
10 cemb_type 1.0 0.9747 COMPLETE
7 cemb_type 1.0 0.9748 COMPLETE
89 cemb_type 1.0 0.9751 PRUNED
96 cemb_type 1.0 0.9751 PRUNED
70 cemb_type 1.0 0.9752 COMPLETE
2 cemb_type 1.0 0.9752 COMPLETE
5 cemb_type 1.0 0.9753 COMPLETE
17 cemb_type 1.0 0.9754 COMPLETE
95 cemb_type 1.0 0.9756 PRUNED
11 cemb_type 1.0 0.9757 COMPLETE
54 cemb_type 1.0 0.9757 COMPLETE
8 cemb_type 1.0 0.9757 COMPLETE
45 cemb_type 1.0 0.9758 COMPLETE
25 cemb_type 1.0 0.976 COMPLETE
51 cemb_type 1.0 0.976 COMPLETE
17 cemb_type 1.0 0.976 COMPLETE
41 cemb_type 1.0 0.9761 COMPLETE
87 cemb_type 1.0 0.9769 COMPLETE
60 cemb_type 1.0 0.977 COMPLETE
18 cemb_type 1.0 0.9772 COMPLETE
21 cemb_type 1.0 0.9772 COMPLETE
28 cemb_type 1.0 0.9773 COMPLETE
42 cemb_type 1.0 0.9773 COMPLETE
77 cemb_type 1.0 0.9774 COMPLETE
86 cemb_type 1.0 0.9775 COMPLETE
32 cemb_type 1.0 0.9776 COMPLETE
91 cemb_type 1.0 0.9776 COMPLETE
61 cemb_type 1.0 0.9778 COMPLETE
23 cemb_type 1.0 0.9779 COMPLETE
24 cemb_type 1.0 0.978 COMPLETE
14 cemb_type 1.0 0.9781 COMPLETE
31 cemb_type 1.0 0.9781 COMPLETE
4 cemb_type 1.0 0.9782 COMPLETE
76 cemb_type 1.0 0.9783 COMPLETE
43 cemb_type 1.0 0.9784 COMPLETE
58 cemb_type 1.0 0.9784 COMPLETE
53 cemb_type 1.0 0.9785 COMPLETE
67 cemb_type 1.0 0.9785 COMPLETE
88 cemb_type 1.0 0.9785 COMPLETE
52 cemb_type 1.0 0.9787 COMPLETE
69 cemb_type 1.0 0.9787 COMPLETE
90 cemb_type 1.0 0.9787 COMPLETE
15 cemb_type 1.0 0.9788 COMPLETE
55 cemb_type 1.0 0.9788 COMPLETE
57 cemb_type 1.0 0.979 COMPLETE
93 cemb_type 1.0 0.9794 COMPLETE
65 cemb_type 1.0 0.9796 COMPLETE
PonteIneptique commented 4 years ago

It seems it has been a while that I did not train on CNN. I leave that open for the future, when comparing with the models we have for Old French.

emanjavacas commented 4 years ago

Hey, I believe bca9516 fixed some previous hacks with the CNNs, it's an old commit I had lying around. I am surprised I hadn't pushed it before. I haven't used cnn embeddings very extensively, so I am cannot rule out a bug for sure. It would need some experimenting to make clear what's going on. Same architecture trained with previous and current implementation and comparison.