search

CogComp / saul

Saul : Declarative Learning-Based Programming
Other
64 stars 18 forks source link

Discrepancy before/after loading? #235

Closed danyaljj closed 8 years ago

danyaljj commented 8 years ago

Consider the two snippets: 1) Training and test+ saving the model 

    EntityRelationDataModel.populateWithConll()
    PersonClassifier.learn(50)
    PersonClassifier.save()
    PersonClassifier.test()

2) load the model and test: 

     EntityRelationDataModel.populateWithConll()
     PersonClassifier.load()
     PersonClassifier.test()

The first one gives me

[info]  Label   Precision Recall    F1   LCount PCount
[info] -----------------------------------------------
[info] false       96.874 100.000 98.412  61178  63152
[info] true       100.000   0.804  1.595   1990     16
[info] -----------------------------------------------
[info] Accuracy    96.875    -      -      -     63168

and the second one gives: 

[info]  Label   Precision Recall    F1   LCount PCount
[info] -----------------------------------------------
[info] false       96.850 100.000 98.400  61178  63168
[info] true         0.000   0.000  0.000   1990      0
[info] -----------------------------------------------
[info] Accuracy    96.850    -      -      -     63168

which is clearly different from the first one. Any ideas what can be the reason for this discrepancy?

kordjamshidi commented 8 years ago

To me it should not be related to the order of save and load but just different rounds of doing the test and probably related to rounding and storing the real values?

kordjamshidi commented 8 years ago

Oh, now I see the difference in true label prediction! I had looked at accuracy. This is absolutely a problem!

kordjamshidi commented 8 years ago

Could you try it with giving a specific path for saving and using the same path for load? To be sure we read the same saved model not some default from a different directory?

kordjamshidi commented 8 years ago

I have been doing save and load with giving explicit paths all the time for SRL and luckily did not see this discrepancy.

danyaljj commented 8 years ago

Still no idea why this is happening.... To simulate the issue run this function, and then run this.

kordjamshidi commented 8 years ago

But this is a cross validation setting, I am not sure how the models (we will have five models) are saved in this case and which model you mean to load then? did you try to save and load to a specific path in your previous above example?

kordjamshidi commented 8 years ago

BTW, are you working on the original version of ER? you know that I have a PR for this, right?

danyaljj commented 8 years ago

But this is a cross validation setting, I am not sure how the models (we will have five models) are saved in this case and which model you mean to load then? did you try to save and load to a specific path in your previous above example?

Yes I did. No luck.

BTW, are you working on the original version of ER? you know that I have a PR for this, right?

Yes I know. I am working on a copy of your PR.

kordjamshidi commented 8 years ago

BTW, did you do setModelDir ?

danyaljj commented 8 years ago

Still no luck. I am using SVM, and I printed out the features during train and test, and they are the same. Also the lexicon sizes are exactly the same size.

Here is the train log: 

cogcomp.saulexamples.nlp.EntityMentionRelation.EntityRelationApp
===============================================
Person Classifier Evaluation
Setting the feature extractors to be [prop1]
==> Learning using the feature extractors to be [prop1]
==> Learning using the labeler to be 'prop12_is_Peop'
[prop1]
prop12_is_Peop
Learnable: Learn with data of size 65549
iter  1 act 2.906e+05 pre 2.906e+05 delta 9.366e-01 f 3.277e+05 |g| 6.206e+05 CG   1
cg reaches trust region boundary
iter  2 act 2.302e+03 pre 1.978e+03 delta 1.354e+00 f 3.710e+04 |g| 3.757e+03 CG   1
cg reaches trust region boundary
iter  3 act 3.598e+03 pre 3.595e+03 delta 4.618e+00 f 3.480e+04 |g| 1.489e+04 CG   2
cg reaches trust region boundary
iter  4 act 7.582e+03 pre 6.370e+03 delta 7.869e+00 f 3.120e+04 |g| 2.324e+03 CG   1
iter  5 act 3.550e+02 pre 3.335e+02 delta 7.869e+00 f 2.362e+04 |g| 1.733e+04 CG   1
cg reaches trust region boundary
iter  6 act 8.675e+03 pre 8.279e+03 delta 1.539e+01 f 2.326e+04 |g| 2.034e+03 CG   2
cg reaches trust region boundary
iter  7 act 8.230e+03 pre 7.487e+03 delta 1.837e+01 f 1.459e+04 |g| 2.226e+03 CG   4
iter  8 act 4.020e+03 pre 4.157e+03 delta 1.837e+01 f 6.356e+03 |g| 4.877e+03 CG   8
PersonClassifier.classifier.demandLexicon().size() = 13263
0.1282815081475907-0.06941581058500883-0.06941581058500883-0.14167405235895664-0.07057498544029626-0.14292358950940962--0.8733423960621698--0.8733423960621698-0.0713940738775311-0.08787257648414948-0.13465914464516965-0.392354731071903-0.06885483036197118-0.13153128695210245-0.14047616619251116-0.13342190259509404-0.06941581058500883-0.1776426735484428-0.06941581058500883-0.15311024199122095-0.17684541949047797-0.1829531875038685-0.1282815081475907-0.10773212760622423-0.06941581058500883-0.10779829912955732-0.13950819727170602-0.12450387129349855-0.08558332639608313-0.14531332637927652-0.13153128695210245-0.23476379141473616-0.1282815081475907-0.14765682143291745-0.16277074805874606-0.1096585470874151-0.1643793888110845-0.1096585470874151-0.12947016692093294-0.37121855659982755-0.16629146322414887-0.06941581058500883-0.16662047923315143--0.8733423960621698--0.8733423960621698-0.6642031894773404-0.14440459447846882-0.06941581058500883-0.14358085597576356-0.1096585470874151-0.06941581058500883-0.1363506328721542-0.1282815081475907-0.1096585470874151-0.3691173561026473-0.1453385015312721-0.19226834788150715-0.1096585470874151-0.11128765492612636-0.08612575288775134-0.025959017433296677-0.06576564347605122-0.06576564347605122-0.1168879764449025-0.13723350669317144-0.13168399973034345-0.11309012004901614-0.09509413958318352-0.13301608495839182-0.13859095699275623-0.0942714311595458-0.1453385015312721-0.06941581058500883-0.1096585470874151-0.06941581058500883-0.1442665605828277-0.06941581058500883-0.1293547016217036-0.1096585470874151-0.08055766441693174-0.08420951501145758-0.029582600805578514-0.31970493421744683-0.0031451539719673273-0.1329036064122293-0.1413702773093163-0.1282815081475907-0.06941581058500883-0.06941581058500883-0.09509413958318352-0.1096585470874151-0.06941581058500883-0.12450387129349855-0.1410744771091542-0.1312708932452353-0.1096585470874151-0.06941581058500883-0.14391134261657107-0.17195909966155848-0.13153128695210245-0.1455276501621586-0.06941581058500883-0.1293375834542444-0.06941581058500883-0.1096585470874151-0.1096585470874151--0.8733423960621698--0.8733423960621698-0.13153128695210245-0.1282815081475907-0.4081211276516122-0.13736142906161428-0.13345881364902948-0.11128765492612636-0.06941581058500883-0.13606552219284396-0.14391134261657107-0.06941581058500883-0.1282815081475907-0.1096585470874151--1.1076129275310234-0.11128765492612636-0.34605729415091324-0.12450387129349855-0.1455276501621586-0.2695359490508259-0.13093907684091172-0.002933045484834586-0.12450387129349855-0.13153128695210245-0.1455276501621586-0.1312708932452353-0.12450387129349855-0.1096585470874151--0.8733423960621698--0.8733423960621698-0.2526404380171763-0.1096585470874151-0.08558332639608313-0.06941581058500883-0.1282815081475907-0.13345881364902948-0.06941581058500883-0.14435040953944225-0.13271746035177615-0.13048921360406285-0.06941581058500883-0.13736142906161428-0.06941581058500883-0.042760502715863576-0.042760502715863576-0.042760502715863576-0.11053195790970478-0.06941581058500883-0.06941581058500883--0.6812799073849207--1.3625598147698414-0.06941581058500883-0.16459904652060006-0.1096585470874151-0.06941581058500883-0.13736142906161428-0.1455276501621586-0.0942714311595458-0.06941581058500883-0.13403548714703226-0.14053183736987795-0.13506841021026667-0.1290282337600676-0.13153128695210245-0.0013979764047079481-0.0013979764047079481-0.04071761000857206-0.0013979764047079481-0.1096585470874151-0.13107157733105795-0.12450387129349855-0.11128765492612636-0.13153128695210245-0.13411733606956341-0.1282815081475907-0.145063940620898-0.17332012138979697-0.12450387129349855-0.06941581058500883-0.07126403484442027-0.1303789075227381-0.1375874472328432-0.1096585470874151-0.1096585470874151-0.1282815081475907--0.8733423960621698--0.8733423960621698-0.06941581058500883-0.010606734732675127-0.33261301035203406-0.010606734732675127-0.07495254167947661-0.13072667548805986-0.06941581058500883-0.06941581058500883-0.11420965675633854-0.13153128695210245-0.1096585470874151--1.1076129275310234-0.1096585470874151-0.1096585470874151-0.06941581058500883-0.1096585470874151-0.06941581058500883-0.107051813253855-0.1340925825824144-0.1282815081475907-0.06941581058500883--1.1076129275310234-0.06941581058500883-0.1096585470874151-0.09509413958318352--0.6812799073849207-0.1282815081475907-0.1096585470874151-0.06941581058500883-0.1096585470874151-0.10473738814263048-0.13153128695210245-0.08703210373092377-0.09509413958318352-0.06941581058500883-0.04714094973900198-0.06318224370757602-0.09977428285702518-0.13783520701630742-0.06941581058500883-0.14167405235895664-0.06941581058500883-0.14167405235895664-0.0049312480177115025-0.0815746961768135-0.09659302593328145-0.0049312480177115025-0.1096585470874151-0.06941581058500883-0.06941581058500883-0.08285255456457988-0.13465914464516965-0.11128765492612636-0.11128765492612636-0.1096585470874151-0.24119908697343434-0.08055766441693174--0.8733423960621698--0.8733423960621698-0.06885483036197118-0.06941581058500883-0.08558332639608313-0.11128765492612636-0.11128765492612636-0.06941581058500883-0.12450387129349855-0.08114684774186709-0.1282815081475907-0.1282815081475907-0.06941581058500883-0.0893819910092721-0.0942714311595458-0.08114684774186709-0.13342190259509404-0.13012217925196334-0.0607711486063487-0.13342190259509404-0.08040647945064933-0.13375068083767264-0.1409716806285322-0.14167405235895664-0.11431103799696968-0.13964106325712586-0.08792995673552792-0.08036341099523289-0.07360413566695052-0.06941581058500883-0.31031681383897847-0.06941581058500883-0.06941581058500883-0.14435040953944225-0.14477831893166782-0.12593471289537722-0.14495155630379314-0.13736142906161428-0.08552876422602984-0.13866382705070404-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.0893819910092721-0.06885483036197118-0.08055766441693174-0.12450387129349855-0.4969465427291546-0.1096585470874151-0.1096585470874151-0.1347646585798743-0.10700326349548954-0.13375068083767264-0.13153128695210245-0.11128765492612636-0.13783520701630742-0.1282815081475907-0.06941581058500883-0.1282815081475907-0.13153128695210245-0.06941581058500883-0.1096585470874151-0.14167405235895664-0.09334952002013494-0.16459904652060006-0.11878092757985305-0.09509413958318352-0.12450387129349855-0.131807923290962-0.09714072564157199-0.1096585470874151-0.06941581058500883-0.06941581058500883-0.11128765492612636-0.1312708932452353-0.1096585470874151-0.06941581058500883-0.1096585470874151-0.10700326349548954-0.14391134261657107-0.06941581058500883-0.1096585470874151-0.12450387129349855-0.1282815081475907-0.06941581058500883-0.1096585470874151-0.14167405235895664-0.1413702773093163-0.06941581058500883-0.09509413958318352-0.14435040953944225-0.1312708932452353-0.13783520701630742-0.1282815081475907-0.14391134261657107-0.1282815081475907-0.12450387129349855-0.09764695667244999-0.041500203763482846--1.381321126749219-0.1096585470874151-0.10982483745388368-0.06941581058500883-0.06941581058500883-0.1096585470874151--1.0581328567479156--0.6455045910947622--0.6455045910947622-0.12182277260171175-0.14391134261657107-0.134450961825248-0.11128765492612636-0.1096585470874151-0.12450387129349855-0.06885483036197118-0.1096585470874151-0.06941581058500883-0.13238175752687534-0.08040647945064933-0.13423058196144108-0.06941581058500883-0.1096585470874151-0.09509413958318352-0.06941581058500883-0.1439506175932362--0.042380598554623436-0.1096585470874151-0.06941581058500883-0.1096585470874151-0.1096585470874151-0.06941581058500883-0.1413702773093163-0.13153128695210245-0.13342190259509404-0.12450387129349855-0.13723350669317144-0.06941581058500883-0.14435040953944225-0.10937703812445258-0.11145241474468827-0.09387181034218947-0.14391134261657107-0.1282815081475907-0.07039017198898147-0.2033404033806755-0.06941581058500883-0.13271746035177615-0.1282815081475907-0.06941581058500883-0.1096585470874151-0.1282815081475907-0.12450387129349855-0.1453385015312721-0.13783520701630742-0.1840918503094343-0.1282815081475907
Testing independent entity models: 
===============================================
Person Classifier Evaluation
 Label   Precision Recall   F1   LCount PCount
----------------------------------------------
false       97.917 99.944 98.920  61178  62445
true        95.297 34.623 50.792   1990    723
----------------------------------------------
Accuracy    97.887   -      -      -     63168
Setting the feature extractors to be [prop1]
Setting the labeler to be 'prop12_is_Peop'

And at the test time: 

Setting the feature extractors to be [prop1]
Setting the feature extractors to be [prop1]
Setting the labeler to be 'prop12_is_Peop'
0.1282815081475907-0.06941581058500883-0.06941581058500883-0.14167405235895664-0.07057498544029626-0.14292358950940962--0.8733423960621698--0.8733423960621698-0.0713940738775311-0.08787257648414948-0.13465914464516965-0.392354731071903-0.06885483036197118-0.13153128695210245-0.14047616619251116-0.13342190259509404-0.06941581058500883-0.1776426735484428-0.06941581058500883-0.15311024199122095-0.17684541949047797-0.1829531875038685-0.1282815081475907-0.10773212760622423-0.06941581058500883-0.10779829912955732-0.13950819727170602-0.12450387129349855-0.08558332639608313-0.14531332637927652-0.13153128695210245-0.23476379141473616-0.1282815081475907-0.14765682143291745-0.16277074805874606-0.1096585470874151-0.1643793888110845-0.1096585470874151-0.12947016692093294-0.37121855659982755-0.16629146322414887-0.06941581058500883-0.16662047923315143--0.8733423960621698--0.8733423960621698-0.6642031894773404-0.14440459447846882-0.06941581058500883-0.14358085597576356-0.1096585470874151-0.06941581058500883-0.1363506328721542-0.1282815081475907-0.1096585470874151-0.3691173561026473-0.1453385015312721-0.19226834788150715-0.1096585470874151-0.11128765492612636-0.08612575288775134-0.025959017433296677-0.06576564347605122-0.06576564347605122-0.1168879764449025-0.13723350669317144-0.13168399973034345-0.11309012004901614-0.09509413958318352-0.13301608495839182-0.13859095699275623-0.0942714311595458-0.1453385015312721-0.06941581058500883-0.1096585470874151-0.06941581058500883-0.1442665605828277-0.06941581058500883-0.1293547016217036-0.1096585470874151-0.08055766441693174-0.08420951501145758-0.029582600805578514-0.31970493421744683-0.0031451539719673273-0.1329036064122293-0.1413702773093163-0.1282815081475907-0.06941581058500883-0.06941581058500883-0.09509413958318352-0.1096585470874151-0.06941581058500883-0.12450387129349855-0.1410744771091542-0.1312708932452353-0.1096585470874151-0.06941581058500883-0.14391134261657107-0.17195909966155848-0.13153128695210245-0.1455276501621586-0.06941581058500883-0.1293375834542444-0.06941581058500883-0.1096585470874151-0.1096585470874151--0.8733423960621698--0.8733423960621698-0.13153128695210245-0.1282815081475907-0.4081211276516122-0.13736142906161428-0.13345881364902948-0.11128765492612636-0.06941581058500883-0.13606552219284396-0.14391134261657107-0.06941581058500883-0.1282815081475907-0.1096585470874151--1.1076129275310234-0.11128765492612636-0.34605729415091324-0.12450387129349855-0.1455276501621586-0.2695359490508259-0.13093907684091172-0.002933045484834586-0.12450387129349855-0.13153128695210245-0.1455276501621586-0.1312708932452353-0.12450387129349855-0.1096585470874151--0.8733423960621698--0.8733423960621698-0.2526404380171763-0.1096585470874151-0.08558332639608313-0.06941581058500883-0.1282815081475907-0.13345881364902948-0.06941581058500883-0.14435040953944225-0.13271746035177615-0.13048921360406285-0.06941581058500883-0.13736142906161428-0.06941581058500883-0.042760502715863576-0.042760502715863576-0.042760502715863576-0.11053195790970478-0.06941581058500883-0.06941581058500883--0.6812799073849207--1.3625598147698414-0.06941581058500883-0.16459904652060006-0.1096585470874151-0.06941581058500883-0.13736142906161428-0.1455276501621586-0.0942714311595458-0.06941581058500883-0.13403548714703226-0.14053183736987795-0.13506841021026667-0.1290282337600676-0.13153128695210245-0.0013979764047079481-0.0013979764047079481-0.04071761000857206-0.0013979764047079481-0.1096585470874151-0.13107157733105795-0.12450387129349855-0.11128765492612636-0.13153128695210245-0.13411733606956341-0.1282815081475907-0.145063940620898-0.17332012138979697-0.12450387129349855-0.06941581058500883-0.07126403484442027-0.1303789075227381-0.1375874472328432-0.1096585470874151-0.1096585470874151-0.1282815081475907--0.8733423960621698--0.8733423960621698-0.06941581058500883-0.010606734732675127-0.33261301035203406-0.010606734732675127-0.07495254167947661-0.13072667548805986-0.06941581058500883-0.06941581058500883-0.11420965675633854-0.13153128695210245-0.1096585470874151--1.1076129275310234-0.1096585470874151-0.1096585470874151-0.06941581058500883-0.1096585470874151-0.06941581058500883-0.107051813253855-0.1340925825824144-0.1282815081475907-0.06941581058500883--1.1076129275310234-0.06941581058500883-0.1096585470874151-0.09509413958318352--0.6812799073849207-0.1282815081475907-0.1096585470874151-0.06941581058500883-0.1096585470874151-0.10473738814263048-0.13153128695210245-0.08703210373092377-0.09509413958318352-0.06941581058500883-0.04714094973900198-0.06318224370757602-0.09977428285702518-0.13783520701630742-0.06941581058500883-0.14167405235895664-0.06941581058500883-0.14167405235895664-0.0049312480177115025-0.0815746961768135-0.09659302593328145-0.0049312480177115025-0.1096585470874151-0.06941581058500883-0.06941581058500883-0.08285255456457988-0.13465914464516965-0.11128765492612636-0.11128765492612636-0.1096585470874151-0.24119908697343434-0.08055766441693174--0.8733423960621698--0.8733423960621698-0.06885483036197118-0.06941581058500883-0.08558332639608313-0.11128765492612636-0.11128765492612636-0.06941581058500883-0.12450387129349855-0.08114684774186709-0.1282815081475907-0.1282815081475907-0.06941581058500883-0.0893819910092721-0.0942714311595458-0.08114684774186709-0.13342190259509404-0.13012217925196334-0.0607711486063487-0.13342190259509404-0.08040647945064933-0.13375068083767264-0.1409716806285322-0.14167405235895664-0.11431103799696968-0.13964106325712586-0.08792995673552792-0.08036341099523289-0.07360413566695052-0.06941581058500883-0.31031681383897847-0.06941581058500883-0.06941581058500883-0.14435040953944225-0.14477831893166782-0.12593471289537722-0.14495155630379314-0.13736142906161428-0.08552876422602984-0.13866382705070404-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.0893819910092721-0.06885483036197118-0.08055766441693174-0.12450387129349855-0.4969465427291546-0.1096585470874151-0.1096585470874151-0.1347646585798743-0.10700326349548954-0.13375068083767264-0.13153128695210245-0.11128765492612636-0.13783520701630742-0.1282815081475907-0.06941581058500883-0.1282815081475907-0.13153128695210245-0.06941581058500883-0.1096585470874151-0.14167405235895664-0.09334952002013494-0.16459904652060006-0.11878092757985305-0.09509413958318352-0.12450387129349855-0.131807923290962-0.09714072564157199-0.1096585470874151-0.06941581058500883-0.06941581058500883-0.11128765492612636-0.1312708932452353-0.1096585470874151-0.06941581058500883-0.1096585470874151-0.10700326349548954-0.14391134261657107-0.06941581058500883-0.1096585470874151-0.12450387129349855-0.1282815081475907-0.06941581058500883-0.1096585470874151-0.14167405235895664-0.1413702773093163-0.06941581058500883-0.09509413958318352-0.14435040953944225-0.1312708932452353-0.13783520701630742-0.1282815081475907-0.14391134261657107-0.1282815081475907-0.12450387129349855-0.09764695667244999-0.041500203763482846--1.381321126749219-0.1096585470874151-0.10982483745388368-0.06941581058500883-0.06941581058500883-0.1096585470874151--1.0581328567479156--0.6455045910947622--0.6455045910947622-0.12182277260171175-0.14391134261657107-0.134450961825248-0.11128765492612636-0.1096585470874151-0.12450387129349855-0.06885483036197118-0.1096585470874151-0.06941581058500883-0.13238175752687534-0.08040647945064933-0.13423058196144108-0.06941581058500883-0.1096585470874151-0.09509413958318352-0.06941581058500883-0.1439506175932362--0.042380598554623436-0.1096585470874151-0.06941581058500883-0.1096585470874151-0.1096585470874151-0.06941581058500883-0.1413702773093163-0.13153128695210245-0.13342190259509404-0.12450387129349855-0.13723350669317144-0.06941581058500883-0.14435040953944225-0.10937703812445258-0.11145241474468827-0.09387181034218947-0.14391134261657107-0.1282815081475907-0.07039017198898147-0.2033404033806755-0.06941581058500883-0.13271746035177615-0.1282815081475907-0.06941581058500883-0.1096585470874151-0.1282815081475907-0.12450387129349855-0.1453385015312721-0.13783520701630742-0.1840918503094343-0.1282815081475907-0.1096585470874151-0.06941581058500883-0.12947016692093294-0.13456968910443687-0.1312708932452353-0.0893819910092721-0.14435040953944225-0.13736142906161428-0.14167405235895664-0.06941581058500883-0.023000059247130555-0.1312708932452353-0.14504484988395902-0.12450387129349855-0.06941581058500883-0.13456968910443687-0.13342190259509404-0.06941581058500883-0.13423058196144108-0.13153128695210245-0.15323361260134--0.3554029216169859-0.01144779721100945-0.06941581058500883-0.021975216145517867-0.06941581058500883-0.12450387129349855-0.13736142906161428-0.11128471842145485-0.06938182588657561-0.13941960102851958-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.0893819910092721-0.13783520701630742-0.06941581058500883-0.06941581058500883-0.1282815081475907-0.1329488393966415-0.06941581058500883-0.14435040953944225-0.13783520701630742-0.11128765492612636-0.1096585470874151-0.1453385015312721-0.06941581058500883-0.1293547016217036-0.1282815081475907-0.14435040953944225-0.08285255456457988-0.13736142906161428-0.06941581058500883-0.08114684774186709-0.1282815081475907-0.1096585470874151-0.1096585470874151-0.06941581058500883-0.1312708932452353-0.12182277260171175-0.13345881364902948-0.10700326349548954-0.06941581058500883-0.08558332639608313-0.11128765492612636-0.14441239777576928--1.3338862980701602-0.4405418717810084-0.1096585470874151-0.06941581058500883-0.12182277260171175-0.06941581058500883-0.06941581058500883--1.3625598147698414--0.6812799073849207-0.12450387129349855-0.06941581058500883-0.06941581058500883-0.12450387129349855-0.06941581058500883-0.1096585470874151-0.1413702773093163-0.13465914464516965-0.1455276501621586-0.06941581058500883-0.16459904652060006-0.06941581058500883-0.1096585470874151-0.06941581058500883-0.14435040953944225-0.1096585470874151-0.1096585470874151-0.1282815081475907-0.06941581058500883-0.06941581058500883-0.20469047429668252-0.06941581058500883-0.1096585470874151-0.08055766441693174-0.08151036134938233-0.1413702773093163-0.13783520701630742-0.1453385015312721-0.09586390142379284-0.13723350669317144-0.06941581058500883-0.12450387129349855-0.0942714311595458-0.13153128695210245-0.06941581058500883-0.13783520701630742-0.1096585470874151-0.09083146114891683-0.1331333066404412-0.1282815081475907-0.1282815081475907-0.06941581058500883-0.06941581058500883-0.13783520701630742-0.13271746035177615--1.1076129275310234-0.1282815081475907-0.06941581058500883--0.4126282656531535--0.4126282656531535-0.1096585470874151-0.07106293860782055-0.03143478918910224-0.004799263017100001-0.004799263017100001-0.06941581058500883--0.9442876321026481-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.1096585470874151-0.06941581058500883--1.4088628468558053--0.5228708057638388-0.13503594534027583-0.14435040953944225-0.13736142906161428-0.06941581058500883-0.1282815081475907-0.12450387129349855-0.06941581058500883-0.12450387129349855-0.1312708932452353-0.13153128695210245-0.1312708932452353-0.09509413958318352-0.11128765492612636--1.7269786543710852--0.32681553201411123--0.32681553201411123-0.13132804083999716-0.06885483036197118-0.131807923290962-0.11128765492612636-0.1282815081475907-0.06941581058500883-0.1096585470874151-0.11128765492612636-0.06941581058500883-0.12450387129349855-0.13153128695210245-0.05564382746306326-0.05564382746306326-0.1282815081475907-0.1096585470874151-0.09280309817998451-0.14435040953944225-0.13238175752687534-0.11128765492612636-0.1312708932452353-0.11128765492612636-0.1453385015312721-0.1096585470874151-0.14435040953944225-0.09509413958318352-0.1453385015312721--0.40397870763566573--1.30514132135577--0.40397870763566573-0.046621316115927326-0.014860215569691713-0.014860215569691713-0.014860215569691713-0.8571679307443857-0.014860215569691713-0.0975419604988284-0.043113765527164186-0.04594483256846842--1.4478031519266494--1.1910310694265411-0.08245534296878879-0.01007028772433626-0.1771302648961845-0.0028535462894469306-0.13656844251161268-0.023565186531314668-0.11128765492612636-0.13736142906161428-0.1312708932452353-0.10700326349548954-0.1312708932452353--1.859998758665623-0.11128765492612636-0.1453385015312721-0.06941581058500883-0.1312708932452353--1.453274691578307--0.5914560176084875-0.1116380843666205-0.08847833611335608-0.09509413958318352-0.1312708932452353-0.1416343109582003-0.06885483036197118-0.14493203836178462-0.1312708932452353-0.08876969317708053-0.13048921360406285--1.1076129275310234-0.06941581058500883-0.08558332639608313-0.1413702773093163-0.1453385015312721-0.13736142906161428-0.13153128695210245-0.13783520701630742-0.1282815081475907-0.09509413958318352-0.14307541888353043-0.06941581058500883-0.1096585470874151--0.4087580714025072-0.06941581058500883-0.1331333066404412-0.08890432088070781-0.005418065112739375-0.13342190259509404-0.1248827901911573-0.28877085938171565-0.05187889410778914-0.05799516705939515--2.120572216440089-0.02183768262073439-0.14167405235895664-0.1096585470874151-0.13153128695210245-0.1282815081475907-0.10700326349548954-0.13153128695210245-0.1312708932452353-0.1282815081475907-0.1331333066404412-0.06941581058500883-0.1282815081475907-0.09083146114891683-0.1413702773093163-0.12450387129349855-0.09364777491695854-0.08823042403727883-0.14167405235895664-0.1096585470874151-0.13153128695210245-0.1312708932452353-0.1453385015312721-0.1413702773093163-0.13736142906161428-0.09444223315145722-0.12947016692093294-0.1453385015312721-0.09509413958318352-0.13491559372562179-0.1455276501621586-0.1096585470874151-0.11128765492612636-0.06941581058500883--1.7466847921243396-0.13132804083999716-0.1348121238921774-0.02092353202652291-0.1096585470874151-0.11128765492612636--2.0069770969722454-0.13153128695210245-0.06941581058500883-0.08151036134938233-0.10700326349548954-0.1096585470874151-0.06941581058500883--1.7193886637924347--0.1532596270617469--0.23026712424084128-0.11128765492612636-0.1282815081475907-0.1455276501621586-0.14435040953944225-0.13153128695210245-0.09444223315145722-0.06941581058500883-0.06941581058500883-0.13153128695210245--1.7595013918404427--0.13445356059834104-0.1312708932452353-0.1282815081475907-0.1312708932452353-0.08558332639608313-0.1282815081475907-0.1455276501621586-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.1282815081475907-0.1455276501621586-0.06941581058500883-0.1455276501621586-0.12450387129349855-0.1312708932452353-0.11128765492612636-0.06941581058500883-0.09509413958318352-0.09509413958318352-0.13736142906161428-0.08558332639608313-0.14391134261657107-0.13153128695210245-0.1096585470874151-0.1282815081475907-0.1413702773093163-0.1282815081475907-0.13681516070986754-0.1453385015312721-0.06941581058500883--1.8206664749220525--1.4471546962731219--0.05628959499585023--1.0022768983938863--0.9634483324498422--0.24780032739332883--1.3625598147698414--0.6812799073849207--1.778892246840768--0.08266776929728459--1.3749742606495325--1.5684149559017624-0.09509413958318352-0.11128765492612636-0.09509413958318352-0.06941581058500883-0.1096585470874151-0.1096585470874151-0.12450387129349855-0.11128765492612636-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.1096585470874151-0.1282815081475907-0.09509413958318352-0.1096585470874151-0.06941581058500883-0.06885483036197118-0.09509413958318352-0.1282815081475907-0.11128765492612636-0.1455276501621586-0.13229220491634622-0.09509413958318352-0.1312708932452353-0.1096585470874151-0.06941581058500883-0.08558332639608313-0.06941581058500883-0.14167405235895664-0.11128765492612636-0.06941581058500883-0.1372006436781094-0.14435040953944225-0.06941581058500883-0.13153128695210245-0.13736142906161428-0.13153128695210245-0.1282815081475907--0.43090933698490974-0.12947016692093294--1.5580174462479486--0.7389760609783196-0.1096585470874151-0.134450961825248-0.062747851069275-0.14391134261657107--1.9074649790871157--0.25181244791193946-0.12450387129349855-0.11128765492612636-0.1282815081475907-0.06941581058500883-0.11128765492612636-0.06885483036197118-0.1282815081475907-0.1413702773093163-0.1096585470874151-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.002997652124415813-0.046629165841294506-0.04491992777170558-0.08774608679110832--0.5876256277535478-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.11128765492612636--1.6841428417424529--0.23504196891320017-0.1453385015312721-0.1312708932452353-0.13736142906161428-0.3376574531498319-0.3029504736268075--0.4568773646510144--1.1076129275310234-0.11128765492612636-0.12450387129349855-0.06941581058500883-0.08151036134938233-0.13342190259509404-0.06941581058500883-0.06941581058500883-0.0995721267938715-0.0995721267938715-0.1227843941150997-0.13736142906161428-0.1360038968784628-0.09489487572620617-0.06885483036197118-0.11128765492612636-0.06941581058500883-0.1096585470874151-0.1003640707575288-0.004953249491728147-0.7991732157621189-0.06941581058500883-0.09509413958318352-0.0942714311595458-0.13342190259509404-0.10985239981075767-9.742924568613689E-4-0.1282815081475907-0.13271746035177615-0.11128765492612636-0.1413702773093163-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.10381848593422668-0.06941581058500883-0.06941581058500883-0.1096585470874151-0.06941581058500883--0.8733423960621698--0.8733423960621698-0.06941581058500883--0.8733423960621698--0.8733423960621698-0.06941581058500883-0.12450387129349855-0.1096585470874151--1.7809220558914958-0.1096585470874151-0.1282815081475907-0.11431103799696966-0.01612916258719046-0.01612916258719046-0.14391134261657107-0.1282815081475907-0.13345881364902948-0.06885483036197118-0.06941581058500883-0.13153128695210245-0.0942714311595458-0.14435040953944225--0.9215160478358256--0.06938313203826876--1.3165231592560336--0.053159764729051186--1.7770615879359872--0.9310920386137924-0.41395505529768467-0.846197931080631-0.8362591937690875-0.10700326349548954-0.13403548714703226-0.13520567831877672-0.1455276501621586-0.08055766441693174-0.06941581058500883-0.1453385015312721-0.06941581058500883-0.1282815081475907-0.13736142906161428--7.978020540001319E-4--7.978020540001319E-4-0.1312708932452353-0.06617463433898661-0.014835661442582297-0.0016717169691738186-0.08055766441693174-0.13153128695210245-0.11128765492612636-0.12182277260171175-0.1455276501621586-0.06941581058500883--0.8733423960621698--0.8733423960621698-0.13783520701630742-0.1147780293501193-0.13736142906161428-0.1096585470874151-0.06941581058500883-0.1096585470874151--2.0069770969722454-0.1282815081475907--1.7466847921243396-0.06941581058500883-0.13783520701630742-0.09509413958318352-0.12450387129349855-0.1096585470874151--1.1076129275310234--1.7466847921243396-0.13403548714703226-0.11128765492612636-0.06941581058500883-0.08151036134938233-0.06941581058500883-0.09444223315145722-0.13736142906161428-0.12450387129349855-0.06941581058500883-0.06885483036197118-0.1413702773093163-0.08480010165227805-0.031178612235988334-0.0033279319253003876-0.0033279319253003876-0.0033279319253003876-0.0033279319253003876-0.0033279319253003876-0.0033279319253003876-0.0033279319253003876-0.0033279319253003876-0.0033279319253003876-0.0033279319253003876-0.08151036134938233-0.13153128695210245-0.06941581058500883-0.13153128695210245-0.1282815081475907-0.06941581058500883-0.06941581058500883-0.13736142906161428-0.13101212663541945-0.13153128695210245-0.14167405235895664-0.1413702773093163-0.12450387129349855-0.1096585470874151-0.1455276501621586-0.1413702773093163-0.007198760497187723-0.13153128695210245-0.1347646585798743-0.14621631051276116-0.06941581058500883-0.11128765492612636-0.1096585470874151-0.06941581058500883-0.11128765492612636-0.12450387129349855-0.1282815081475907-0.1096585470874151-0.0828323174791454-0.0852827538918257-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.12450387129349855-0.08151036134938233-0.13783520701630742-0.13153128695210245-0.13153128695210245-0.06941581058500883-0.06941581058500883--1.453274691578307--0.43090933698490974-0.1282815081475907-0.1096585470874151-0.1096585470874151-0.06941581058500883-0.13153128695210245-0.1096585470874151-0.12450387129349855--0.8733423960621698--0.8733423960621698-0.09509413958318352-0.12450387129349855-0.1097293437896297-0.06941581058500883-0.1282815081475907-0.06941581058500883-0.11128765492612636--1.1076129275310234-0.1455276501621586-0.1413702773093163-0.11128765492612636-0.1282815081475907-0.1096585470874151-0.06941581058500883-0.13783520701630742--0.8733423960621698--0.8733423960621698-0.06941581058500883-0.06941581058500883-0.11128765492612636-0.1096585470874151-0.1096585470874151-0.1282815081475907-0.06941581058500883-0.1282815081475907-0.1282815081475907-0.12450387129349855-0.1096585470874151-0.12947016692093294-0.08058687198593852-0.06941581058500883-0.13736142906161428-0.1096585470874151-0.1312708932452353-0.06941581058500883-0.1282815081475907-0.1455276501621586-0.13153128695210245-0.11128765492612636-0.13470869570767113-0.13153128695210245-0.13153128695210245-0.13153128695210245-0.05482927354370755-0.05482927354370755-0.1282815081475907-0.14435040953944225-0.1455276501621586-0.14391134261657107-0.13736142906161428-0.13072667548805986-0.12894631555340155-0.08151036134938233-0.06941581058500883-0.1084182413163338-9.473429743417777E-4-0.06990264509856832-0.0022395593981095726-0.015699235089959584--0.8733423960621698--0.8733423960621698-0.08040647945064933-0.01612916258719046-0.01612916258719046-0.13736142906161428-0.06941581058500883-0.1312708932452353-0.13153128695210245-0.13470869570767113-0.08055766441693174-0.14167405235895664-0.14435040953944225-0.07274308089776102-0.048109546480811935-0.1096585470874151-0.1096585470874151-0.1096585470874151-0.06941581058500883-0.06941581058500883-0.06941581058500883-0.1282815081475907-0.1096585470874151-0.13153128695210245-0.1282815081475907-0.06941581058500883-0.06941581058500883-0.042760502715863576-0.042760502715863576-0.042760502715863576-0.13375068083767264-0.13783520701630742-0.1455276501621586-0.13153128695210245-0.13783520701630742

p.bias = 1.0
p.C = 1.0
p.displayLL = false
p.epsilon = 0.1
p.solverType = L2LOSS_SVM
Testing independent entity models: 
===============================================
Person Classifier Evaluation
 Label   Precision Recall    F1   LCount PCount
-----------------------------------------------
false       96.850 100.000 98.400  61178  63168
true         0.000   0.000  0.000   1990      0
-----------------------------------------------
Accuracy    96.850    -      -      -     63168
PersonClassifier.classifier.demandLexicon().size() = 13263
danyaljj commented 8 years ago

I have found the issue, but now I don't understand why serialization/deserialization in POS works.

After serialization for each instances, we do feature extraction. Then in order to do prediction we need to lookup the features in the Lexicon, but our classifier never finds the features in the Lexicon; the culprit is this like in the equals function of Feature: https://github.com/IllinoisCogComp/lbjava/blob/master/lbjava/src/main/java/edu/illinois/cs/cogcomp/lbjava/classify/Feature.java#L378

If you change this line to f.generatingClassifier.equals(generatingClassifier) things would start to work. (note the asserts right before this line). In other words, there are two generatingClassifier set in the code which have the same string, but different addresses.

One of the generatingClassifier is internal for Lexicon and it is set when saving the lexicon on disk and reading it back.

While the other generatingClassifier is set when we are creating the data model and its properties: https://github.com/IllinoisCogComp/saul/blob/master/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/datamodel/property/CombinedDiscreteProperty.scala#L25

danyaljj commented 8 years ago

Found the difference between POS (and previous tasks) and this one: All previous tasks had name of the property defined in data model, but I define no names for properties. More concretely, for example, here I have: 

  val word = property(tokens) {
    t: ConllRawToken => t.getWords(false).toList
  }

If you add name of the property to its definition things start to work fine: 

  val word = property(tokens, "wordFeature") {
    t: ConllRawToken => t.getWords(false).toList
  }

Still don't understand what causes the issue, when not definition the name for property in here: https://github.com/IllinoisCogComp/saul/blob/master/saul-core/src/main/scala/edu/illinois/cs/cogcomp/saul/datamodel/DataModel.scala#L281-L282

danyaljj commented 8 years ago

Here is the likely explanation: When defining the names for properties in DataModel, the string gets assigned the same address in memory across different runs. While when it gets assigned automatically, the address changes in each run, because of PROPERTIES.size.

danyaljj commented 8 years ago

Fixed in LBJava: https://github.com/IllinoisCogComp/lbjava/pull/50 Closing.