JointTrain Initialization

[kordjamshidi]

I was trying to run the ER jointTraining without loading models and I see it runs while before we got NPE if we had not built lexicons before hand and we used to do it by calling the independent training at least one round. But now when I remove the LoadModels here (https://github.com/IllinoisCogComp/saul/blob/master/saul-examples/src/main/scala/edu/illinois/cs/cogcomp/saulexamples/nlp/EntityRelation/EntityRelationApp.scala#L108), it is still running. I can not see why this is running now, any immediate thought? Did any thing happen in the last modifications?

[danyaljj]

Currently if you don't specify the model location it uses a default (empty) model (and if there is no model yet, it creates a new one). When you disable the load, it would start joint training on empty/fresh classifiers.

[kordjamshidi]

But I see labels in the label lexicon!

[danyaljj]

Did you remove the existing models in your models/ folder?

[kordjamshidi]

No, but I ran Forget.

[danyaljj]

???

[kordjamshidi]

ClassifierUtils.ForgetAll(PersonClassifier,OrganizationClassifier,LocationClassifier,WorksForClassifier, LivesInClassifier, LocatedInClassifier, OrgBasedInClassifier) then JointTrain.train[ConllRelation]...

[danyaljj]

and what happens? Be specific.

[kordjamshidi]

I already started the issue by describing what happens.

[danyaljj]

With all due respect, I think you're wrong. I tested it and things look ok to me. If there is anything you think is wrong/suspicious, you need to spell it out more specifically, or at least put it's logs here, so that I can compare it with what I see.

[kordjamshidi]

What you get? For me, it runs normally while it shouldn't.

[kordjamshidi]

I don't get your point, what information you need me to put here? the results of the run with the numbers? I expected that the joint does not run when we don't call Load but it does.

[kordjamshidi]

16:27:57 INFO EntityRelationClassifiers$PersonClassifier$:84 - Setting the feature extractors to be [prop1, null, prop2, prop4, prop5, prop8] 16:27:57 INFO EntityRelationClassifiers$PersonClassifier$:95 - Setting the labeler to be 'prop12_is_Peop 16:27:57 INFO EntityRelationClassifiers$OrganizationClassifier$:84 - Setting the feature extractors to be [prop1, null, prop2, prop4, prop5, prop8] 16:27:57 INFO EntityRelationClassifiers$OrganizationClassifier$:95 - Setting the labeler to be 'prop12_is_Org 16:27:57 INFO EntityRelationClassifiers$LocationClassifier$:84 - Setting the feature extractors to be [prop1, null, prop2, prop4, prop5, prop8] 16:27:57 INFO EntityRelationClassifiers$LocationClassifier$:95 - Setting the labeler to be 'prop12_is_Loc 16:27:57 INFO EntityRelationClassifiers$WorksForClassifier$:84 - Setting the feature extractors to be [prop9, prop10] 16:27:57 INFO EntityRelationClassifiers$WorksForClassifier$:95 - Setting the labeler to be 'prop13_is_Work_For 16:27:57 INFO EntityRelationClassifiers$LivesInClassifier$:84 - Setting the feature extractors to be [prop9, prop10] 16:27:57 INFO EntityRelationClassifiers$LivesInClassifier$:95 - Setting the labeler to be 'prop13_is_Live_In 16:27:57 INFO EntityRelationClassifiers$LocatedInClassifier$:84 - Setting the feature extractors to be [prop9, prop10, prop11, prop13] 16:27:57 INFO EntityRelationClassifiers$LocatedInClassifier$:95 - Setting the labeler to be 'prop13_is_Located_In 16:27:57 INFO EntityRelationClassifiers$OrgBasedInClassifier$:84 - Setting the feature extractors to be [prop9, prop10, prop11, prop13] 16:27:57 INFO EntityRelationClassifiers$OrgBasedInClassifier$:95 - Setting the labeler to be 'prop13_is_OrgBased_In Joint training 5 iterations. Training iteration: 5 Training iteration: 4 Training iteration: 3 Training iteration: 2 Training iteration: 1

Training iteration: 0

Evaluating PerConstrainedClassifier$

63169 examples tested at Mon Jun 06 16:28:05 CDT 2016

Average evaluation time: 3.71859614684418E-5 seconds

Label Precision Recall F1 LCount PCount

false 97.912 99.889 98.891 61178 62413

true 90.993 34.523 50.055 1990 755

Accuracy 97.830 - - - 63168

Evaluating OrgConstrainedClassifier$

63169 examples tested at Mon Jun 06 16:28:07 CDT 2016

Average evaluation time: 3.603033133340721E-5 seconds

Label Precision Recall F1 LCount PCount

false 98.779 98.011 98.393 61896 61415

true 29.778 41.038 34.512 1272 1753

Accuracy 96.864 - - - 63168

Evaluating LocConstrainedClassifier$

63169 examples tested at Mon Jun 06 16:28:10 CDT 2016

Average evaluation time: 3.663189222561699E-5 seconds

Label Precision Recall F1 LCount PCount

false 97.307 99.498 98.390 60760 62128

true 70.673 30.523 42.633 2408 1040

Accuracy 96.869 - - - 63168

Evaluating WorksFor_PerOrg_ConstrainedClassifier$

970 examples tested at Mon Jun 06 16:28:10 CDT 2016

Average evaluation time: 5.154639175257732E-4 seconds

Label Precision Recall F1 LCount PCount

false 97.032 50.000 65.994 850 438

true 19.962 89.076 32.615 119 531

Accuracy 54.799 - - - 969

Evaluating LivesIn_PerOrg_relationConstrainedClassifier$

970 examples tested at Mon Jun 06 16:28:11 CDT 2016

Average evaluation time: 5.103092783505155E-4 seconds

Label Precision Recall F1 LCount PCount

false 73.549 95.231 82.997 692 896

true 54.795 14.440 22.857 277 73

Accuracy 72.136 - - - 969

[danyaljj]

Did you read what I said earlier?

Currently if you don't specify the model location it uses a default (empty) model (and if there is no model yet, it creates a new one). When you disable the load, it would start joint training on empty/fresh classifiers.

Hence if you don't load the model, the joint training should run without error; it will starts/initializes each of the single classifiers with zero/empty classifier, and will trains each classifier inside the joint training.

[kordjamshidi]

Back to my question: Do you mean we do not need to run independent training before join? How the LabelLexicon of the fresh classifiers is built before seeing any examples? **Notice that JointTraining needs all labels to be in the lexicon when seeing the very first example because it needs to make inference using all labels in the objective.

[kordjamshidi]

Or maybe this is now working only with the SparsePerceptron which is by default binary? I am going to check with SparseNetworks.

[kordjamshidi]

Back to this issue, as far as I can see, the current initialization helps the case of binary SparsePerceptron apparently, so we don't need to run the independent training for that specific case but we still have the old problem with SparseNetworks since the label lexicon is built gradually by observing examples. So the issue #248 still is an issue and the PR #252 still makes sense to be worked out. (let me know if I am wrong.)

[kordjamshidi]

this will be addressed in PR #380

[kordjamshidi]

This seems to be fixed now.