search

J-zin / DHIM

source code for paper "Refining BERT Embeddings for Document Hashing via Mutual Information Maximization"
MIT License
11 stars 1 forks source link

Validation stopping criteria using precision for an unsupervised model? #2

Open casperhansen opened 3 years ago

casperhansen commented 3 years ago

I really enjoyed reading your paper and am excited to see the improvements BERT can also contribute in the domain of unsupervised document hashing.

I am trying to understand the stopping criteria used for training your model and the baselines. As the domain is unsupervised document hashing I excepted the model loss to be used, as no document labels are available for training and validation, but it appears that precision computed from validation labels is used (https://github.com/J-zin/DHIM/blob/main/model/base_model.py#L131). Am I looking in the wrong part of the code? Otherwise, it is quite unclear how the model would perform in an actual unsupervised setting, where the loss is used as stopping criteria, as the loss might not correlate well with the precision score.

J-zin commented 3 years ago

Hi, Casper. Thanks for your interest. Actually, you're right, we use the performance of the validation set as the stopping criteria. We feel sorry that this setting might be a little bit different from VDSH, but we just follow this criteria used in AMMI (https://github.com/karlstratos/ammi/blob/53367a3b70f21bd3d92bc1fb09e907d120da2723/model.py#L152). And we apply this stopping criteria for the baselines.

Another thing I wanna emphasize is, yep, BERT embedding has a strong potential to enhance the performance, but one should carefully tune the model parameters to find better gains. That's the reason why we carefully select the parameters of models for different experimental settings.

Overall, thanks for your interest again. Hope you enjoy the codes and paper. If you still have some questions, please feel free to contact me~