search

kwangh / Semantic-Textual-Similarity-Tensorflow

SICK train/test data set, Tensorflow Siamese Network, Semantic Textual Similarity measure
GNU General Public License v3.0
1 stars 4 forks source link

Prediction calibration step #2

Open kurtespinosa opened 7 years ago

kurtespinosa commented 7 years ago

Hi Kyle, I wonder if the prediction calibration step in the original paper (p. 4) has been implemented in your code? I don't seem to find it in your code although you did a conversion of the input values to lie between [0,1]. Particularly, this is their description of the calibration step:

Due to the simple construction of our similarity function, the predictions of our model are constrained to follow the exp(−x) curve and are thus not suited for these evaluation metrics. After training our model, we apply an additional nonparametric regression step to obtain better-calibrated predictions (with respect to MSE). Over the training set, the given labels (under original [1, 5] scale) are regressed against the univariate MaLSTM g-predicted relatedness as the sole covariate, and the fitted regression function is evaluated on the MaLSTM-predicted relatedness of the test pairs to produce adjusted final predictions. We use the classical local-linear estimator discussed in Fan and Gijbels (1992) with bandwidth selected using leave-one-out cross-validation. This calibration step serves as a minor correction for our restrictively simple similarity function (which is necessary to retain interpretability of the sentence representations).

Cheers, Kurt

kwangh commented 7 years ago

Hi Kurt,

That was my issue. I could not manage to include the classical local-linear estimator discussed in Fan and Gijbels (1992). I used a simple linear multiplication to convert both the input values and the test values to lie between [0, 1].

Best, Kyle

kurtespinosa commented 7 years ago

Yeah, I saw that. Thank you Kyle for that information.

Cheers, Kurt

On 8 Apr 2017, at 4:36 pm, Kwang Hun notifications@github.com wrote:

Hi Kurt,

That was my issue too. I could not manage to include the classical local-linear estimator discussed in Fan and Gijbels (1992). I used a simple linear multiplication to convert both the input values and the test values to lie between [0, 1].

Best, Kyle

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kyle29/Semantic-Textual-Similarity-Tensorflow/issues/2#issuecomment-292725595, or mute the thread https://github.com/notifications/unsubscribe-auth/AUBcQls54yf2qDPpJmwHa8OyFzS2Qudtks5rt6l3gaJpZM4M3pyK.