Prediction calibration step

[kurtespinosa]

Hi aditya1503, thanks for sharing your code. It was stated in your paper (p. 4) that a prediction calibration step was implemented to convert the predictions into values similar to the input. I copied below the exact text in your paper:

Due to the simple construction of our similarity function, the predictions of our model are constrained to follow the exp(−x) curve and are thus not suited for these evaluation metrics. After training our model, we apply an additional nonparametric regression step to obtain better-calibrated predictions (with respect to MSE). Over the training set, the given labels (under original [1, 5] scale) are regressed against the univariate MaLSTM g-predicted relatedness as the sole covariate, and the fitted regression function is evaluated on the MaLSTM-predicted relatedness of the test pairs to produce adjusted final predictions. We use the classical local-linear estimator discussed in Fan and Gijbels (1992) with bandwidth selected using leave-one-out cross-validation. This calibration step serves as a minor correction for our restrictively simple similarity function (which is necessary to retain interpretability of the sentence representations).

I have two clarifications:

1. Could you please point out to me which part in your code this is? Sorry, I'm not sure in which part of the code it is implemented.
2. Would converting the input labels to lie between [0,1] at the start have the same effect?((relatedness_score) - 1) / 4

Cheers, Kurt

[jx00109]

@kurtespinosa I have the same questions, have you already solve those questions?

[kurtespinosa]

I'm sorry mate but I haven't gotten any response yet.