The implementation of your importance sampling for estimating likelihood

[Xiaohui9607]

Hi, I was wondering which part is the implementation of your importance sampling? I couldn't find it...

[phlippe]

Hi, you are right that this seems to be missing from the code, not sure whether I had done it for simplicity back then since the scores didn't differ significantly or whether it was actually missing. In any case, if you want to re-introduce the importance sampling during testing, you can replace the following lines in def eval(...) function in task.py:

batch_size = batch[0].size(0) if isinstance(batch, tuple) else batch.size(0)
batch_nll = self._eval_batch(batch, is_test=is_test)

replace with

batch_size = batch[0].size(0) if isinstance(batch, tuple) else batch.size(0)
if not is_test:
    batch_nll = self._eval_batch(batch, is_test=False)
else:
    all_ll = []
    for _ in range(num_rep):
        batch_ll, _ = -self._eval_batch(batch, is_test=True)
        all_ll.append(batch_ll)
    batch_ll = torch.logsumexp(torch.stack(all_ll, dim=-1), dim=-1) - np.log(num_rep)
    batch_nll = -batch_ll.mean()

Additionally, the task you are interested should not average the batch dimension of the NLL score under is_test.

[Xiaohui9607]

logsumexp should operate on ll, not nll right?

[phlippe]

True, I had in my code that the eval_batch function actually returned log-likelihoods if is_test=True. Sorry for forgetting it here, I'll adjust the code