Evaluating perplexity

[philipcori]

Hello, I am trying to evaluate my model using the perplexity metric provided in keras-wrapper. However, it seems that nmt-keras was not integrated with this, as I get the following error:

[12/04/2020 13:16:12] Evaluating on metric ppl
Traceback (most recent call last):
  File "train.py", line 271, in <module>
    nmt_model.trainNet(dataset, training_params)
  File "/data/home/pcori/Chatbot/chat-env/lib/python3.6/site-packages/keras_wrapper/cnn_model.py", line 923, in trainNet
    self.__train(ds, params)
  File "/data/home/pcori/Chatbot/chat-env/lib/python3.6/site-packages/keras_wrapper/cnn_model.py", line 1152, in __train
    initial_epoch=params['epoch_offset'])
  File "/data/home/pcori/Chatbot/chat-env/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/data/home/pcori/Chatbot/chat-env/lib/python3.6/site-packages/keras/engine/training.py", line 1709, in fit_generator
    initial_epoch=initial_epoch)
  File "/data/home/pcori/Chatbot/chat-env/lib/python3.6/site-packages/keras/engine/training_generator.py", line 255, in fit_generator
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/data/home/pcori/Chatbot/chat-env/lib/python3.6/site-packages/keras/callbacks.py", line 152, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/data/home/pcori/Chatbot/chat-env/lib/python3.6/site-packages/keras_wrapper/extra/callbacks.py", line 290, in on_epoch_end
    self.evaluate(epoch, counter_name='epoch')
  File "/data/home/pcori/Chatbot/chat-env/lib/python3.6/site-packages/keras_wrapper/extra/callbacks.py", line 544, in evaluate
    split=s)
TypeError: compute_perplexity() got an unexpected keyword argument 'pred_list'

compute_perplexity() expects arguments y_pred, and y_true, and does not expect extra_vars or pred_list. Is there a way I can still evaluate perplexity?

[lvapeab]

Oh, that function was a utility that I quickly wrote to evaluate a model. It shouldn't be there. I've removed it.

To evaluate the perplexity of a model, you can pass 'perplexity' as a KERAS_METRIC in config.py. This is now done by default:

https://github.com/lvapeab/nmt-keras/blob/5a29099098e47ff84579b5e8c28c7769c952918c/config.py#L32

PS: it requires to update Keras.

[philipcori]

Got it, thanks. However, it only computes perplexity during training. Is there a way to compute perplexity on the validation set? I would like to have the metric_name here be 'perplexity' rather than sacrebleu:

callbacks.append(PrintPerformanceMetricOnEpochEndOrEachNUpdates(nmt_model,
                                                                dataset,
                                                                gt_id='target_text',
                                                                metric_name=['sacrebleu'],
                                                                set_name=['val'],
                                                                batch_size=256,
                                                                each_n_epochs=20,
                                                                extra_vars=search_params,
                                                                reload_epoch=0,
                                                                is_text=True,
                                                                input_text_id=input_text_id,
                                                                index2word_y=vocab,
                                                                sampling_type='max_likelihood',
                                                                beam_search=True,
                                                                save_path=nmt_model.model_path,
                                                                start_eval_on_epoch=0,
                                                                write_samples=True,
                                                                write_type='list',
                                                                verbose=True))

Thanks again for the help.

[lvapeab]

Added (https://github.com/MarcBS/multimodal_keras_wrapper/commit/22d10a950cec72fc64ea727bbfa1c02c2463edfa).

You can now evaluate with Perpleixty as any other metric, e.g.:

    METRICS = ['sacrebleu', 'perplexity']

You'll need to update multimodal-keras-wrapper and nmt-keras.

[philipcori]

Great, thank you.