Problems about the evaluation script

[erkamkavak]

Hello, Thank you for the great efforts.

I was able to successfully finetune the Turna model with a custom dataset. But I had some problems running the evaluation script. I had to change the test params with the following in the evaluation script(otherwise it was giving errors):

test_params = {
    'per_device_eval_batch_size': 4,
    'output_dir': './',
    'predict_with_generate': True
}

I don't know if that is necessary for every case but I just want to let you know.

[onurgu]

Can you give a listing of the errors?

Was it completely failing?

[onurgu]

Hi @erkamkavak , can you elaborate?

[erkamkavak]

Sorry for the late reply. When I comment out output_dir from test_params, I get the following error:

  File "inference.py", line 36, in <module>
    results = evaluator.evaluate_model(test_dataset)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "evaluator.py", line 51, in evaluate_model
    trainer = self.initialize_trainer(model)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "evaluator.py", line 139, in initialize_trainer
    test_args = Seq2SeqTrainingArguments(
                ^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Seq2SeqTrainingArguments.__init__() missing 1 required positional argument: 'output_dir'

When I comment out predict_with_generate from test_params, I get the following error:

    text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument 'ids': 'list' object cannot be interpreted as an integer

[zeynepyirmibes]

Hi @erkamkavak, thank you for bringing this error to attention. We fixed the evaluation script in PR #74 .