evaluation accuracy remains 0

[2proveit]

Thanks for your great work in creating this dataset, I have questions while evaluating llama2-7b-chat on this dataset.

• The accuracy of the llama2-7b-chat output remains 0 when the training goes, here is my code:

  def acc(eval_preds:EvalPrediction):
      logits, labels = eval_preds
      preds = tokenizer.batch_decode(logits, skip_special_tokens=True)
      labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
      save_results(preds, labels) # save results to json file
      preds = [last_boxed_only_string(s) for s in preds]
      correct = 0
      total = 0
      for pred, label in zip(preds, labels):
          if is_equiv(pred, label):
              correct += 1
          total += 1
      return {"accuracy": correct / total}
  return acc

• whether the preprocess is required or recommended to use?