brmson / dataset-sts

Semantic Text Similarity Dataset Hub
711 stars 206 forks source link

Problem with evaluating "termfreq" model #41

Open harkous opened 7 years ago

harkous commented 7 years ago

Hi,

Thanks for this useful project. There is an issue I guess in the evaluation of the termfreq model.

I'm running: python3 tools/train.py termfreq anssel ./data/anssel/wang/train.csv ./data/anssel/wang/test.csv inp_e_dropout=1/2 nb_epoch=1

However, I get this error:

"... tools/pysts/eval.py", line 28, in binclass_accuracy rawacc = np.sum((ypred > 0.5) == (y > 0.5)) / ypred.shape[0]
TypeError: unorderable types: dict() > float()

The origin of this issue is that y_pred should be replaced by y_pred['score'] for this particular task (as the function predict in termfreq.py returns a dictionary).

This is also the case in the other function aggregate_s0.

Still after fixing this, I get another issue:

... /tools/pysts/eval.py", line 122, in mrr if yy[1] in ysd: TypeError: unhashable type: 'numpy.ndarray'

I appreciate your feedback on this issue or on whether I am running something incorrectly.