SNLI issue

[chledowski]

Hi, my evaluation is not working with SNLI dataset. The problem is there: https://github.com/facebookresearch/SentEval/blob/master/senteval/snli.py#L95

this list should be converted to a numpy array, afterwards everything works well.

Below the logs:

TypeError                                 Traceback (most recent call last)
<ipython-input-5-70817b0caa6d> in <module>
     30 #                   'BigramShift', 'Tense', 'SubjNumber', 'ObjNumber',
     31 #                   'OddManOut', 'CoordinationInversion']
---> 32 results = se.eval(transfer_tasks)

/data/jchledowski/ss/shooka/SentEval/senteval/engine.py in eval(self, name)
     57         # evaluate on evaluation [name], either takes string or list of strings
     58         if (isinstance(name, list)):
---> 59             self.results = {x: self.eval(x) for x in name}
     60             return self.results
     61 

/data/jchledowski/ss/shooka/SentEval/senteval/engine.py in <dictcomp>(.0)
     57         # evaluate on evaluation [name], either takes string or list of strings
     58         if (isinstance(name, list)):
---> 59             self.results = {x: self.eval(x) for x in name}
     60             return self.results
     61 

/data/jchledowski/ss/shooka/SentEval/senteval/engine.py in eval(self, name)
    119         self.evaluation.do_prepare(self.params, self.prepare)
    120 
--> 121         self.results = self.evaluation.run(self.params, self.batcher)
    122 
    123         return self.results

/data/jchledowski/ss/shooka/SentEval/senteval/snli.py in run(self, params, batcher)
    107 
    108         clf = SplitClassifier(self.X, self.y, config)
--> 109         devacc, testacc = clf.run()
    110         logging.debug('Dev acc : {0} Test acc : {1} for SNLI\n'
    111                       .format(devacc, testacc))

/data/jchledowski/ss/shooka/SentEval/senteval/tools/validation.py in run(self)
    216                 # TODO: Find a hack for reducing nb epoches in SNLI
    217                 clf.fit(self.X['train'], self.y['train'],
--> 218                         validation_data=(self.X['valid'], self.y['valid']))
    219             else:
    220                 clf = LogisticRegression(C=reg, random_state=self.seed)

/data/jchledowski/ss/shooka/SentEval/senteval/tools/classifier.py in fit(self, X, y, validation_data, validation_split, early_stop)
     67         # Preparing validation data
     68         trainX, trainy, devX, devy = self.prepare_split(X, y, validation_data,
---> 69                                                         validation_split)
     70 
     71         # Training

/data/jchledowski/ss/shooka/SentEval/senteval/tools/classifier.py in prepare_split(self, X, y, validation_data, validation_split)
     52 
     53         trainX = torch.from_numpy(trainX).to(device, dtype=torch.float32)
---> 54         trainy = torch.from_numpy(trainy).to(device, dtype=torch.int64)
     55         devX = torch.from_numpy(devX).to(device, dtype=torch.float32)
     56         devy = torch.from_numpy(devy).to(device, dtype=torch.int64)

TypeError: expected np.ndarray (got list)

[nstfk]

hey. i am facing the same problem, did you find a solution?

[chledowski]

yes, you need to convert the self.y into np.array