[HELP] Error during evaluation

[karthikraja95]

Hi @amoussawi

I split up my dataset set into 80% training and 20% testing, and I split up the test data set into two sets of 50%. One split will be used as input to the model to generate predictions, and the other is which the model predictions will be evaluated on.

Then I passed those two testing split into dataframe_to_csr_matrix to get csr matrix and saved them as npz files.

I trained the model using 80% of the train data and saved the model as 'autoencoder_epoch_20.model'. I used CPUs not GPUs.

Then I followed the following lines:

val_input_matrix_1 = sparse.load_npz('val_input.npz') val_test_matrix_1 = sparse.load_npz('val_test.npz')

val_dataset_1 = RecommendationDataset(val_input_matrix_1,val_test_matrix_1)

from recoder.metrics import AveragePrecision, Recall, NDCG

model_file = 'autoencoder_epoch_20.model'

testing = DynamicAutoencoder()

metrics = [NDCG(k=10)]

test_recoder = Recoder(model=model, use_cuda=False) test_recoder.init_from_model_file(model_file)

num_recommendations = 10

test_recoder.evaluate(eval_dataset=val_dataset_1, num_recommendations=num_recommendations, metrics=metrics)

The program worked until the loading the saved model part, and then it fails during the evaluation phase. Can you help me solve it?

Any help would be appreciated.

Heres the Traceback:

---

RuntimeError Traceback (most recent call last)

in 2 3 test_recoder.evaluate(eval_dataset=val_dataset_1, num_recommendations=num_recommendations, ----> 4 metrics=metrics) C:\ProgramData\Anaconda3\lib\site-packages\recoder\model.py in evaluate(self, eval_dataset, num_recommendations, metrics, batch_size, num_users) 555 """ 556 results = self._evaluate(eval_dataset, num_recommendations, metrics, --> 557 batch_size=batch_size, num_users=num_users) 558 for metric in results: 559 log.info('{}: {}'.format(metric, np.mean(results[metric]))) C:\ProgramData\Anaconda3\lib\site-packages\recoder\model.py in _evaluate(self, eval_dataset, num_recommendations, metrics, batch_size, num_users) 520 evaluator = RecommenderEvaluator(recommender, metrics) 521 --> 522 results = evaluator.evaluate(eval_dataset, batch_size=batch_size, num_users=num_users) 523 return results 524 C:\ProgramData\Anaconda3\lib\site-packages\recoder\metrics.py in evaluate(self, eval_dataset, batch_size, num_users, num_workers) 202 input, target = batch 203 --> 204 recommendations = self.recommender.recommend(input) 205 206 relevant_items = [target.interactions_matrix[i].nonzero()[1] for i in range(len(target.users))] C:\ProgramData\Anaconda3\lib\site-packages\recoder\recommender.py in recommend(self, users_hist) 116 117 def recommend(self, users_hist): --> 118 return self.model.recommend(users_hist, self.num_recommendations) C:\ProgramData\Anaconda3\lib\site-packages\recoder\model.py in recommend(self, users_interactions, num_recommendations) 534 list: list of recommended items for each user in users_interactions. 535 """ --> 536 output, input = self.predict(users_interactions, return_input=True) 537 # Set input items output to -inf so that they don't get recommended 538 output[input > 0] = - float('inf') C:\ProgramData\Anaconda3\lib\site-packages\recoder\model.py in predict(self, users_interactions, return_input) 508 input_dense = torch.sparse.FloatTensor(batch.indices, batch.values, batch.size) \ 509 .to(device=self.device).to_dense() --> 510 output = self.model(input_dense, input_users=batch.users.to(device=self.device)) 511 return output, input_dense if return_input else output 512 C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs) 491 result = self._slow_forward(*input, **kwargs) 492 else: --> 493 result = self.forward(*input, **kwargs) 494 for hook in self._forward_hooks.values(): 495 hook_result = hook(self, input, result) C:\ProgramData\Anaconda3\lib\site-packages\recoder\nn.py in forward(self, input, input_users, input_items, target_users, target_items) 237 z = self.noise_layer(z) 238 --> 239 z = self.__en_linear_embedding_layer(input_items, z) 240 z = activation(z, self.activation_type) 241 C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs) 491 result = self._slow_forward(*input, **kwargs) 492 else: --> 493 result = self.forward(*input, **kwargs) 494 for hook in self._forward_hooks.values(): 495 hook_result = hook(self, input, result) C:\ProgramData\Anaconda3\lib\site-packages\recoder\nn.py in forward(self, x, y) 276 277 if self.input_based: --> 278 return F.linear(y, _weight.t(), _bias) 279 else: 280 return F.linear(y, _weight, _bias) C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias) 1404 if input.dim() == 2 and bias is not None: 1405 # fused op is marginally faster -> 1406 ret = torch.addmm(bias, input, weight.t()) 1407 else: 1408 output = input.matmul(weight.t()) RuntimeError: size mismatch, m1: [1 x 121672], m2: [254084 x 200] at ..\aten\src\TH/generic/THTensorMath.cpp:961

[amoussawi]

could you please share the script with which you converted your validation dataframes to csr matrix

[karthikraja95]

X_test is the test dataset ( contains 20% of the whole data)

val_train, val_test = train_test_split(X_test, test_size=0.50, random_state=42)

I used the following code to create validation csr matrix

val_input_matrix, val_input_item_id_map, val_input_user_id_map = dataframe_to_csr_matrix(val_train, user_col='userID', item_col='itemID', inter_col='rating')

val_test_matrix, val_test_item_id_map, val_test_user_id_map = dataframe_to_csr_matrix(val_test, user_col='userID', item_col='itemID', inter_col='rating')

Then I saved them as npz files with the following code

save_npz('val_input.npz', matrix=val_input_matrix)

save_npz('val_test.npz', matrix=val_test_matrix)

After that, I used the script which was in the previous comment and that gives the resulting traceback.

[amoussawi]

when converting your validation dataframe, you have to pass the item_id_map returned when converting your training dataframe to dataframe_to_csr_matrix, so that your validation data map to the same items in the training data (same for user_id_map, but not necessary in your case).

[karthikraja95]

val_input_matrix, val_input_item_id_map, val_input_user_id_map = dataframe_to_csr_matrix(val_train, user_col='userID', item_col='itemID', inter_col='rating', item_id_map=item_id_map, user_id_map=user_id_map )

val_test_matrix, val_test_item_id_map, val_test_user_id_map = dataframe_to_csr_matrix(val_test, user_col='userID', item_col='itemID', inter_col='rating', item_id_map=item_id_map, user_id_map=user_id_map )

_I tried the above code, Now I got a negative column found error

Heres the traceback:_

---

ValueError Traceback (most recent call last)

in 4 inter_col='rating', 5 item_id_map=item_id_map, ----> 6 user_id_map=user_id_map 7 ) 8 C:\ProgramData\Anaconda3\lib\site-packages\recoder\utils.py in dataframe_to_csr_matrix(dataframe, user_col, item_col, inter_col, item_id_map, user_id_map) 62 matrix_inters = dataframe[inter_col] 63 ---> 64 csr_matrix = coo_matrix((matrix_inters, (matrix_users, matrix_items)), shape=matrix_size).tocsr() 65 66 return csr_matrix, item_id_map, user_id_map C:\ProgramData\Anaconda3\lib\site-packages\scipy\sparse\coo.py in __init__(self, arg1, shape, dtype, copy) 190 self.data = self.data.astype(dtype, copy=False) 191 --> 192 self._check() 193 194 def reshape(self, *args, **kwargs): C:\ProgramData\Anaconda3\lib\site-packages\scipy\sparse\coo.py in _check(self) 281 raise ValueError('column index exceeds matrix dimensions') 282 if self.row.min() < 0: --> 283 raise ValueError('negative row index found') 284 if self.col.min() < 0: 285 raise ValueError('negative column index found') ValueError: negative row index found _**I tried removing the user_id_map from the validation set, and ran the following code. I got negative row index found error**_ val_input_matrix, val_input_item_id_map, val_input_user_id_map = dataframe_to_csr_matrix(val_train, user_col='userID', item_col='itemID', inter_col='rating', item_id_map=item_id_map, #user_id_map=user_id_map ) val_test_matrix, val_test_item_id_map, val_test_user_id_map = dataframe_to_csr_matrix(val_test, user_col='userID', item_col='itemID', inter_col='rating', item_id_map=item_id_map, #user_id_map=user_id_map ) Traceback: ValueError Traceback (most recent call last) in 3 item_col='itemID', 4 inter_col='rating', ----> 5 item_id_map=item_id_map, 6 #user_id_map=user_id_map 7 ) C:\ProgramData\Anaconda3\lib\site-packages\recoder\utils.py in dataframe_to_csr_matrix(dataframe, user_col, item_col, inter_col, item_id_map, user_id_map) 62 matrix_inters = dataframe[inter_col] 63 ---> 64 csr_matrix = coo_matrix((matrix_inters, (matrix_users, matrix_items)), shape=matrix_size).tocsr() 65 66 return csr_matrix, item_id_map, user_id_map C:\ProgramData\Anaconda3\lib\site-packages\scipy\sparse\coo.py in __init__(self, arg1, shape, dtype, copy) 190 self.data = self.data.astype(dtype, copy=False) 191 --> 192 self._check() 193 194 def reshape(self, *args, **kwargs): C:\ProgramData\Anaconda3\lib\site-packages\scipy\sparse\coo.py in _check(self) 283 raise ValueError('negative row index found') 284 if self.col.min() < 0: --> 285 raise ValueError('negative column index found') 286 287 def transpose(self, axes=None, copy=False): ValueError: negative column index found **_Any ideas how to solve those errors? Thanks in advance @amoussawi_**

[amoussawi]

it seems you have items in the validation dataset that are missing in the training set (replicated the exception on my side).

one more thing to note, in your case if the training dataset users are not the same one in the validation dataset, then no need to pass the user_id_map returned when converting the training set. however, you should pass the user_id_map returned when converting the input split of the validation set to the dataframe_to_csr_matrix when converting the test split of the validation set.

[karthikraja95]

Okay let me try that and get back to you

[karthikraja95]

Hi @amoussawi

Now I have the same users in the train and test set and I followed the procedure you mentioned

Here's the code

X_train = pd.read_pickle('train_stratified_split.pkl') X_test = pd.read_pickle('test_stratified_split.pkl')

val_train, val_test = train_test_split(X_test, test_size=0.50, random_state=42)

train_matrix, item_id_map, user_id_map = dataframe_to_csr_matrix(X_train, user_col='userID', item_col='itemID', inter_col='rating') val_input_matrix, val_input_item_id_map, val_input_user_id_map = dataframe_to_csr_matrix(val_train, user_col='userID', item_col='itemID', inter_col='rating', user_id_map=user_id_map )

val_test_matrix, val_test_item_id_map, val_test_user_id_map = dataframe_to_csr_matrix(val_test, user_col='userID', item_col='itemID', inter_col='rating', user_id_map=user_id_map )

save_npz('val_input.npz', matrix=val_input_matrix)

save_npz('val_test.npz', matrix=val_test_matrix)

train_matrix_1 = sparse.load_npz('train.npz') val_input_matrix_1 = sparse.load_npz('val_input.npz') val_test_matrix_1 = sparse.load_npz('val_test.npz')

train_dataset_1 = RecommendationDataset(train_matrix_1)

val_dataset_1 = RecommendationDataset(val_input_matrix_1,val_test_matrix_1)

model = DynamicAutoencoder() model_file = 'autoencoder_epoch_20.model'

testing = DynamicAutoencoder()

metrics = [NDCG(k=10)]

test_recoder = Recoder(model=model, use_cuda=False) test_recoder.init_from_model_file(model_file)

NO ERRORS TILL THE ABOVE PART - EVERYTHING IS GOOD

WHEN I RUN THE FOLLOWING CODE IT THROWS ERRORS I TRIED TO DEBUG IT BUT I CANT

num_recommendations = 10

test_recoder.evaluate(eval_dataset=val_dataset_1, num_recommendations=num_recommendations, metrics=metrics, batch_size=100)

HERES THE TRACEBACK

RuntimeError Traceback (most recent call last)

in 2 3 test_recoder.evaluate(eval_dataset=val_dataset_1, num_recommendations=num_recommendations, ----> 4 metrics=metrics, batch_size=100) C:\ProgramData\Anaconda3\lib\site-packages\recoder\model.py in evaluate(self, eval_dataset, num_recommendations, metrics, batch_size, num_users) 555 """ 556 results = self._evaluate(eval_dataset, num_recommendations, metrics, --> 557 batch_size=batch_size, num_users=num_users) 558 for metric in results: 559 log.info('{}: {}'.format(metric, np.mean(results[metric]))) C:\ProgramData\Anaconda3\lib\site-packages\recoder\model.py in _evaluate(self, eval_dataset, num_recommendations, metrics, batch_size, num_users) 520 evaluator = RecommenderEvaluator(recommender, metrics) 521 --> 522 results = evaluator.evaluate(eval_dataset, batch_size=batch_size, num_users=num_users) 523 return results 524 C:\ProgramData\Anaconda3\lib\site-packages\recoder\metrics.py in evaluate(self, eval_dataset, batch_size, num_users, num_workers) 202 input, target = batch 203 --> 204 recommendations = self.recommender.recommend(input) 205 206 relevant_items = [target.interactions_matrix[i].nonzero()[1] for i in range(len(target.users))] C:\ProgramData\Anaconda3\lib\site-packages\recoder\recommender.py in recommend(self, users_hist) 116 117 def recommend(self, users_hist): --> 118 return self.model.recommend(users_hist, self.num_recommendations) C:\ProgramData\Anaconda3\lib\site-packages\recoder\model.py in recommend(self, users_interactions, num_recommendations) 534 list: list of recommended items for each user in users_interactions. 535 """ --> 536 output, input = self.predict(users_interactions, return_input=True) 537 # Set input items output to -inf so that they don't get recommended 538 output[input > 0] = - float('inf') C:\ProgramData\Anaconda3\lib\site-packages\recoder\model.py in predict(self, users_interactions, return_input) 508 input_dense = torch.sparse.FloatTensor(batch.indices, batch.values, batch.size) \ 509 .to(device=self.device).to_dense() --> 510 output = self.model(input_dense, input_users=batch.users.to(device=self.device)) 511 return output, input_dense if return_input else output 512 C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs) 491 result = self._slow_forward(*input, **kwargs) 492 else: --> 493 result = self.forward(*input, **kwargs) 494 for hook in self._forward_hooks.values(): 495 hook_result = hook(self, input, result) C:\ProgramData\Anaconda3\lib\site-packages\recoder\nn.py in forward(self, input, input_users, input_items, target_users, target_items) 237 z = self.noise_layer(z) 238 --> 239 z = self.__en_linear_embedding_layer(input_items, z) 240 z = activation(z, self.activation_type) 241 C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs) 491 result = self._slow_forward(*input, **kwargs) 492 else: --> 493 result = self.forward(*input, **kwargs) 494 for hook in self._forward_hooks.values(): 495 hook_result = hook(self, input, result) C:\ProgramData\Anaconda3\lib\site-packages\recoder\nn.py in forward(self, x, y) 276 277 if self.input_based: --> 278 return F.linear(y, _weight.t(), _bias) 279 else: 280 return F.linear(y, _weight, _bias) C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias) 1404 if input.dim() == 2 and bias is not None: 1405 # fused op is marginally faster -> 1406 ret = torch.addmm(bias, input, weight.t()) 1407 else: 1408 output = input.matmul(weight.t()) RuntimeError: size mismatch, m1: [100 x 32328], m2: [254084 x 200] at ..\aten\src\TH/generic/THTensorMath.cpp:961 **THE TRACEBACK IS SAME AS BEFORE WHEN I FIRST STARTED THE ISSUE**

[amoussawi]

you are passing user_id_map instead of item_id_map. here's how it should be

train_matrix, item_id_map, user_id_map = dataframe_to_csr_matrix(X_train,
user_col='userID',
item_col='itemID',
inter_col='rating')

val_input_matrix, val_input_item_id_map, val_input_user_id_map = dataframe_to_csr_matrix(val_train,
user_col='userID',
item_col='itemID',
inter_col='rating',
item_id_map=item_id_map
)

val_test_matrix, val_test_item_id_map, val_test_user_id_map = dataframe_to_csr_matrix(val_test,
user_col='userID',
item_col='itemID',
inter_col='rating',
item_id_map=item_id_map,
user_id_map=val_input_user_id_map
)

[karthikraja95]

Okay, now I have to split the dataset based on items not based on users. After that, I will try the above code and get back to you. Thanks @amoussawi

[amoussawi]

Closing. Feel free to reopen if you are still facing any issue.