Error while reproducing the Colab Example

[mskiki777]

I was trying to reproduce the example notebook https://colab.research.google.com/github/georgianpartners/Multimodal-Toolkit/blob/master/notebooks/text_w_tabular_classification.ipynb#scrollTo=ABT1hK9cRsuk and got the error

---

RuntimeError Traceback (most recent call last)

in ~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in train(self, model_path, trial) 750 751 if self.args.evaluate_during_training and self.global_step % self.args.eval_steps == 0: --> 752 metrics = self.evaluate() 753 self._report_to_hp_search(trial, epoch, metrics) 754 ~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in evaluate(self, eval_dataset) 1155 eval_dataloader = self.get_eval_dataloader(eval_dataset) 1156 -> 1157 output = self.prediction_loop(eval_dataloader, description="Evaluation") 1158 1159 self.log(output.metrics) ~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in prediction_loop(self, dataloader, description, prediction_loss_only) 1236 samples_count = 0 1237 for inputs in tqdm(dataloader, desc=description, disable=disable_tqdm): -> 1238 loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only) 1239 batch_size = inputs[list(inputs.keys())[0]].shape[0] 1240 samples_count += batch_size ~/anaconda3/lib/python3.7/site-packages/transformers/trainer.py in prediction_step(self, model, inputs, prediction_loss_only) 1327 1328 with torch.no_grad(): -> 1329 outputs = model(**inputs) 1330 if has_labels: 1331 loss, logits = outputs[:2] ~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs) 725 result = self._slow_forward(*input, **kwargs) 726 else: --> 727 result = self.forward(*input, **kwargs) 728 for hook in itertools.chain( 729 _global_forward_hooks.values(), ~/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs) 159 return self.module(*inputs[0], **kwargs[0]) 160 replicas = self.replicate(self.module, self.device_ids[:len(inputs)]) --> 161 outputs = self.parallel_apply(replicas, inputs, kwargs) 162 return self.gather(outputs, self.output_device) 163 ~/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in parallel_apply(self, replicas, inputs, kwargs) 169 170 def parallel_apply(self, replicas, inputs, kwargs): --> 171 return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) 172 173 def gather(self, outputs, output_device): ~/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py in parallel_apply(modules, inputs, kwargs_tup, devices) 84 output = results[i] 85 if isinstance(output, ExceptionWrapper): ---> 86 output.reraise() 87 outputs.append(output) 88 return outputs ~/anaconda3/lib/python3.7/site-packages/torch/_utils.py in reraise(self) 426 # have message field 427 raise self.exc_type(message=msg) --> 428 raise self.exc_type(msg) 429 430 RuntimeError: Caught RuntimeError in replica 0 on device 0. Original Traceback (most recent call last): File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker output = module(*input, **kwargs) File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/multimodal_transformers/model/tabular_transformers.py", line 116, in forward numerical_feats) File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/multimodal_transformers/model/tabular_combiner.py", line 426, in forward cat_feats = self.cat_layer(cat_feats) File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/multimodal_transformers/model/layer_utils.py", line 52, in forward layer_inputs.append(layer(input)) File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward return F.linear(input, self.weight, self.bias) File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1690, in linear ret = torch.addmm(bias, input, weight.t()) RuntimeError: mat1 dim 1 must match mat2 dim 0 I did the following modifications to the notebook which I don't think would cause the error: Change 1: Use load_data to create train, val and test datasets ``` # Get Datasets train_dataset = load_data( df_train, data_args.column_info['text_cols'], tokenizer, label_col=data_args.column_info['label_col'], label_list=data_args.column_info['label_list'], categorical_cols=data_args.column_info['cat_cols'], numerical_cols=data_args.column_info['num_cols'], sep_text_token_str=tokenizer.sep_token, ) val_dataset = load_data( df_val, data_args.column_info['text_cols'], tokenizer, label_col=data_args.column_info['label_col'], label_list=data_args.column_info['label_list'], categorical_cols=data_args.column_info['cat_cols'], numerical_cols=data_args.column_info['num_cols'], sep_text_token_str=tokenizer.sep_token, ) test_dataset = load_data( df_test, data_args.column_info['text_cols'], tokenizer, label_col=data_args.column_info['label_col'], label_list=data_args.column_info['label_list'], categorical_cols=data_args.column_info['cat_cols'], numerical_cols=data_args.column_info['num_cols'], sep_text_token_str=tokenizer.sep_token, ) ``` change 2: didn't use cache_dir since Colab Example didn't specify it and the default is None ``` tokenizer = AutoTokenizer.from_pretrained( model_args.model_name_or_path ) config = AutoConfig.from_pretrained(model_args.model_name_or_path) ```

[petulla]

I have this same issue when trying to use load_data() on separate test and train dataframes. Weirdly, one of the two tabular_torch_dataset.TorchTextDataset` returned from load_data will train; the other will not.

I have to use the code with the load from file and setup just as in the colab to make work.

@codeKgu Seems like you put a ton of work into this repo. Would be great to get this fixed.

[spencernelsonucla]

@petulla and @mikiwz - I think I found the reason for this issue. In this line: https://github.com/georgian-io/Multimodal-Toolkit/blob/master/multimodal_transformers/data/load_data.py#L228

The package concatenates the train, val, and test dfs. Then, if you're precessing the categorical features via one hot encoding, which is the default, it will one hot encode with ALL of those dfs together.

For example, say your train df has a categorical feature with values ["a", "b"]. This would get one hot encoded as 2 separate columns (a and b). However, say your test data has values ["a", "c"]. Well, with the way this is currently packaged, the train and test data is concatenated together and so there will be one hot encoding to produce 3 columns (a, b, and c). But, if you load your test dataset separately, you would only one hot encode "a" and "c" - resulting in 2 columns instead of 3. This is the issue. The model was thus trained on 3 columns, but you're giving it 2 columns to predict with.

The way around this is to either not use categorical data, or use label encoding instead:

test_dataset_2 = load_data(
    test_data,
    data_args.column_info['text_cols'],
    tokenizer,
    label_col=data_args.column_info['label_col'],
    label_list=data_args.column_info['label_list'],
    numerical_cols=data_args.column_info['num_cols'],
    sep_text_token_str=tokenizer.sep_token,
    categorical_encode_type="label"
)

[akashsaravanan-georgian]

Closing as this has been answered.