Foundations --> Transformers

[shashankvasisht]

Hi Goku... I am really thankful for all your amazing tutorials.

I however was facing some issues in the Transformers lecture. There are a few minor bugs here with missing variables and imports; which was not an issue.

The training code however is missing the block:

# Train
best_model = trainer.train(
    num_epochs, patience, train_dataloader, val_dataloader)

Also when i wrote this and ran it, I got an error:

/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:14: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).

/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:15: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  from ipykernel import kernelapp as app
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-68-8d0f0dee99db>](https://localhost:8080/#) in <module>()
      1 # Train
      2 best_model = trainer.train(
----> 3     num_epochs, patience, train_dataloader, val_dataloader)

6 frames
[/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py](https://localhost:8080/#) in dropout(input, p, training, inplace)
   1277     if p < 0.0 or p > 1.0:
   1278         raise ValueError("dropout probability has to be between 0 and 1, " "but got {}".format(p))
-> 1279     return _VF.dropout_(input, p, training) if inplace else _VF.dropout(input, p, training)
   1280 
   1281 

TypeError: dropout(): argument 'input' (position 1) must be Tensor, not str

Apparently, the issue comes from the line :

seq, pool = self.transformer(input_ids=ids, attention_mask=masks)

wherein the "pool" returned is of class string. Upon printing the type and the value of it i get the following :

<class 'str'>
pooler_output

Can you please have a look into this. Thanks in Advance!!

[GokuMohandas]

HI @shashankvasisht, based on the error messages, it looks like you're running this on a local notebook instead of running on Google colab directly? If not, make sure you have the exact version of the transformers (3.0.2) package (their future releases might be not be backwards compatible) and other packages as well (use pip freeze inside the notebook to check). I just reran the notebook and it ran successfully to completion.

[knosing]

@shashankvasisht Hi, I just got the same error and found the issue. This error pops up if you are using the newer 4.0 ver of transformers. So one solution is what @GokuMohandas recommended and the other would be to add the argument return_dict=False when you pass inputs to your model:
seq, pool = self.transformer(input_ids=ids, attention_mask=masks, return_dict=False)

Rest all works fine, cheers!