search

LLNL / LUAR

Transformer-based model for learning authorship representations.
Apache License 2.0
25 stars 5 forks source link

AttributeError: add_special_tokens conflicts with the method add_special_tokens in RobertaTokenizerFast #2

Open berkatil opened 1 week ago

berkatil commented 1 week ago

Hi, When I run AutoTokenizer.from_pretrained("rrivera1849/LUAR-CRUD"), I get AttributeError: add_special_tokens conflicts with the method add_special_tokens in RobertaTokenizerFast error. transformers verision is 4.44.2

thanks.

JackOE3 commented 5 days ago

I got the same error.

spookyQubit commented 5 days ago

@berkatil , the requirements.txt mentions transformers==4.14.1 (see https://github.com/LLNL/LUAR/blob/main/requirements.txt#L42C1-L42C21). Maybe that is the reason you are getting this error?

Have you tried installing by following https://github.com/LLNL/LUAR?tab=readme-ov-file#installation ?

berkatil commented 5 days ago

Thank you. Yes, actually I tried with transformers==4.25, it works with that too, probably would work with 4.14.1. I did not realize the requirements file. It might be good to add to huggingface too because I was just checking that page first.