Using in Google Colab

[SalvatoreRa]

Hi,

I wanted to use in Google Colab just to test it, however it seems is not downloading:

!pip install transformers
!pip install SentencePiece
import transformers
import torch
tokenizer = transformers.LlamaTokenizer.from_pretrained('chaoyi-wu/PMC_LLAMA_7B')
model = transformers.LlamaForCausalLM.from_pretrained('chaoyi-wu/PMC_LLAMA_7B')
sentence = 'Hello, doctor' 
batch = tokenizer(
            sentence,
            return_tensors="pt", 
            add_special_tokens=False
        )
with torch.no_grad():
    generated = model.generate(inputs = batch["input_ids"], max_length=200, do_sample=True, top_k=50)
    print('model predict: ',tokenizer.decode(generated[0]))

then is returning this error:

TypeError                                 Traceback (most recent call last)
[<ipython-input-5-192441241e3a>](https://localhost:8080/#) in <cell line: 3>()
      1 import transformers
      2 import torch
----> 3 tokenizer = transformers.LlamaTokenizer.from_pretrained('chaoyi-wu/PMC_LLAMA_7B')
      4 model = transformers.LlamaForCausalLM.from_pretrained('chaoyi-wu/PMC_LLAMA_7B')
      5 sentence = 'Hello, doctor'

TypeError: 'NoneType' object is not callable

thank you for your help

[WeixiongLin]

Thanks for your comments, we would provide an official colab notebook later. There seems to be an error in our tokenizer now.

[WeixiongLin]

You are not using the recommended transformers version(4.28.1). They have remove the LlamaTokenizer module in new transformers version, so it would return None type.

[WeixiongLin]

The google colab has resource limitation so that even PMC-LLaMA 7B cannot be served in colab. We are now quantizing the model and try to serve it in colab.