IndexError: tuple index out of range

[jaskirat8]

To get bootstrapped, I tried to use the example from Readme

from transformers import AutoTokenizer
from petals import AutoDistributedModelForCausalLM
import torch 

# Choose any model available at https://health.petals.dev
model_name = "petals-team/StableBeluga2"  # This one is fine-tuned Llama 2 (70B)

# Connect to a distributed network hosting model layers
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoDistributedModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float32)

# Run the model as if it were on your computer
inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"]
outputs = model.generate(input_ids=inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))  # A cat sat on a mat...

The torch_dtype=torch.float32 was added due to CPU support warning but apart from that rest is the same as the original example yet I am facing the error and unable to complete the inference.

OS : Ubuntu 22.04 CPU : i7-7700K GPU: Nvidia 1070

Please guide if i am missing something here.

[AIAnytime]

Same error... this doesn't work anymore. Creators are also not active.

[daspartho]

+1

running into the same error while trying to generate by calling model.generate() method in the getting started colab notebook.

[daspartho]

found relevant issue huggingface/transformers#10160

[daspartho]

this seems to be an issue with the petals library itself instead of the transformers library since replacing AutoDistributedModelForCausalLM with AutoModelForCausalLM seems to work fine

[jaskirat8]

@daspartho, I have the same thoughts as I have been using the same models for months directly, and it works. I just wanted to validate that it is not something due to my misconfiguration or overlooking some setting; we need to isolate the culprit and work towards PR since other folks are also facing this.

[daspartho]

yes i agree!

also gently pinging @borzunov here

[justheuristic]

[working on it]