TzurV commented 4 years ago

Environment info

AttributeError Traceback (most recent call last)

in () 6 input = tokenizer.encode(sequence, return_tensors="pt") 7 mask_token_index = torch.where(input == tokenizer.mask_token_id)[1] ----> 8 token_logits = model(input).logits 9 mask_token_logits = token_logits[0, mask_token_index, :] 10 top_5_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist() AttributeError: 'tuple' object has no attribute 'logits' - `transformers` version: Successfully installed sacremoses-0.0.43 sentencepiece-0.1.91 tokenizers-0.8.1rc2 transformers-3.3.1 - Platform: Masked Language Modeling - Colab PyTorch - Python version: Python 3.6.9 - PyTorch version (GPU?): - Tensorflow version (GPU?): - Using GPU in script?: - Using distributed or parallel set-up in script?: ### Who can help ## Information Model I am using (Bert, XLNet ...): The problem arises when using: * [V ] the official example scripts: (give details below) * [ ] my own modified scripts: (give details below) The tasks I am working on is: * [ ] an official GLUE/SQUaD task: (give the name) * [ ] my own task or dataset: (give details below) ## To reproduce Steps to reproduce the behavior: 1. Summary of the tasks Open Colab Pytorch 2. Masked Language Modeling example 3. ## Expected behavior

patrickvonplaten commented 4 years ago

Can you replace:

token_logits = model(input).logits

by

token_logits = model(input, return_dict=True).logits

and see if the error persists?

patrickvonplaten commented 4 years ago

Can you give a link to the example, so that we can fix the code snippet?

TzurV commented 4 years ago

Hi Patrick,

The suggested change fixed the problem.

Thank you. Tzur

On Wed, 7 Oct 2020 at 18:34, Patrick von Platen notifications@github.com wrote:

Can you give a link to the example, so that we can fix the code snippet?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/7638#issuecomment-705087242, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVYDXL4DBQIJO4MPX3QZS3SJSRBXANCNFSM4SHNAGUQ .

-- Email: tzurvaich@gmail.com Home: +44 (0) 1480 839198 Mobile: +44 (0) 7825 363873 Israel: +972 (0) 3 7201013 Address: 23 Old Pinewood Way, Papworth Everard Cambridge CB23 3GT UK

abhi1868sharma commented 3 years ago

@patrickvonplaten The same issue is on this page https://huggingface.co/transformers/training.html

In these 2 lines outputs = model(input_ids, attention_mask=attention_mask, labels=labels)

outputs = model(input_ids, attention_mask=attention_mask)

Thanks for the help!!

WeiShi-9 commented 2 years ago

I fixed this problem by update transfromers from 3.0.2 to 4.23.1(the latest version in 2022.10.16)

tejavenkat473 commented 11 months ago

import torch from torch.utils.data import DataLoader, TensorDataset from transformers import DistilBertTokenizer, DistilBertForQuestionAnswering, AdamW

Assuming you have already preprocessed the questions and answers

questions = [ "How do I initiate a policy action?", "What is the difference between policy and procedure?", "On average, how long does it take to complete a policy action?", "What happens to the policy once I submit my draft to UPO?", "Who do I talk to if I have a question about the content of a policy?", "Where do I find the previously approved versions of the policy?", "What is the legal sufficiency review?", "Substantive vs. non-substantive change?", "How do I know when the policy was last updated?", "Are there any rules about posting UNT policy on our departmental web page?" ]

answers = [ "To initiate a policy action, contact the University Policy Office at policy@unt.edu.", "A policy is a governing principle that mandates or constrains actions. Procedures outline the steps necessary to implement the policy.", " ", " ", " ", " ", " ", " ", " ", " ",

Rest of answers

]

Make sure questions and answers have the same length

assert len(questions) == len(answers), "Mismatch in the number of questions and answers" labels = torch.tensor([1] * len(questions)) # Assuming all questions have correct answers

def preprocess(questions, answers): tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')

# Combine questions and answers into a list of strings
inputs = [f"{q} {a}" for q, a in zip(questions, answers)]

# Tokenize the combined strings
tokenized_inputs = tokenizer(inputs, return_tensors='pt', padding=True, truncation=True)

# Create tensors for input_ids, attention_mask
input_ids = tokenized_inputs['input_ids']
attention_mask = tokenized_inputs['attention_mask']

return input_ids, attention_mask

input_ids, attention_mask = preprocess(questions, answers)

Create a DataLoader

dataset = TensorDataset(input_ids, attention_mask, labels) dataloader = DataLoader(dataset, batch_size=8, shuffle=True)

Modify the labels to represent whether each question has an answer or not

For simplicity, let's assume 1 represents having an answer and 0 represents not having an answer

labels = torch.tensor([1 if a.strip() else 0 for a in answers])

Update the training loop

def train(model, dataloader, lr=5e-5, epochs=3): device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model.to(device)

optimizer = AdamW(model.parameters(), lr=lr)
criterion = torch.nn.BCEWithLogitsLoss()  # Assuming binary classification (answer or not)

for epoch in range(epochs):
    model.train()
    total_loss = 0.0

    for batch in dataloader:
        input_ids, attention_mask, labels = batch
        input_ids, attention_mask, labels = input_ids.to(device), attention_mask.to(device), labels.to(device)

        optimizer.zero_grad()

        outputs = model(input_ids, attention_mask=attention_mask)
        logits = outputs.logits

        loss = criterion(logits.squeeze(), labels.float())
        total_loss += loss.item()

        loss.backward()
        optimizer.step()

    print(f"Epoch {epoch + 1}, Loss: {total_loss / len(dataloader)}")

Instantiate the model

bert_model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased')

Train the model

train(bert_model, dataloader)

Save the trained model

torch.save(bert_model.state_dict(), 'saved_model.pth')

Load the trained model

loaded_model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased') loaded_model.load_state_dict(torch.load('saved_model.pth')) loaded_model.eval()

Prediction function

def get_answer(question, model, tokenizer): inputs = tokenizer(question, return_tensors='pt') input_ids = inputs['input_ids'] attention_mask = inputs['attention_mask']

with torch.no_grad():
    outputs = model(input_ids, attention_mask=attention_mask)

start_logits, end_logits = outputs.start_logits, outputs.end_logits
start_idx = torch.argmax(start_logits)
end_idx = torch.argmax(end_logits)

answer = tokenizer.decode(input_ids[0, start_idx:end_idx+1], skip_special_tokens=True)

return answer

Example usage

question_to_predict = "How do I initiate a policy action?" answer = get_answer(question_to_predict, loaded_model, DistilBertTokenizer.from_pretrained('distilbert-base-uncased')) print("Answer:", answer)

for the above code i am getting the bellow error

Some weights of DistilBertForQuestionAnswering were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

AttributeError Traceback (most recent call last) ~\AppData\Local\Temp\ipykernel_18184\3727763023.py in 93 94 # Train the model ---> 95 train(bert_model, dataloader) 96 97 # Save the trained model

~\AppData\Local\Temp\ipykernel_18184\3727763023.py in train(model, dataloader, lr, epochs) 79 80 outputs = model(input_ids, attention_mask=attention_mask) ---> 81 logits = outputs.logits 82 83 loss = criterion(logits.squeeze(), labels.float())

AttributeError: 'QuestionAnsweringModelOutput' object has no attribute 'logits'

WeiShi-9 commented 11 months ago

你好，我已经收到您的邮件~