[GenAI] Expertiments with DeBERTa model

[head-iie-vnr]

Running below code

from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline

# Load the DeBERTa model and tokenizer
model_name = "microsoft/deberta-large"
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Create a QA pipeline
qa_pipeline = pipeline("question-answering", model=model, tokenizer=tokenizer)

# Example context and question
context = """
Mahatma Gandhi was a leader of the Indian independence movement against British rule. He led various campaigns and movements advocating for civil rights, social justice, and self-rule (Swaraj). Gandhi's philosophy of nonviolent resistance (Satyagraha) inspired mass participation and drew global attention to the cause of Indian independence. Key movements led by Gandhi included the Non-Cooperation Movement, the Salt March, and the Quit India Movement. His efforts were instrumental in uniting Indians across different regions and communities, ultimately leading to India's independence in 1947.
"""
question = "What was Gandhi's role in the Indian independence movement?"

# Get the answer
answer = qa_pipeline(question=question, context=context)
print("Answer:", answer['answer'])

[head-iie-vnr]

The results are decent. it seems to be getting trained.

Question: What was Mahatma Gandhi's role in the Indian independence movement? Answer: Cooperation Movement, the Salt March, and the Quit India Question: What is the significance of the Salt March? Answer: Cooperation Movement, the Salt March, and the Quit India Question: What were Mahatma Gandhi's main principles? Answer: Cooperation Movement, the Salt March, and the Quit India

But I cant give bigger context. So I would like to retrain the model using QnA.

[head-iie-vnr]

But it was hitting the memory limits soon after the job starts. (11GB to 16GB+)

I will clear some memory and try it again.