HelloNicoo commented 1 year ago

System Info

transformers version: 4.34.1

Platform: Linux-5.15.0-69-generic-x86_64-with-glibc2.17
Python version: 3.8.18
Huggingface_hub version: 0.17.3
Safetensors version: 0.4.0
Accelerate version: 0.24.1
Accelerate config: - compute_environment: LOCAL_MACHINE
- distributed_type: NO
- mixed_precision: fp16
- use_cpu: True
- debug: False
- num_processes: 1
- machine_rank: 0
- num_machines: 1
- rdzv_backend: static
- same_network: True
- main_training_function: main
- downcast_bf16: no
- tpu_use_cluster: False
- tpu_use_sudo: False
PyTorch version (GPU?): 2.1.0+cu121 (True)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?:
Using distributed or parallel set-up in script?:

Who can help?

No response

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

import torch import warnings from dataclasses import dataclass, field from typing import Dict

import numpy as np import torch.nn as nn from datasets import load_dataset from sklearn.metrics import f1_score, precision_score, recall_score from transformers import (BertModel, BertPreTrainedModel, BertTokenizer, Trainer, TrainingArguments) from transformers.modeling_outputs import SequenceClassifierOutput from transformers.trainer_utils import EvalPrediction

warnings.filterwarnings("ignore")

MODEL_NAME = 'bert-base-chinese' token = BertTokenizer.from_pretrained(MODEL_NAME, local_files_only=True)

@dataclass class ModelArguments: """ 模型参数定义 """ ner_num_labels: int = field(default=2, metadata={"help": "需要预测的标签数量"})

def compute_metrics(eval_output): """ 该函数是回调函数，Trainer会在进行评估时调用该函数。 (如果使用Pycharm等IDE进行调试，可以使用断点的方法来调试该函数，该函数在进行评估时被调用) """ print('qqqqqqqq') preds = eval_output.predictions preds = np.argmax(preds, axis=-1).flatten() labels = eval_output.label_ids.flatten()

labels为0表示为，因此计算时需要去掉该部分

mask = labels != 0
preds = preds[mask]
labels = labels[mask]
metrics = dict()
metrics["f1"] = f1_score(labels, preds, average="macro")
metrics["precision"] = precision_score(labels, preds, average="macro")
metrics["recall"] = recall_score(labels, preds, average="macro")
# 必须以字典的形式返回，后面会用到字典的key
print(metrics)
return metrics

class CustomBertModel(BertPreTrainedModel): """ 自定义模型 """

def __init__(self, config, *model_args, **model_kargs):
    super().__init__(config)
    if "model_args" in model_kargs:
        model_args = model_kargs["model_args"]
        self.config.__dict__.update(model_args.__dict__)
    self._num_labels = self.config.ner_num_labels
    self._bert = BertModel(config, add_pooling_layer=False)
    self._classifier = nn.Linear(
        self.config.hidden_size, self._num_labels)
    self.init_weights()

def forward(
        self,
        input_ids=None,
        attention_mask=None,
        target=None,
        return_dict=None,
):
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    outputs = self._bert(
        input_ids,
        attention_mask=attention_mask,
    )
    sequence_output = outputs[0]
    logits = self._classifier(sequence_output[:, 0, :])
    loss_fct = nn.CrossEntropyLoss()
    loss = loss_fct(logits, target)
    return {'loss': loss}

def tokenize_function(examples): """ map处理数据 """ new_data = token(examples['text'], padding='max_length', truncation=True)

new_data['labels'] = [cate_dict[label] for label in examples["cat_leaf_name_old"]]

return new_data

if name == 'main': model_args = ModelArguments() model = CustomBertModel.from_pretrained(MODEL_NAME, model_args=model_args, local_files_only=True)

dataset = load_dataset('json', data_files='./data/all.json')

shuffled_dataset = dataset.shuffle()
shuffled_dataset_all = shuffled_dataset['train'].train_test_split(
    test_size=0.1)
tokenized_shuffled_dataset_all = shuffled_dataset_all.map(
    tokenize_function, batched=True)
training_args = TrainingArguments(output_dir="custom_bert_model",
                                  evaluation_strategy="epoch",
                                  num_train_epochs=2,
                                  per_device_eval_batch_size=8,
                                  per_device_train_batch_size=8,
                                  do_eval=True,
                                  )

trainer = Trainer(model=model,
                  args=training_args,
                  compute_metrics=compute_metrics,
                  train_dataset=tokenized_shuffled_dataset_all['train'],
                  eval_dataset=tokenized_shuffled_dataset_all['test'],
                  )
trainer.train()
# trainer.save_model('saved_custom_model')
trainer.evaluate()

Expected behavior

Hello, I would like to ask about the situation that rewriting the compute_metrics function does not take effect after rewriting the model part in the transformers framework. Is there any solution Here is my code, can you help me, thx !

amyeroberts commented 1 year ago

Hi @HelloNicoo, thanks for raising an issue.

We get many issues and feature requests and so need to you help us so that we can get through them at a reasonable pace. Could you:

Make sure all of the code is properly formated in markdown - between three backticks i.e. ``` code goes here ```
Give a full description of what the issue is e.g. errors occured with full traceback, your observations, what you've tried so far. In this case, this would involve clarifying what you mean by "take effect" (how this was measure and observed)

github-actions[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

huggingface / transformers

some errors about the compute_mertics function #27426