Use unidic-lite instead of ipadic for Japanese tokenization

KanTakahiro commented 1 month ago

System Info

transformers version: 4.44.0
Platform: Linux-6.9.3-76060903-generic-x86_64-with-glibc2.35
Python version: 3.10.12
Huggingface_hub version: 0.23.3
Safetensors version: 0.4.3
Accelerate version: 0.31.0
Accelerate config: not found
PyTorch version (GPU?): 2.4.0+cu121 (True)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using distributed or parallel set-up in script?: No
Using GPU in script?: Yes
GPU type: NVIDIA GeForce RTX 2080 Ti

Who can help?

@ArthurZucker

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction

In line 383 of this file src/transformers/models/bert_japanese/tokenization_bert_japanese.py, the default dictionary is set to be ipadic and I have to install ipadic-py. But ipadic-py's GitHub page said "You Shouldn't Use This" and recommend using UniDic. However, although I installed unidic-lite only, transformers still need ipadic. I have to modify the transformers source code to use unidic-lite. I changed the line 383 of src/transformers/models/bert_japanese/tokenization_bert_japanese.py:

- mecab_dic: Optional[str] = "ipadic",
+ mecab_dic: Optional[str] = "unidic-lite",

I think the official version should also be updated to use unidic-lite for a modern Japanese tokenization.

My script:

import random
import glob
from tqdm import tqdm

import torch
from torch.utils.data import DataLoader
from transformers import BertJapaneseTokenizer, BertForSequenceClassification
import pytorch_lightning as pl

MODEL_NAME = 'cl-tohoku/bert-base-japanese-whole-word-masking'

tokenizer = BertJapaneseTokenizer.from_pretrained(MODEL_NAME)
bert_sc = BertForSequenceClassification.from_pretrained(
    MODEL_NAME, num_labels=10
)
# bert_sc = bert_sc.cuda()

tokenizer = BertJapaneseTokenizer.from_pretrained(MODEL_NAME)

Expected behavior

Change the default dictionary for Japanese tokenization from ipadic to unidic-lite.

ArthurZucker commented 2 weeks ago

Hey! Sorry for the delay, would you like to open a PR for a fix? As long as the outputs are unaffected this would be nice indeed!

KanTakahiro commented 6 days ago

Hello! I have just open a PR for this issue. Please check it and told me if there is anything I need to adjust or improve.

huggingface / transformers