Juman++でトークナイズできる環境を構築

[tomishima2904]

やったこと

• dockerでJuman++を使える環境を構築 41f3894bb35cc71e1ae4c62f28d7cf787a903706
• tokenizerに早稲田のRoBERTaのやつを使える！

参考文献

• https://github.com/utanaka2000/fairseq
• https://github.com/ku-nlp/jumanpp
• https://huggingface.co/ku-nlp/bart-base-japanese
• https://hub.docker.com/r/kunlp/jumanpp こちらのイメージを使用 (linux-alpine)
• nlp-waseda/roberta-base-japanese-with-auto-jumanpp
• https://github.com/ku-nlp/rhoknp 早稲田のTokenizerを使うのに必要
• Install Rust

環境

# jumanpp --version
Juman++ Version: 2.0.0-d1 / Dictionary: 20210715-14b97cb / LM: K:20210809-0011280 L:20210116-10aab9fb F:20171214-9d125cb

rhoknp==1.3.2
sentencepiece==0.1.99
torch==1.13.1+cpu
transformers==4.30.1

# rustc --version
rustc 1.70.0 (90c541806 2023-05-31)

# rustup --version
rustup 1.26.0 (2023-04-05)

# cargo --version
cargo 1.70.0 (ec8a8a0ca 2023-04-25)