ModuleNotFoundError: No module named 'tokenizers.tokenizers'

[fspanda]

안녕하세요.

KoSenteceBERT를 사용해보려 했는데

sentence_trainsformers 를 불러오는 과정에서

tokenizers.tokenizers 란 모듈이 없다는 에러가 발생합니다.

SKT 버전의 이슈를 참조해서 tokenizer 의 init.py 파일을 수정했는데도 같은 증상입니다.

오류 전문을 첨부합니다.

감사합니다.

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-0adb4463bb6c> in <module>
----> 1 from sentence_transformers import SentenceTransformer, util
      2 import numpy as np
      3 
      4 model_path = './output/training_nli_sts_ETRI_KoBERT-003_bert_eojeol'
      5 

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/sentence_transformers/__init__.py in <module>
      1 __version__ = "0.3.6"
      2 __DOWNLOAD_SERVER__ = 'https://sbert.net/models/'
----> 3 from .datasets import SentencesDataset, SentenceLabelDataset, ParallelSentencesDataset
      4 from .LoggingHandler import LoggingHandler
      5 from .SentenceTransformer import SentenceTransformer

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/sentence_transformers/datasets/__init__.py in <module>
----> 1 from .sampler import *
      2 from .ParallelSentencesDataset import ParallelSentencesDataset
      3 from .SentenceLabelDataset import SentenceLabelDataset
      4 from .SentencesDataset import SentencesDataset
      5 from .EncodeDataset import EncodeDataset

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/sentence_transformers/datasets/sampler/__init__.py in <module>
----> 1 from .LabelSampler import *

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/sentence_transformers/datasets/sampler/LabelSampler.py in <module>
      4 from torch.utils.data import Sampler
      5 import numpy as np
----> 6 from ...datasets import SentenceLabelDataset
      7 
      8 

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/sentence_transformers/datasets/SentenceLabelDataset.py in <module>
      6 import numpy as np
      7 from tqdm import tqdm
----> 8 from .. import SentenceTransformer
      9 from ..readers.InputExample import InputExample
     10 from multiprocessing import Pool, cpu_count

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/sentence_transformers/SentenceTransformer.py in <module>
      8 import requests
      9 import numpy as np
---> 10 import transformers
     11 import torch
     12 from numpy import ndarray

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/transformers/__init__.py in <module>
     53 from .configuration_xlm_roberta import XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, XLMRobertaConfig
     54 from .configuration_xlnet import XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP, XLNetConfig
---> 55 from .data import (
     56     DataProcessor,
     57     InputExample,

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/transformers/data/__init__.py in <module>
      4 
      5 from .metrics import is_sklearn_available
----> 6 from .processors import (
      7     DataProcessor,
      8     InputExample,

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/transformers/data/processors/__init__.py in <module>
      4 
      5 from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels
----> 6 from .squad import SquadExample, SquadFeatures, SquadV1Processor, SquadV2Processor, squad_convert_examples_to_features
      7 from .utils import DataProcessor, InputExample, InputFeatures, SingleSentenceClassificationProcessor
      8 from .xnli import xnli_output_modes, xnli_processors, xnli_tasks_num_labels

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/transformers/data/processors/squad.py in <module>
      9 
     10 from ...file_utils import is_tf_available, is_torch_available
---> 11 from ...tokenization_bert import whitespace_tokenize
     12 from .utils import DataProcessor
     13 

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/transformers/tokenization_bert.py in <module>
     22 from typing import List, Optional
     23 
---> 24 from tokenizers import BertWordPieceTokenizer
     25 
     26 from .tokenization_utils import PreTrainedTokenizer, PreTrainedTokenizerFast

~/python3/Census - 800m Data/ko-sentence/KoSentenceBERT/tokenizers/__init__.py in <module>
     37 
     38 
---> 39 from .tokenizers import Tokenizer, Encoding
     40 from .tokenizers import models
     41 from .tokenizers import normalizers

ModuleNotFoundError: No module named 'tokenizers.tokenizers'
`

[fspanda]

폴더 이동을 안했네요..

죄송합니다 이슈는 닫겠습니다.

[fenneccat]

@fspanda 혹시 어떻게 해결하셨는지요?

[fspanda]

@fenneccat 세세한 오류가 계속 나서 제공해주시는 도커를 다운받아 실행했습니다.