[BUG] colab환경에서 라이브러리 설치 오류, import 오류

grulla99 commented 1 year ago

🐛 Bug

원래 일주일 전 정도 colab에서 잘 작동하던 kobert가 transformers==4.8.2 혹은 3.0.2 를 설치하는 과정에서 자꾸 해당 오류가 발생하여 transformers의 버전을 최신버전인 4.28.1로 다시 하였더니 이번에는

이 부분에서 오류가 생겼습니다.

그래서 pip install 'git+https://github.com/SKTBrain/KoBERT.git#egg=kobert_tokenizer&subdirectory=kobert_hf' 로 변경하여 실행했더니 이번에는 와 같이 gluonnlp 를 import하는 과정에서 오류가 계속해서 발생합니다.

며칠동한 해결하지 못하고 있습니다.. 부탁드립니다.

To Reproduce

버그를 재현하기 위한 재현절차를 작성해주세요.

- colab 환경에서 실행
- !pip install mxnet !pip install gluonnlp pandas tqdm !pip install sentencepiece !pip install transformers !pip install torch
- pip install 'git+https://github.com/SKTBrain/KoBERT.git#egg=kobert_tokenizer&subdirectory=kobert_hf' or !pip install git+https://git@github.com/SKTBrain/KoBERT.git@master 5 - import torch from torch import nn import torch.nn.functional as F import torch.optim as optim from torch.utils.data import Dataset, DataLoader import gluonnlp as nlp import numpy as np from tqdm import tqdm, tqdm_notebook

Expected behavior

이 버그가 발생하기 이전에는 무난하게 코드가 실행되었습니다. 아마 colab에서 파이썬 버전이 업데이트 되었다고 들었는데 해당 문제인 것 같습니다. 하지만 colab에서 python버전을 3.9 3.7로 바꾸어도 해당 문제들이 반복해서 발생합니다.

Environment

colab환경입니다.

Additional context

이전 환경에서 실행했을 때의 출력창입니다. pip install 햇을때

Output exceeds the size limit. Open the full output data in a text editorLooking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Requirement already satisfied: mxnet in /usr/local/lib/python3.9/dist-packages (1.7.0.post2) Requirement already satisfied: numpy<2.0.0,>1.16.0 in /usr/local/lib/python3.9/dist-packages (from mxnet) (1.22.4) Requirement already satisfied: requests<3,>=2.20.0 in /usr/local/lib/python3.9/dist-packages (from mxnet) (2.27.1) Requirement already satisfied: graphviz<0.9.0,>=0.8.1 in /usr/local/lib/python3.9/dist-packages (from mxnet) (0.8.4) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.20.0->mxnet) (2022.12.7) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.20.0->mxnet) (3.4) Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.20.0->mxnet) (2.0.12) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.20.0->mxnet) (1.25.11) Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Requirement already satisfied: gluonnlp in /usr/local/lib/python3.9/dist-packages (0.10.0) Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (1.5.3) Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (4.65.0) Requirement already satisfied: packaging in /usr/local/lib/python3.9/dist-packages (from gluonnlp) (23.1) Requirement already satisfied: numpy>=1.16.0 in /usr/local/lib/python3.9/dist-packages (from gluonnlp) (1.22.4) Requirement already satisfied: cython in /usr/local/lib/python3.9/dist-packages (from gluonnlp) (0.29.34) Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.9/dist-packages (from pandas) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas) (2022.7.1) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/dist-packages (from python-dateutil>=2.8.1->pandas) (1.16.0) Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Requirement already satisfied: sentencepiece in /usr/local/lib/python3.9/dist-packages (0.1.96) Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Collecting transformers==4.8.2 Using cached transformers-4.8.2-py3-none-any.whl (2.5 MB) Requirement already satisfied: huggingface-hub==0.0.12 in /usr/local/lib/python3.9/dist-packages (from transformers==4.8.2) (0.0.12) ... Successfully installed transformers-4.8.2 Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Requirement already satisfied: torch in /usr/local/lib/python3.9/dist-packages (1.10.1) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch) (4.5.0)

=============== !pip install git+https://git@github.com/SKTBrain/KoBERT.git@master 햇을 때

Output exceeds the size limit. Open the full output data in a text editorLooking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Collecting git+https://****@github.com/SKTBrain/KoBERT.git@master Cloning https://****@github.com/SKTBrain/KoBERT.git (to revision master) to /tmp/pip-req-build-dlt6nomc Running command git clone --filter=blob:none --quiet 'https://****@github.com/SKTBrain/KoBERT.git' /tmp/pip-req-build-dlt6nomc Resolved https://****@github.com/SKTBrain/KoBERT.git to commit 47a69af87928fc24e20f571fe10c3cc9dd9af9a3 Preparing metadata (setup.py) ... done Requirement already satisfied: boto3<=1.15.18 in /usr/local/lib/python3.9/dist-packages (from kobert==0.2.3) (1.15.18) Requirement already satisfied: gluonnlp<=0.10.0,>=0.6.0 in /usr/local/lib/python3.9/dist-packages (from kobert==0.2.3) (0.10.0) Requirement already satisfied: mxnet<=1.7.0.post2,>=1.4.0 in /usr/local/lib/python3.9/dist-packages (from kobert==0.2.3) (1.7.0.post2) Requirement already satisfied: onnxruntime<=1.8.0,==1.8.0 in /usr/local/lib/python3.9/dist-packages (from kobert==0.2.3) (1.8.0) Requirement already satisfied: sentencepiece<=0.1.96,>=0.1.6 in /usr/local/lib/python3.9/dist-packages (from kobert==0.2.3) (0.1.96) Requirement already satisfied: torch<=1.10.1,>=1.7.0 in /usr/local/lib/python3.9/dist-packages (from kobert==0.2.3) (1.10.1) Collecting transformers<=4.8.1,>=4.8.1 Using cached transformers-4.8.1-py3-none-any.whl (2.5 MB) Requirement already satisfied: protobuf in /usr/local/lib/python3.9/dist-packages (from onnxruntime<=1.8.0,==1.8.0->kobert==0.2.3) (3.20.3) Requirement already satisfied: flatbuffers in /usr/local/lib/python3.9/dist-packages (from onnxruntime<=1.8.0,==1.8.0->kobert==0.2.3) (23.3.3) Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.9/dist-packages (from onnxruntime<=1.8.0,==1.8.0->kobert==0.2.3) (1.22.4) Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.9/dist-packages (from boto3<=1.15.18->kobert==0.2.3) (0.3.7) Requirement already satisfied: botocore<1.19.0,>=1.18.18 in /usr/local/lib/python3.9/dist-packages (from boto3<=1.15.18->kobert==0.2.3) (1.18.18) Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.9/dist-packages (from boto3<=1.15.18->kobert==0.2.3) (0.10.0) Requirement already satisfied: cython in /usr/local/lib/python3.9/dist-packages (from gluonnlp<=0.10.0,>=0.6.0->kobert==0.2.3) (0.29.34) Requirement already satisfied: packaging in /usr/local/lib/python3.9/dist-packages (from gluonnlp<=0.10.0,>=0.6.0->kobert==0.2.3) (23.1) Requirement already satisfied: requests<3,>=2.20.0 in /usr/local/lib/python3.9/dist-packages (from mxnet<=1.7.0.post2,>=1.4.0->kobert==0.2.3) (2.27.1) Requirement already satisfied: graphviz<0.9.0,>=0.8.1 in /usr/local/lib/python3.9/dist-packages (from mxnet<=1.7.0.post2,>=1.4.0->kobert==0.2.3) (0.8.4) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch<=1.10.1,>=1.7.0->kobert==0.2.3) (4.5.0) ... Found existing installation: transformers 4.8.2 Uninstalling transformers-4.8.2: Successfully uninstalled transformers-4.8.2 Successfully installed transformers-4.8.1

hooNpk commented 1 year ago

저도 같은 문제를 겪고 있습니다. 며칠째 계속 시도 중입니다.. https://medium.com/google-colab/colab-updated-to-python-3-10-27eb02daa162 6일 전에 colab이 python 3.10으로 올라가면서 발생하는 문제로 추정됩니다.

kibeomi commented 1 year ago

from kobert import get_tokenizer from kobert import get_pytorch_kobert_model

이 부분에서 ModuleNotFoundError: No module named 'kobert' 이런 오류가 생깁니다.

yunseochnn commented 1 year ago

!pip install gluonnlp==0.8.0 이 코드 추가해보세요.

kibeomi commented 1 year ago

!pip install gluonnlp==0.8.0 이 코드 추가해도 안되네요 ㅠㅠ

yunseochnn commented 1 year ago

!pip install gluonnlp==0.8.0 이 코드 추가해도 안되네요 ㅠㅠ

https://complexoftaste.tistory.com/2 여기 코드 참고해보세요.

kibeomi commented 1 year ago

참고해서 해결된 것 같아요. 감사합니다!

grulla99 commented 1 year ago

해당 이슈를 해결해서 close 하겠습니다.

hooNpk commented 1 year ago

다른 분들을 위해 정리하자면 !pip install mxnet !pip install gluonnlp==0.8.0 !pip install tqdm pandas !pip install sentencepiece !pip install transformers !pip install torch 를 한 뒤

!pip install 'git+https://github.com/SKTBrain/KoBERT.git#egg=kobert_tokenizer&subdirectory=kobert_hf' 명령을 실행하니 환경이 세팅 됐습니다.