opencv-contrib-python 설치하고 싶습니다

yongjun823 commented 5 years ago

Informations

CLI

OS (Mac, Windows, Linux, etc) and version: Windows 10
client version(please show nsml --version): Built: 2019-01-22T22:32:44+09:00 Go: go1.11.4 GOARCH: amd64 GOOS: windows Protocol: 20181214 Revision: d621f8dcb7d17e9ff1d1be4c8822fb25a4cc05cc (not-modified) Preconfigured connection: hack-cli.nsml.navercorp.com
전체 실행 명령어: nsml run -d ir_ph2 -e rr.py

WEB

Brower (chrome, firfox, etc):
URL: docker build 실패해서 세션이 없습니다

NSML login ID 가 무엇인가요? audzms2776

문제가 발생한 세션은 어떤건가요? (bug message or screenshot)

Building docker image. It might take for a while
...Sending build context to Docker daemon  61.44kB
Step 1/11 : FROM nsml/ml:cuda9.0-cudnn7-tf-1.11torch0.4keras2.2
 ---> 051afca46a0a
Step 2/11 : USER root
 ---> Using cache
 ---> 8cb5acc7adb8
Step 3/11 : WORKDIR /app
 ---> Using cache
 ---> 6d2bf6258f65
Step 4/11 : RUN groupadd -r nsml && useradd --no-log-init -u 1002 -ms /bin/bash -r -g nsml nsml
 ---> Using cache
 ---> 0702d35e0930
Step 5/11 : RUN touch /var/log/nsml.log && chown nsml:nsml /var/log/nsml.log && chown nsml: /app && mkdir /nsml_nfs_output && chown nsml:nsml /nsml_nfs_output
 ---> Using cache
 ---> 634643283aa1
Step 6/11 : LABEL "ai.clova"="1"
 ---> Using cache
 ---> 10a956b30052
Step 7/11 : RUN pip --no-cache-dir install redis==2.10.6 minio==4.0.0 dill pymongo simplejson confluent-kafka[avro]
 ---> Using cache
 ---> c3954a2b8ada
Step 8/11 : RUN echo H4sICJ4LSFwAA3NldHVwLnB5AH2NwWrEMAxE74H8g2EPaaF2k91C6UK+pJSiOt6NSCx7ZSWwf984oS25VAdpeJphDpT8eFZ5Py/CTh28mVovl+hVy0U3pmkksO1r8zI4hnQ0x7IoiwsHrzpMMgmOydjATqGPgUUlJ1PMnlU8lIVahsC7tso9asaEgVQPdgDpA1VPm2V2nB9t1Zj6h3UuWcYoK/83jZQExvGT3W1Cdql933ieKkRHdtY2kDB+6XjPybY9mZOpTfPbtnqTxQFFjw6Ydo8YGUl2CP0VpusOya3z+xhQB2mHgCjcqz+wqY+yeCyLb2EIHSySAQAA | base64 -d | gunzip - > /app/setup.py
 ---> Running in 1b27df88e5a2
Removing intermediate container 1b27df88e5a2
 ---> 6c76a18d78d2
Step 9/11 : RUN cd /app; ([ ! -f setup.py ] || pip install .) && ([ ! -f requirements.txt ] || pip install -r requirements.txt)
 ---> Running in a03d53a27a19
Processing /app
Collecting opencv-contrib-python==3.3.0.10 (from nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/e8/79/1b51e9678f446bed632634973255987deb669154399192a8475bb2e47159/opencv_contrib_python-3.3.0.10-cp35-cp35m-manylinux1_x86_64.whl (21.4MB)
Collecting scikit-learn (from nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/18/d9/bea927c86bf78d583d517f24cbc87606cb333bfb3a5c99cb85b547305f0f/scikit_learn-0.20.2-cp35-cp35m-manylinux1_x86_64.whl (5.3MB)
Collecting pprint (from nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/99/12/b6383259ef85c2b942ab9135f322c0dce83fdca8600d87122d2b0181451f/pprint-0.1.tar.gz
Collecting imgaug (from nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/af/fc/c56a7da8c23122b7c5325b941850013880a7a93c21dc95e2b1ecd4750108/imgaug-0.2.7-py3-none-any.whl (644kB)
Collecting tqdm (from nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/ed/d6/3458d39cf4978f4ece846295e83daf5ece710ab0a4106774f7f7b3a68697/tqdm-4.29.1-py2.py3-none-any.whl (46kB)
Collecting pandas (from nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/5d/d4/6e9c56a561f1d27407bf29318ca43f36ccaa289271b805a30034eb3a8ec4/pandas-0.23.4-cp35-cp35m-manylinux1_x86_64.whl (8.7MB)
Collecting annoy (from nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/b5/28/f8ce7400344090bb5cfa1c9fae3fdb03b9438db082d458686442633d82a3/annoy-1.15.0.tar.gz (635kB)
Requirement already satisfied: numpy>=1.11.1 in /opt/conda/lib/python3.5/site-packages (from opencv-contrib-python==3.3.0.10->nsml-vision-hackathon==1.0) (1.15.4)
Requirement already satisfied: scipy>=0.13.3 in /opt/conda/lib/python3.5/site-packages (from scikit-learn->nsml-vision-hackathon==1.0) (1.2.0)
Requirement already satisfied: six in /opt/conda/lib/python3.5/site-packages (from imgaug->nsml-vision-hackathon==1.0) (1.11.0)
Collecting imageio (from imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/28/b4/cbb592964dfd71a9de6a5b08f882fd334fb99ae09ddc82081dbb2f718c81/imageio-2.4.1.tar.gz (3.3MB)
Collecting scikit-image>=0.11.0 (from imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/82/73/4fbb789c741daf2530a96c74d37f2143162c30d512e68ac6cf3bbb9bf3dc/scikit_image-0.14.2-cp35-cp35m-manylinux1_x86_64.whl (25.2MB)
Requirement already satisfied: Pillow in /opt/conda/lib/python3.5/site-packages (from imgaug->nsml-vision-hackathon==1.0) (5.3.0)
Collecting Shapely (from imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/a2/6c/966fa320a88fc685c956af08135855fa84a1589631256abebf73721c26ed/Shapely-1.6.4.post2-cp35-cp35m-manylinux1_x86_64.whl (1.5MB)
Collecting matplotlib (from imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/ad/4c/0415f15f96864c3a2242b1c74041a806c100c1b21741206c5d87684437c6/matplotlib-3.0.2-cp35-cp35m-manylinux1_x86_64.whl (12.9MB)
Requirement already satisfied: pytz>=2011k in /opt/conda/lib/python3.5/site-packages (from pandas->nsml-vision-hackathon==1.0) (2018.7)
Requirement already satisfied: python-dateutil>=2.5.0 in /opt/conda/lib/python3.5/site-packages (from pandas->nsml-vision-hackathon==1.0) (2.7.5)
Collecting dask[array]>=1.0.0 (from scikit-image>=0.11.0->imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/b1/96/bce48a61f6d84f5db6dc4febb4be96009c95d6bdfc67e08cc55f644ae36b/dask-1.1.0-py2.py3-none-any.whl (699kB)
Collecting PyWavelets>=0.4.0 (from scikit-image>=0.11.0->imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/24/3a/08106f608c5aceced7cd5c628f509f0a10214132a30ca99f5115121f902d/PyWavelets-1.0.1-cp35-cp35m-manylinux1_x86_64.whl (4.4MB)
Collecting networkx>=1.8 (from scikit-image>=0.11.0->imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/f3/f4/7e20ef40b118478191cec0b58c3192f822cace858c19505c7670961b76b2/networkx-2.2.zip (1.7MB)
Collecting cloudpickle>=0.2.1 (from scikit-image>=0.11.0->imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/fc/87/7b7ef3038b4783911e3fdecb5c566e3a817ce3e890e164fc174c088edb1e/cloudpickle-0.6.1-py2.py3-none-any.whl
Collecting cycler>=0.10 (from matplotlib->imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 (from matplotlib->imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/de/0a/001be530836743d8be6c2d85069f46fecf84ac6c18c7f5fb8125ee11d854/pyparsing-2.3.1-py2.py3-none-any.whl (61kB)
Collecting kiwisolver>=1.0.1 (from matplotlib->imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/7e/31/d6fedd4fb2c94755cd101191e581af30e1650ccce7a35bddb7930fed6574/kiwisolver-1.0.1-cp35-cp35m-manylinux1_x86_64.whl (949kB)
Collecting toolz>=0.7.3; extra == "array" (from dask[array]>=1.0.0->scikit-image>=0.11.0->imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/14/d0/a73c15bbeda3d2e7b381a36afb0d9cd770a9f4adc5d1532691013ba881db/toolz-0.9.0.tar.gz (45kB)
Collecting decorator>=4.3.0 (from networkx>=1.8->scikit-image>=0.11.0->imgaug->nsml-vision-hackathon==1.0)
  Downloading https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /opt/conda/lib/python3.5/site-packages (from kiwisolver>=1.0.1->matplotlib->imgaug->nsml-vision-hackathon==1.0) (39.1.0)
Building wheels for collected packages: nsml-vision-hackathon, pprint, annoy, imageio, networkx, toolz
  Running setup.py bdist_wheel for nsml-vision-hackathon: started
  Running setup.py bdist_wheel for nsml-vision-hackathon: finished with status 'done'
  Stored in directory: /tmp/pip-ephem-wheel-cache-4dgibb25/wheels/4b/1c/59/a69d3cb46a693578104e640ef6e42bbb128292df4586f1cb10
  Running setup.py bdist_wheel for pprint: started
  Running setup.py bdist_wheel for pprint: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/42/d4/c6/16a6495aecc1bda5d5857bd036efd50617789ba9bea4a05124
  Running setup.py bdist_wheel for annoy: started
  Running setup.py bdist_wheel for annoy: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/9a/61/5c/2a83890a4531ed5177d6df7f667539316aa65a573cb4d44d71
  Running setup.py bdist_wheel for imageio: started
  Running setup.py bdist_wheel for imageio: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/e0/43/31/605de9372ceaf657f152d3d5e82f42cf265d81db8bbe63cde1
  Running setup.py bdist_wheel for networkx: started
  Running setup.py bdist_wheel for networkx: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/68/f8/29/b53346a112a07d30a5a84d53f19aeadaa1a474897c0423af91
  Running setup.py bdist_wheel for toolz: started
  Running setup.py bdist_wheel for toolz: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/f4/0c/f6/ce6b2d1aa459ee97cc3c0f82236302bd62d89c86c700219463
Successfully built nsml-vision-hackathon pprint annoy imageio networkx toolz
Installing collected packages: opencv-contrib-python, scikit-learn, pprint, imageio, cycler, pyparsing, kiwisolver, matplotlib, toolz, dask, PyWavelets, decorator, networkx, cloudpickle, scikit-image, Shapely, imgaug, tqdm, pandas, annoy, nsml-vision-hackathon
[91mCould not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.5/site-packages/cv2/LICENSE-3RD-PARTY.txt'

[0m[91mYou are using pip version 18.1, however version 19.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[0m
Vision_Magicians/ir_ph2/9: building is failed
[]
Done
Error: Failed to build docker image
time="2019/01/23 15:38:00.905" level=fatal msg="Internal server error"

재현방법은 어떻게 되나요? setup.py에 'opencv-contrib-python', 추가했습니다

예상했던 동작방식은 무엇인가요? 라이브러리 설치

제안하고 싶은 해결방법이 있나요?

nsml-admin commented 5 years ago

안녕하세요.

해당 문제는 새로운 도커 이미지를 만들고, 그 안에 원하는 라이브러리를 설치하셔서 새로 세팅하셔서 해결하시면 됩니다.

다음 문서 링크에서 "apt-get 으로 패키지를 설치해야 할 필요가 있을 경우" 파트를 참조하시기 바랍니다. https://n-clair.github.io/vision-docs/_build/html/ko_KR/contents/session/prepare_a_session_environment.html

yongjun823 commented 5 years ago

FROM nsml/ml:cuda9.0-cudnn7-tf-1.11torch0.4keras2.2
RUN  pip3 install --upgrade pip
RUN pip3 install --user opencv-contrib-python==3.3.0.10

기존 nsml 이미지에 명령어를 추가해서 Dockerfile을 만들었습니다

Step 7/11 : RUN pip --no-cache-dir install redis==2.10.6 minio==4.0.0 dill pymongo simplejson confluent-kafka[avro]
 ---> Running in 9677fec7ca02
Collecting redis==2.10.6
  Downloading https://files.pythonhosted.org/packages/3b/f6/7a76333cf0b9251ecf49efff635015171843d9b977e4ffcf59f9c4428052/redis-2.10.6-py2.py3-none-any.whl (64kB)
Collecting minio==4.0.0
  Downloading https://files.pythonhosted.org/packages/bb/82/7137e57d625b362756ef28e7c44cd32b5c03b87b02057e642062187011e1/minio-4.0.0-py2.py3-none-any.whl (50kB)
Collecting dill
  Downloading https://files.pythonhosted.org/packages/fe/42/bfe2e0857bc284cbe6a011d93f2a9ad58a22cb894461b199ae72cfef0f29/dill-0.2.9.tar.gz (150kB)
Collecting pymongo
  Downloading https://files.pythonhosted.org/packages/93/bf/d5234f8dc5eaae6d8391154f282dbefb7f82c2d8305af60cb5face86e880/pymongo-3.7.2-cp35-cp35m-manylinux1_x86_64.whl (408kB)
Collecting simplejson
  Downloading https://files.pythonhosted.org/packages/e3/24/c35fb1c1c315fc0fffe61ea00d3f88e85469004713dab488dee4f35b0aff/simplejson-3.16.0.tar.gz (81kB)
Collecting confluent-kafka[avro]
  Downloading https://files.pythonhosted.org/packages/30/7b/11880d74f6af94729fa794292127e9df44d3398868a03f0b980fdb4b693a/confluent_kafka-0.11.6-cp35-cp35m-manylinux1_x86_64.whl (3.9MB)
Requirement already satisfied: certifi in /opt/conda/lib/python3.5/site-packages (from minio==4.0.0) (2018.8.24)
Requirement already satisfied: urllib3 in /opt/conda/lib/python3.5/site-packages (from minio==4.0.0) (1.23)
Collecting pytz (from minio==4.0.0)
  Downloading https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl (510kB)
Collecting python-dateutil (from minio==4.0.0)
  Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)
Collecting avro-python3; extra == "avro" (from confluent-kafka[avro])
  Downloading https://files.pythonhosted.org/packages/63/b4/7a81640bb69c844c912987007ba36276c29690d6fde2cd41fec16cd24c3b/avro-python3-1.8.2.tar.gz
Collecting fastavro; extra == "avro" (from confluent-kafka[avro])
  Downloading https://files.pythonhosted.org/packages/bd/60/98df68f3702b84325abb64ffadc5fd6db07734ca8303f79305386b74159e/fastavro-0.21.17-cp35-cp35m-manylinux1_x86_64.whl (1.2MB)
Requirement already satisfied: requests; extra == "avro" in /opt/conda/lib/python3.5/site-packages (from confluent-kafka[avro]) (2.19.1)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.5/site-packages (from python-dateutil->minio==4.0.0) (1.11.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.5/site-packages (from requests; extra == "avro"->confluent-kafka[avro]) (3.0.4)
Requirement already satisfied: idna<2.8,>=2.5 in /opt/conda/lib/python3.5/site-packages (from requests; extra == "avro"->confluent-kafka[avro]) (2.7)
[91mException:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/opt/conda/lib/python3.5/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/opt/conda/lib/python3.5/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError
[0m
Vision_Magicians/ir_ph2/10: building is failed
[]
Done
Error: Failed to build docker image
time="2019/01/23 17:22:21.037" level=fatal msg="Internal server error"

Step 7/11 : RUN pip --no-cache-dir install redis==2.10.6 minio==4.0.0 dill pymongo simplejson confluent-kafka[avro]
 ---> Running in ab66434b2ea6

새로 만든 이미지로 실행시키면 오류가 뜹니다

FROM nsml/ml:cuda9.0-cudnn7-tf-1.11torch0.4keras2.2
RUN  pip3 install --upgrade pip
RUN pip3 install --user opencv-contrib-python==3.3.0.10
RUN pip3 install redis==2.10.6 minio==4.0.0 dill pymongo simplejson

제가 만든 Dockerfile로 테스트해봤는데 7번 명령어에서 --no-cache-dir 이부분을 제거하면 이미지가 잘 생성됩니다

Dockerfile 만들때 추가해야하는 부분이 있을까요?

yongjun823 commented 5 years ago

FROM nsml/ml:cuda9.0-cudnn7-tf-1.11torch0.4keras2.2
RUN  pip3 install --upgrade pip
RUN pip3 install --user opencv-contrib-python==3.3.0.10
RUN pip3 install redis==2.10.6 minio==4.0.0 dill pymongo simplejson confluent-kafka[avro]

Dockerfile 이렇게 만들어도 같은 오류가 발생합니다

nsml-admin commented 5 years ago

안녕하세요, Dockerfile 만들때는 제한이 없지만 위와같이 redis,minio,pymongo등은 nsml에서사용하는 패키지들이라서 세션실행에 필요한 패키지들입니다.

아무래도 opencv-contrib-python 패키지설치하다가 위의 패키지들과 충들이 난것같은데요, 그부분을 고려해서 도커파일을 다시 만들어주셔야 할것같습니다.

그냥 opencv는 이미지내에 이미설치가되어서 따로 설치할필요가없는데요 그부분도 고려해보시기바랍니다.

참고로 nsml/ml:cuda9.0-cudnn7-tf-1.11torch0.4keras2.2가 빌드된 도커파일은 다음 링크에서 확인하실수있습니다. 감사합니다.

Naver-AI-Hackathon / AI-Vision

opencv-contrib-python 설치하고 싶습니다 #204

CLI

WEB