ksugar / samapi

Segment Anything Model API
BSD 3-Clause "New" or "Revised" License
75 stars 12 forks source link

build fail with 0.3.1, was working with 0.3.0 #12

Closed romainGuiet closed 1 year ago

romainGuiet commented 1 year ago

Hi @ksugar ,

With docker I created (16 days ago) a conda env (using a yml file) to create a samapi environment (see below )

DockerFile

FROM nvcr.io/nvidia/pytorch:23.07-py3

...
...
...

# install samapi
COPY conda_envs/env_samapi.yml /tmp/env_samapi.yml
RUN conda env update -n samapi -f /tmp/env_samapi.yml \
    && conda clean --all -f -y \
    && conda env export -n "biop" 

env_samapi.yml

name: "samapi"
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.11
  - pip
  - pip:
      - git+https://github.com/ksugar/samapi.git
prefix: "/opt/conda/envs/samapi"

But now I get an error

...
 => ERROR [16/18] RUN conda env update -n samapi -f /tmp/env_samapi.yml     && conda clean --all -f -y     && conda env export -n "biop"                                       183.2s
------
 > [16/18] RUN conda env update -n samapi -f /tmp/env_samapi.yml     && conda clean --all -f -y     && conda env export -n "biop":
1.667 Channels:
1.667  - conda-forge
1.667  - defaults
1.667 Platform: linux-64
1.667 Collecting package metadata (repodata.json): ...working... done
46.59 Solving environment: ...working... done
47.75 
47.75 Downloading and Extracting Packages

53.45 Preparing transaction: ...working... done
53.66 Verifying transaction: ...working... done
54.83 Executing transaction: ...working... done
57.20 Installing pip dependencies: ...working... Pip subprocess error:
182.9   Running command git clone --filter=blob:none --quiet https://github.com/ksugar/samapi.git /tmp/pip-req-build-c_iehv3a
182.9   Running command git clone --filter=blob:none --quiet https://github.com/ksugar/MobileSAM.git /tmp/pip-install-52glofzl/mobile-sam_7aa91f777a704ab2840ce452a2a8d188
182.9   WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f676c388850>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /packages/f4/74/1c0c87dcf7ee0d69d383448182ca8b675d230f7329411f88067cba874ecd/fastapi-0.96.1-py3-none-any.whl.metadata
182.9 ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11; 3.0.0 Requires-Python >=3.7, <=3.11
182.9 ERROR: Could not find a version that satisfies the requirement torchvision<0.15.0,>=0.14.1 (from samapi) (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.15.0, 0.15.1, 0.15.2)
182.9 ERROR: No matching distribution found for torchvision<0.15.0,>=0.14.1
182.9 
182.9 Ran pip subprocess with arguments:
182.9 ['/opt/conda/envs/samapi/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.gbpw91fb.requirements.txt', '--exists-action=b']
182.9 Pip subprocess output:
182.9 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
182.9 Collecting git+https://github.com/ksugar/samapi.git (from -r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Cloning https://github.com/ksugar/samapi.git to ./pip-req-build-c_iehv3a
182.9   Resolved https://github.com/ksugar/samapi.git to commit 6be6c93d40fc30bdd6d5279c7945e1e8996324a6
182.9   Installing build dependencies: started
182.9   Installing build dependencies: finished with status 'done'
182.9   Getting requirements to build wheel: started
182.9   Getting requirements to build wheel: finished with status 'done'
182.9   Preparing metadata (pyproject.toml): started
182.9   Preparing metadata (pyproject.toml): finished with status 'done'
182.9 Collecting mobile_sam@ git+https://github.com/ksugar/MobileSAM.git (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Cloning https://github.com/ksugar/MobileSAM.git to ./pip-install-52glofzl/mobile-sam_7aa91f777a704ab2840ce452a2a8d188
182.9   Resolved https://github.com/ksugar/MobileSAM.git to commit 41ac534a20a4779fab4f8c84e158ca7c7b40ff26
182.9   Preparing metadata (setup.py): started
182.9   Preparing metadata (setup.py): finished with status 'done'
182.9 Collecting fastapi[all]<0.97.0,>=0.96.0 (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Obtaining dependency information for fastapi[all]<0.97.0,>=0.96.0 from https://files.pythonhosted.org/packages/f4/74/1c0c87dcf7ee0d69d383448182ca8b675d230f7329411f88067cba874ecd/fastapi-0.96.1-py3-none-any.whl.metadata
182.9   Downloading fastapi-0.96.1-py3-none-any.whl.metadata (24 kB)
182.9 Collecting geojson<4.0.0,>=3.0.1 (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Downloading geojson-3.0.1-py3-none-any.whl (15 kB)
182.9 Collecting numpy<2.0.0,>=1.23.5 (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Obtaining dependency information for numpy<2.0.0,>=1.23.5 from https://files.pythonhosted.org/packages/32/6a/65dbc57a89078af9ff8bfcd4c0761a50172d90192eaeb1b6f56e5fbf1c3d/numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
182.9   Downloading numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
182.9 Collecting opencv-python<5.0.0.0,>=4.7.0.72 (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Obtaining dependency information for opencv-python<5.0.0.0,>=4.7.0.72 from https://files.pythonhosted.org/packages/f5/d0/2e455d894ec0d6527e662ad55e70c04f421ad83a6fd0a54c3dd73c411282/opencv_python-4.8.0.76-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
182.9   Downloading opencv_python-4.8.0.76-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
182.9 Collecting pillow<10.0.0,>=9.5.0 (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Downloading Pillow-9.5.0-cp311-cp311-manylinux_2_28_x86_64.whl (3.4 MB)
182.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 28.2 MB/s eta 0:00:00
182.9 Collecting scikit-image<0.22.0,>=0.21.0 (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Obtaining dependency information for scikit-image<0.22.0,>=0.21.0 from https://files.pythonhosted.org/packages/22/c3/c5f3c351d6337a18d07c3fb04475626c106cd3dc3d59b85ec50d07656db0/scikit_image-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
182.9   Downloading scikit_image-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (14 kB)
182.9 Collecting shapely<3.0.0,>=2.0.1 (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Downloading shapely-2.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)
182.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 26.9 MB/s eta 0:00:00
182.9 Collecting timm<0.10.0,>=0.9.2 (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Obtaining dependency information for timm<0.10.0,>=0.9.2 from https://files.pythonhosted.org/packages/7a/bd/2c56be7a3b5bc71cf85a405246b89d5359f942c9f7fb6db6306d9d056092/timm-0.9.7-py3-none-any.whl.metadata
182.9   Downloading timm-0.9.7-py3-none-any.whl.metadata (58 kB)
182.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.8/58.8 kB 33.6 MB/s eta 0:00:00
182.9 Collecting torch<2.0.0,>=1.13.1 (from samapi==0.3.1->-r /tmp/condaenv.gbpw91fb.requirements.txt (line 1))
182.9   Downloading torch-1.13.1-cp311-cp311-manylinux1_x86_64.whl (887.4 MB)
182.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 887.4/887.4 MB 14.0 MB/s eta 0:00:00
182.9 INFO: pip is looking at multiple versions of samapi to determine which version is compatible with other requirements. This could take a while.
182.9
182.9 failed
182.9
182.9 CondaEnvException: Pip failed

I tried to modify to the version below (from the updated README)

name: "samapi"
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - cudatoolkit=11.7
  - pip
  - pip:
      - git+https://github.com/ksugar/samapi.git
prefix: "/opt/conda/envs/samapi"

but still get an ERROR :

 => ERROR [16/18] RUN conda env update -n samapi -f /tmp/env_samapi.yml     && conda clean --all -f -y     && conda env export -n "biop"                                       118.5s 
------
 > [16/18] RUN conda env update -n samapi -f /tmp/env_samapi.yml     && conda clean --all -f -y     && conda env export -n "biop":
1.982 Channels:
1.982  - conda-forge
1.982  - defaults
1.982 Platform: linux-64
1.982 Collecting package metadata (repodata.json): ...working... done
51.91 Solving environment: ...working... done
53.11
53.11 Downloading and Extracting Packages
libsqlite-3.43.0     | 821 KB    |            |   0%
53.14 cxx-ng-13.1.0  | 3.7 MB    |            |   0%
53.14 s-6.4          | 860 KB    |            |   0%
53.14 ools-68.1.2    | 451 KB    |            |   0%
53.14 -3.10.12       | 24.4 MB   |            |   0%
53.14 1.0.8          | 484 KB    |            |   0%
libsqlite-3.43.0     | 821 KB    | ########## | 100% 
53.98 cxx-ng-13.1.0  | 3.7 MB    | #########2 |  92% 
54.04 s-6.4          | 860 KB    | 1          |   2% 
54.04 ools-68.1.2    | 451 KB    | 3          |   4% 
54.04 -3.10.12       | 24.4 MB   | ##9        |  30% 
54.05 1.0.8          | 484 KB    | 3          |   3% 
54.05 0.41.2         | 56 KB     | ########## | 100% 
57.39 cxx-ng-13.1.0  | 3.7 MB    | ########## | 100% 
57.39 p_mutex-4.5    | 23 KB     | ######9    |  69% 
57.45 ools-68.1.2    | 451 KB    | ########## | 100% 
57.45 -3.10.12       | 24.4 MB   | #########3 |  94% 
57.45 1.0.8          | 484 KB    | ########## | 100% 
57.45 l_linux-64-2.4 | 688 KB    | 2          |   2% 
57.45 -ng-13.1.0     | 758 KB    | 2          |   2% 
57.45 p_mutex-4.5    | 23 KB     | ########## | 100% 
57.52 .6             | 409 KB    | ########## | 100% 
57.52 l-3.1.2        | 2.5 MB    | ########## | 100% 
57.52 tificates-2023 | 146 KB    | ########## | 100% 
57.52 l_linux-64-2.4 | 688 KB    | ########## | 100% 
75.70 -ng-13.1.0     | 758 KB    | ########## | 100% 
75.70 s-6.4          | 860 KB    | ########## | 100% 

75.71                                                
75.71                                                
75.71                                                
75.71 Preparing transaction: ...working... done      
75.92 Verifying transaction: ...working... done      
77.24 Executing transaction: ...working... By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html       
79.84                                                
79.84 done                                           
79.91 Installing pip dependencies: ...working... Pip subprocess error:
96.66   Running command git clone --filter=blob:none --quiet https://github.com/ksugar/samapi.git /tmp/pip-req-build-zqqgui68
96.66   fatal: unable to access 'https://github.com/ksugar/samapi.git/': Could not resolve host: github.com
96.66   error: subprocess-exited-with-error          
96.66                                                
96.66   × git clone --filter=blob:none --quiet https://github.com/ksugar/samapi.git /tmp/pip-req-build-zqqgui68 did not run successfully.
96.66   │ exit code: 128
96.66   ╰─> See above for output.
96.66   
96.66   note: This error originates from a subprocess, and is likely not a problem with pip.
96.66 error: subprocess-exited-with-error
96.66
96.66 × git clone --filter=blob:none --quiet https://github.com/ksugar/samapi.git /tmp/pip-req-build-zqqgui68 did not run successfully.
96.66 │ exit code: 128
96.66 ╰─> See above for output.
96.66
96.66 note: This error originates from a subprocess, and is likely not a problem with pip.
96.66
96.66 Ran pip subprocess with arguments:
96.66 ['/opt/conda/envs/samapi/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.zcouky75.requirements.txt', '--exists-action=b']
96.66 Pip subprocess output:
96.66 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
96.66 Collecting git+https://github.com/ksugar/samapi.git (from -r /tmp/condaenv.zcouky75.requirements.txt (line 1))
96.66   Cloning https://github.com/ksugar/samapi.git to ./pip-req-build-zqqgui68
96.66
96.66 failed
96.66
96.66 CondaEnvException: Pip failed
96.66
------
Dockerfile:95
--------------------
  94 |     COPY conda_envs/env_samapi.yml /tmp/env_samapi.yml
  95 | >>> RUN conda env update -n samapi -f /tmp/env_samapi.yml \
  96 | >>>     && conda clean --all -f -y \
  97 | >>>     && conda env export -n "biop"
  98 |
--------------------
ERROR: failed to solve: process "/bin/sh -c conda env update -n samapi -f /tmp/env_samapi.yml     && conda clean --all -f -y     && conda env export -n \"biop\"" did not complete successfully: exit code: 1

Could it be because of https://github.com/ksugar/samapi/commit/ab8845eb131e96d41ed4a52ee9704aeba76135ad ?

Thank you for your inputs,

Cheers,

R

ksugar commented 1 year ago

Hi @romainGuiet, thank you for sharing the issue.

I think the direct cause was shown in the following line.

96.66 fatal: unable to access 'https://github.com/ksugar/samapi.git/': Could not resolve host: github.com

It seems that it can happen when you connect to the internet using a proxy server.

If you have HTTP_PROXY environment variable, you can pass it to docker build as follows.

docker build -t samapi --build-arg HTTP_PROXY=${HTTP_PROXY} .

with the following Dockerfile:

FROM nvcr.io/nvidia/pytorch:23.07-py3

ARG HTTP_PROXY
ENV PATH="~/miniconda3/bin:$PATH"

RUN mkdir -p ~/miniconda3 &&\
 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh &&\
 bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 &&\
 rm -rf ~/miniconda3/miniconda.sh &&\
 conda update -n base -c defaults conda

# install samapi
COPY conda_envs/env_samapi.yml /tmp/env_samapi.yml
RUN conda env update -n samapi -f /tmp/env_samapi.yml \
    && conda clean --all -f -y \
    && conda env export -n "biop"

and the following condo_envs/env_samapi.yml file:

name: "samapi"
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - cudatoolkit=11.7
  - pip
  - pip:
      - git+https://github.com/ksugar/samapi.git
prefix: "/opt/conda/envs/samapi"

Please try it and let me know if it works.

Could it be because of https://github.com/ksugar/samapi/commit/ab8845eb131e96d41ed4a52ee9704aeba76135ad ?

I don't think this causes the issue because this is just a setting file for the VSCode. Anyway, it was pushed by mistake and I will remove this line in the next release.

romainGuiet commented 1 year ago

Thanks @ksugar for your swift answer !

As far as I know , no proxy set ...

What I find weird is that with my previous yml :

name: "samapi"
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.11
  - pip
  - pip:
      - git+https://github.com/ksugar/samapi.git
prefix: "/opt/conda/envs/samapi"

it seems that the git clone works (see error above) but pip complains because now they are versions problems . I thought it could be due to the modification of the versions of torch in the commit https://github.com/ksugar/samapi/commit/ab8845eb131e96d41ed4a52ee9704aeba76135ad , file pyproject.toml , because the container I start from is torch >2 ( see doc )

so I modified my yml file to target the 0.3.0 :

name: "samapi"
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.11
  - pip
  - pip:
      - https://github.com/ksugar/samapi/archive/refs/tags/v0.3.0.tar.gz
prefix: "/opt/conda/envs/samapi"

and now it builds again... so I would guess the issue is not a proxy but some changes in the 0.3.1 ?

Best,

R

ksugar commented 1 year ago

HI @romainGuiet , thank you for the update. Could you try again it with the following yml file?

name: "samapi"
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - cudatoolkit=11.7
  - pip
  - pip:
      - https://github.com/ksugar/samapi/archive/refs/tags/v0.3.1.tar.gz
prefix: "/opt/conda/envs/samapi"

It is possible that a temporal network problem caused the previous error. From my understanding, the following error is related to a network problem.

96.66 fatal: unable to access 'https://github.com/ksugar/samapi.git/': Could not resolve host: github.com
ksugar commented 1 year ago

In my environment, the following Dockerfile and condo_envs/env_samapi.yml work to build a Docker image for v0.3.1 without errors.

FROM nvcr.io/nvidia/pytorch:23.07-py3

ENV PATH="~/miniconda3/bin:$PATH"

RUN mkdir -p ~/miniconda3 &&\
 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh &&\
 bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 &&\
 rm -rf ~/miniconda3/miniconda.sh &&\
 conda update -n base -c defaults conda

# install samapi
COPY conda_envs/env_samapi.yml /tmp/env_samapi.yml
RUN conda env update -n samapi -f /tmp/env_samapi.yml \
    && conda clean --all -f -y \
    && conda env export -n "biop"
name: "samapi"
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - cudatoolkit=11.7
  - pip
  - pip:
      - git+https://github.com/ksugar/samapi.git
prefix: "/opt/conda/envs/samapi"
romainGuiet commented 1 year ago

I just tested consequently and everything works now 🤯

name: "samapi"
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - cudatoolkit=11.7
  - pip
  - pip:
      - git+https://github.com/ksugar/samapi.git
prefix: "/opt/conda/envs/samapi"

and

name: "samapi"
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - cudatoolkit=11.7
  - pip
  - pip:
      - https://github.com/ksugar/samapi/archive/refs/tags/v0.3.1.tar.gz
prefix: "/opt/conda/envs/samapi"

So I really don't know what was going wong...

Thank you again 🙏

Cheers,

R