Closed GabrielGosden closed 1 month ago
To further debug the issue I've now followed the same installation procedure on another machine. I have the same issues on this machine.
We are also running behind a corporate proxy. So to be able to install CVAT and deploy the functions i have modified the Dockerfile
and function.yaml
to use our company proxy when using apt-get
, pip
and curl
The function.yaml file now looks like this:
# Copyright (C) 2023-2024 CVAT.ai Corporation
#
# SPDX-License-Identifier: MIT
metadata:
name: pth-facebookresearch-sam-vit-h
namespace: cvat
annotations:
name: Segment Anything
version: 2
type: interactor
spec:
min_pos_points: 1
min_neg_points: 0
animated_gif: https://raw.githubusercontent.com/cvat-ai/cvat/develop/site/content/en/images/hrnet_example.gif
help_message: The interactor allows to get a mask of an object using at least one positive, and any negative points inside it
spec:
description: Interactive object segmentation with Segment-Anything
runtime: 'python:3.8'
handler: main:handler
eventTimeout: 30s
env:
- name: PYTHONPATH
value: /opt/nuclio/sam
build:
image: cvat.pth.facebookresearch.sam.vit_h
baseImage: ubuntu:22.04
directives:
preCopy:
# disable interactive frontend
- kind: ENV
value: DEBIAN_FRONTEND=noninteractive
# set workdir
- kind: WORKDIR
value: /opt/nuclio/sam
# Setup proxy
- kind: ENV
value: http_proxy="http://<username>:<password>@<ip>:<port>"
- kind: ENV
value: https_proxy="http://<username>:<password>@<ip>:<port>"
- kind: ENV
value: HTTP_PROXY="http://<username>:<password>@<ip>:<port>"
- kind: ENV
value: HTTPS_PROXY="http://<username>:<password>@<ip>:<port>"
# install basic deps
- kind: RUN
value: apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6
# install sam deps
- kind: RUN
value: pip3 install --trusted-host pypi.org --trusted-host files.pythonhosted.org torch torchvision torchaudio pycocotools matplotlib onnxruntime onnx
# install sam code
- kind: RUN
value: pip3 install --trusted-host pypi.org --trusted-host files.pythonhosted.org git+https://github.com/facebookresearch/segment-anything.git
# download sam weights
- kind: RUN
value: curl --insecure -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
# map pip3 and python3 to pip and python
- kind: RUN
value: ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python
triggers:
myHttpTrigger:
maxWorkers: 2
kind: 'http'
workerAvailabilityTimeoutMilliseconds: 10000
attributes:
maxRequestBodySize: 33554432 # 32MB
platform:
attributes:
restartPolicy:
name: always
maximumRetryCount: 3
mountMode: volume
When trying to use SAM from UI the following error from network analyzer is shown:
But when using http://localhost:8080/api/lambda/functions/pth-facebookresearch-sam-vit-h?org=
It works without any issues
It works without any issues
These requests are different. The first one is POST, the second one is GET.
POST is trying to reach container/host with nuclio function and raises 503 because it is unavailable. Perhaps there are some issues with infrastructure, but without knowing details I may only suggest that it happens because of set HTTP_PROXY in CVAT container. Thus, instead of host with nuclio function, it tries to find the function on your proxy server and fails.
You may try to setup NO_PROXY properly, adding localhost
and host.docker.internal
to the variable.
Thanks @bsekachev for the good idea. After a bit of experimenting i was able to fix the issue by adding the NO_PROXY
you suggested to the serverless/openvino/base/Dockerfile
. This is the base image for openvino on which nuclio deploys their functions.
Actions before raising this issue
Steps to Reproduce
Follow the Serverless tutorial
git clone https://github.com/cvat-ai/cvat
docker compose -f docker-compose.yml -f docker-compose.dev.yml -f components/serverless/docker-compose.serverless.yml up -d --build
Follow the Semi-automatic and Automatic Annotation tutorial
wget https://github.com/nuclio/nuclio/releases/download/1.13.0/nuctl-1.13.0-linux-amd64
which is the same version as in docker-compose.serverless.ymlsudo chmod +x nuctl-1.13.0-linux-amd64
sudo ln -sf $(pwd)/nuctl-1.13.0-linux-amd64 /usr/local/bin/nuctl
./serverless/deploy_cpu.sh serverless/pytorch/facebookresearch/sam/
Expected Behavior
After installing CVAT and deploying function i would expect to be able to use it for automatic segmentation within CVAT. When try to use the tool i get the following error:
503 Server Error: Service Unavailable for url: http://host.docker.internal:32770/
The output of
nuctl get function
looks good:From http://localhost:8070/projects/cvat/functions the function is deployed with zero errors and is running:![image](https://github.com/user-attachments/assets/720c1bf1-93bc-4b2b-9ab4-a93f6815d8ba)
The
docker logs nuclio-nuclio-pth-facebookresearch-sam-vit-h
also has zero errors:Output of
docker compose -f docker-compose.yml -f docker-compose.dev.yml -f components/serverless/docker-compose.serverless.yml ps
Possible Solution
I have found the following possible related issues. I have tried to implement the suggestions from there without success:
https://github.com/cvat-ai/cvat/issues/6582 https://github.com/cvat-ai/cvat/issues/4904 https://github.com/cvat-ai/cvat/issues/5205
Context
No response
Environment