search

HumanSignal / label-studio-ml-backend

Configs and boilerplates for Label Studio's Machine Learning backend
Apache License 2.0
587 stars 261 forks source link

Issue running SAM 2 Video tracking #645

Open vs-nth opened 1 month ago

vs-nth commented 1 month ago

Hello, I'm trying to run Sam2 with docker compose on my local system for video tracking. However, the upstream repo for Segment anything 2 updated the checkpoints to point to Sam2.1 and as a result the default configuration does not work. Here's the contents of my docker-compose.yml file. 

version: "3.8"
services:
  label_studio:
    image: heartexlabs/label-studio:latest
    container_name: label_studio
    ports:
      - 8508:8080
    volumes:
      - ./mydata:/label-studio/data:rw

  segment_anything_2_video:
    container_name: segment_anything_2_video
    image: humansignal/segment_anything_2_video:v0
    build:
      context: .
      args:
        TEST_ENV: ${TEST_ENV}
    environment:
      # specify these parameters if you want to use basic auth for the model server
      - BASIC_AUTH_USER=
      - BASIC_AUTH_PASS=
      # set the log level for the model server
      - LOG_LEVEL=DEBUG
      # any other parameters that you want to pass to the model server
      - ANY=PARAMETER
      # specify the number of workers and threads for the model server
      - WORKERS=1
      - THREADS=8
      # specify the model directory (likely you don't need to change this)
      - MODEL_DIR=/data/models
      # specify device
      - DEVICE=cuda  # or 'cpu' (coming soon)
      # SAM2 model config
      - MODEL_CONFIG=sam2.1_hiera_l.yaml
      # SAM2 checkpoint
      - MODEL_CHECKPOINT=sam2.1_hiera_large.pt

      # Specify the Label Studio URL and API key to access
      # uploaded, local storage and cloud storage files.
      # Do not use 'localhost' as it does not work within Docker containers.
      # Use prefix 'http://' or 'https://' for the URL always.
      # Determine the actual IP using 'ifconfig' (Linux/Mac) or 'ipconfig' (Windows).
      - LABEL_STUDIO_URL=http://label_studio:8080
      - LABEL_STUDIO_API_KEY=
      - CUDA_VISIBLE_DEVICES=0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    ports:
      - "9090:9090"
    volumes:
      - "./data:/data:rw"

volumes:
  mydata:
  data:

I set the checkpoint and the config to Sam2.1 and on running docker compose up, i encounter this error: 

segment_anything_2_video  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
segment_anything_2_video  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
segment_anything_2_video  |   File "/app/_wsgi.py", line 33, in <module>
segment_anything_2_video  |     from model import NewModel
segment_anything_2_video  |   File "/app/model.py", line 39, in <module>
segment_anything_2_video  |     predictor = build_sam2_video_predictor(MODEL_CONFIG, sam2_checkpoint)
segment_anything_2_video  |   File "/segment-anything-2/sam2/build_sam.py", line 127, in build_sam2_video_predictor
segment_anything_2_video  |     cfg = compose(config_name=config_file, overrides=hydra_overrides)
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/hydra/compose.py", line 32, in compose
segment_anything_2_video  |     assert (
segment_anything_2_video  | AssertionError: GlobalHydra is not initialized, use @hydra.main() or call one of the hydra initialization methods first
segment_anything_2_video  | [2024-10-06 21:12:17 +0000] [7] [INFO] Worker exiting (pid: 7)

Anything obvious that i'm doing wrong?

Thank you!

FunnyPocketBook commented 1 month ago

Hey! It looks like with the update to SAM2.1 some things in the back have changed. If you just want to use SAM2 and not necessarily SAM 2.1, then you can change the following in the Dockerfile in the segment_anything_2 directory:

-RUN cd / && git clone --depth 1 --branch main --single-branch https://github.com/facebookresearch/segment-anything-2.git
+RUN cd / && git clone --branch main --single-branch https://github.com/facebookresearch/segment-anything-2.git && cd segment-anything-2 && git checkout 0f6515ae853c40420ea8e3dd250f8031bbf03023

This essentially just checks out the repo to the commit before SAM2.1 was added. Then changing the env variables MODEL_CHECKPOINT and MODEL_CONFIG back to sam2 and rebuilding the image worked for me!

timmermansjoy commented 1 month ago

Hey there i have a similar bit different issue where I do use the setup as you suggested however, I get a MissingConfigException when trying to run it, I can find the file in the repo tho. https://github.com/facebookresearch/sam2/blob/main/sam2/configs/sam2.1/sam2.1_hiera_l.yaml Screenshot 2024-10-18 at 11 03 31 AM

FunnyPocketBook commented 1 month ago

Hi @timmermansjoy

If you did what I suggested, you also need to change the config to sam2 instead sam2.1

timmermansjoy commented 1 month ago

hi @FunnyPocketBook Thanks, however I still see something strange happening. I have the model up and running in label studio.

Screenshot 2024-10-21 at 10 27 57 AM

But now when I try to annotate something in the video I get the following:

label_studio              | [2024-10-21 08:30:02,678] [django.server::log_message::161] [INFO] "GET /api/label_links?project=2&expand=label HTTP/1.1" 200 52
label_studio              | [2024-10-21 08:30:02,678] [django.server::log_message::161] [INFO] "GET /api/label_links?project=2&expand=label HTTP/1.1" 200 52
label_studio              | [2024-10-21 08:30:02,699] [django.server::log_message::161] [INFO] "GET /api/tasks/1?project=2 HTTP/1.1" 200 1991
label_studio              | [2024-10-21 08:30:02,699] [django.server::log_message::161] [INFO] "GET /api/tasks/1?project=2 HTTP/1.1" 200 1991
label_studio              | [2024-10-21 08:30:02,800] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 42458
label_studio              | [2024-10-21 08:30:02,800] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 42458
label_studio              | [2024-10-21 08:30:03,036] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 30516698
label_studio              | [2024-10-21 08:30:03,036] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 30516698
label_studio              | [2024-10-21 08:30:03,510] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 30451162
label_studio              | [2024-10-21 08:30:03,510] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 30451162
label_studio              | [2024-10-21 08:30:04,835] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 19080666
label_studio              | [2024-10-21 08:30:04,835] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 19080666
label_studio              | [2024-10-21 08:30:05,495] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 23143898
label_studio              | [2024-10-21 08:30:05,495] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 23143898
label_studio              | [2024-10-21 08:30:06,827] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 23504346
label_studio              | [2024-10-21 08:30:06,827] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 23504346
label_studio              | [2024-10-21 08:30:07,223] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 19015130
label_studio              | [2024-10-21 08:30:07,223] [django.server::log_message::161] [INFO] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 206 19015130
segment_anything_2_video  | [2024-10-21 08:30:19,284] [DEBUG] [label_studio_ml.api::log_request_info::185] Request headers: Host: segment_anything_2_video:9090
segment_anything_2_video  | Sentry-Trace: aa9d328d696641a0b8e32d83b77fd6af-afc468bb03f8aa38-0
segment_anything_2_video  | Baggage: sentry-trace_id=aa9d328d696641a0b8e32d83b77fd6af,sentry-environment=opensource,sentry-release=label-studio%401.13.1,sentry-public_key=68b045ab408a4d32a910d339be8591a4,sentry-transaction=/api/ml/%7Bpk%7D/interactive-annotating,sentry-sample_rate=0.02,sentry-sampled=false
segment_anything_2_video  | User-Agent: heartex/1.13.1+0.gd9b816a
segment_anything_2_video  | Accept-Encoding: gzip, deflate
segment_anything_2_video  | Accept: */*
segment_anything_2_video  | Connection: keep-alive
segment_anything_2_video  | Content-Length: 2311
segment_anything_2_video  | Content-Type: application/json
segment_anything_2_video  | Authorization: Basic am95QHNlY3VyeS0zNjAuY29tOkVkY3JmdjEwMTI=
segment_anything_2_video  | 
segment_anything_2_video  | 
segment_anything_2_video  | [2024-10-21 08:30:19,284] [DEBUG] [label_studio_ml.api::log_request_info::186] Request body: b'{"tasks": [{"id": 1, "annotations": [{"id": 1, "completed_by": 1, "result": [{"value": {"framesCount": 1489, "duration": 59.5224, "sequence": [{"frame": 336, "enabled": true, "rotation": 0, "x": 2.7804410354745928, "y": 19.31927133269415, "width": 17.16203259827421, "height": 12.613188452114626, "time": 13.44}], "labels": ["Player"]}, "id": "92KZTb8u-o", "from_name": "box", "to_name": "video", "type": "videorectangle", "origin": "manual"}], "was_cancelled": false, "ground_truth": false, "created_at": "2024-10-21T08:07:23.125233Z", "updated_at": "2024-10-21T08:26:18.534020Z", "draft_created_at": "2024-10-21T08:07:22.758230Z", "lead_time": 415.37699999999995, "prediction": {}, "result_count": 0, "unique_id": "e0106f61-e15b-4eda-b873-57195aae150d", "import_id": null, "last_action": null, "task": 1, "project": 2, "updated_by": 1, "parent_prediction": null, "parent_annotation": null, "last_created_by": null}], "file_upload": "bc1bdf8c-202410111301_1.mp4", "drafts": [], "predictions": [], "data": {"video": "/data/upload/2/bc1bdf8c-202410111301_1.mp4"}, "meta": {}, "created_at": "2024-10-21T07:52:32.111880Z", "updated_at": "2024-10-21T08:26:18.664429Z", "inner_id": 1, "total_annotations": 1, "cancelled_annotations": 0, "total_predictions": 0, "comment_count": 0, "unresolved_comment_count": 0, "last_comment_updated_at": null, "project": 2, "updated_by": 1, "comment_authors": []}], "project": "2.1729497082", "label_config": "<View>\\n    <Labels name=\\"videoLabels\\" toName=\\"video\\" allowEmpty=\\"true\\">\\n        <Label value=\\"Player\\" background=\\"#11A39E\\"/>\\n        <Label value=\\"Ball\\" background=\\"#D4380D\\"/>\\n    </Labels>\\n\\n    <!-- Please specify FPS carefully, it will be used for all project videos -->\\n    <Video name=\\"video\\" value=\\"$video\\" framerate=\\"25.0\\"/>\\n    <VideoRectangle name=\\"box\\" toName=\\"video\\" smart=\\"true\\"/>\\n</View>", "params": {"login": null, "password": null, "context": {"result": [{"value": {"framesCount": 1489, "duration": 59.5224, "sequence": [{"frame": 336, "enabled": true, "rotation": 0, "x": 2.7804410354745928, "y": 19.31927133269415, "width": 17.16203259827421, "height": 12.613188452114626, "time": 13.44}], "labels": ["Ball"]}, "id": "92KZTb8u-o", "from_name": "box", "to_name": "video", "type": "videorectangle", "origin": "manual"}]}}}'
segment_anything_2_video  | [2024-10-21 08:30:19,285] [DEBUG] [label_studio_sdk._extensions.label_studio_tools.core.utils.io::get_local_path::117] Image and upload dirs: image_dir=/root/.local/share/label-studio/media/upload, upload_dir=/root/.local/share/label-studio/media/upload
segment_anything_2_video  | [2024-10-21 08:30:19,285] [INFO] [label_studio_sdk._extensions.label_studio_tools.core.utils.io::get_local_path::144] Resolving url using hostname [http://label_studio:8080]: http://label_studio:8080/data/upload/2/bc1bdf8c-202410111301_1.mp4
segment_anything_2_video  | [2024-10-21 08:30:19,285] [INFO] [label_studio_sdk._extensions.label_studio_tools.core.utils.io::download_and_cache::205] Download http://label_studio:8080/data/upload/2/bc1bdf8c-202410111301_1.mp4 to /root/.cache/label-studio/e51bccd7__bc1bdf8c-202410111301_1.mp4
segment_anything_2_video  | [2024-10-21 08:30:19,285] [DEBUG] [label_studio_sdk._extensions.label_studio_tools.core.utils.io::download_and_cache::218] Authorization token is used for download_and_cache
segment_anything_2_video  | [2024-10-21 08:30:19,286] [DEBUG] [urllib3.connectionpool::_new_conn::243] Starting new HTTP connection (1): label_studio:8080
label_studio              | [2024-10-21 08:30:19,288] [django.security.DisallowedHost::response_for_exception::99] [ERROR] Invalid HTTP_HOST header: 'label_studio:8080'. The domain name provided is not valid according to RFC 1034/1035.
label_studio              | [2024-10-21 08:30:19,288] [django.security.DisallowedHost::response_for_exception::99] [ERROR] Invalid HTTP_HOST header: 'label_studio:8080'. The domain name provided is not valid according to RFC 1034/1035.
label_studio              | [2024-10-21 08:30:19,288] [django.request::log_response::224] [WARNING] Bad Request: /data/upload/2/bc1bdf8c-202410111301_1.mp4
label_studio              | [2024-10-21 08:30:19,288] [django.request::log_response::224] [WARNING] Bad Request: /data/upload/2/bc1bdf8c-202410111301_1.mp4
label_studio              | [2024-10-21 08:30:19,289] [django.server::log_message::161] [WARNING] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 400 143
label_studio              | [2024-10-21 08:30:19,289] [django.server::log_message::161] [WARNING] "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/1.1" 400 143
segment_anything_2_video  | [2024-10-21 08:30:19,289] [DEBUG] [urllib3.connectionpool::_make_request::546] http://label_studio:8080 "GET /data/upload/2/bc1bdf8c-202410111301_1.mp4 HTTP/11" 400 None
segment_anything_2_video  | [2024-10-21 08:30:19,289] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
segment_anything_2_video  |     return f(*args, **kwargs)
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/label_studio_ml/api.py", line 69, in _predict
segment_anything_2_video  |     response = model.predict(tasks, context=context, **params)
segment_anything_2_video  |   File "/app/model.py", line 227, in predict
segment_anything_2_video  |     video_path = get_local_path(video_url, task_id=task_id)
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/label_studio_sdk/_extensions/label_studio_tools/core/utils/io.py", line 166, in get_local_path
segment_anything_2_video  |     filepath = download_and_cache(
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/label_studio_sdk/_extensions/label_studio_tools/core/utils/io.py", line 221, in download_and_cache
segment_anything_2_video  |     r.raise_for_status()
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
segment_anything_2_video  |     raise HTTPError(http_error_msg, response=self)
segment_anything_2_video  | requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://label_studio:8080/data/upload/2/bc1bdf8c-202410111301_1.mp4
segment_anything_2_video  | 
segment_anything_2_video  | Traceback (most recent call last):
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
segment_anything_2_video  |     return f(*args, **kwargs)
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/label_studio_ml/api.py", line 69, in _predict
segment_anything_2_video  |     response = model.predict(tasks, context=context, **params)
segment_anything_2_video  |   File "/app/model.py", line 227, in predict
segment_anything_2_video  |     video_path = get_local_path(video_url, task_id=task_id)
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/label_studio_sdk/_extensions/label_studio_tools/core/utils/io.py", line 166, in get_local_path
segment_anything_2_video  |     filepath = download_and_cache(
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/label_studio_sdk/_extensions/label_studio_tools/core/utils/io.py", line 221, in download_and_cache
segment_anything_2_video  |     r.raise_for_status()
segment_anything_2_video  |   File "/opt/conda/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
segment_anything_2_video  |     raise HTTPError(http_error_msg, response=self)
segment_anything_2_video  | requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://label_studio:8080/data/upload/2/bc1bdf8c-202410111301_1.mp4
segment_anything_2_video  | 
segment_anything_2_video  | [2024-10-21 08:30:19,290] [DEBUG] [label_studio_ml.api::log_response_info::191] Response status: 500 INTERNAL SERVER ERROR
segment_anything_2_video  | [2024-10-21 08:30:19,290] [DEBUG] [label_studio_ml.api::log_response_info::192] Response headers: Content-Type: application/json
segment_anything_2_video  | Content-Length: 1277
segment_anything_2_video  | 
segment_anything_2_video  | 
segment_anything_2_video  | [2024-10-21 08:30:19,290] [DEBUG] [label_studio_ml.api::log_response_info::193] Response body: b'{"detail":"HTTPError: 400 Client Error: Bad Request for url: http://label_studio:8080/data/upload/2/bc1bdf8c-202410111301_1.mp4","request":{},"result":{"traceback":"Traceback (most recent call last):\\n  File \\"/opt/conda/lib/python3.10/site-packages/label_studio_ml/exceptions.py\\", line 39, in exception_f\\n    return f(*args, **kwargs)\\n  File \\"/opt/conda/lib/python3.10/site-packages/label_studio_ml/api.py\\", line 69, in _predict\\n    response = model.predict(tasks, context=context, **params)\\n  File \\"/app/model.py\\", line 227, in predict\\n    video_path = get_local_path(video_url, task_id=task_id)\\n  File \\"/opt/conda/lib/python3.10/site-packages/label_studio_sdk/_extensions/label_studio_tools/core/utils/io.py\\", line 166, in get_local_path\\n    filepath = download_and_cache(\\n  File \\"/opt/conda/lib/python3.10/site-packages/label_studio_sdk/_extensions/label_studio_tools/core/utils/io.py\\", line 221, in download_and_cache\\n    r.raise_for_status()\\n  File \\"/opt/conda/lib/python3.10/site-packages/requests/models.py\\", line 1024, in raise_for_status\\n    raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://label_studio:8080/data/upload/2/bc1bdf8c-202410111301_1.mp4\\n"},"status":500}\n'

The video looks normal, my docker-compose looks like

services:
  label_studio:
    image: heartexlabs/label-studio:latest
    container_name: label_studio
    ports:
      - 8508:8080
    volumes:
      - ./mydata:/label-studio/data:rw

  segment_anything_2_video:
    container_name: segment_anything_2_video
    image: humansignal/segment_anything_2_video:v0
    build:
      context: .
      args:
        TEST_ENV: ${TEST_ENV}
    environment:
      - BASIC_AUTH_USER=joy@secury-360.com
      - BASIC_AUTH_PASS=ABC123
      - LOG_LEVEL=DEBUG
      - ANY=PARAMETER
      - WORKERS=1
      - THREADS=8
      - MODEL_DIR=/data/models
      - DEVICE=cuda
      - MODEL_CONFIG=sam2_hiera_l.yaml
      - MODEL_CHECKPOINT=sam2_hiera_large.pt
      - LABEL_STUDIO_URL=http://label_studio:8080
      - LABEL_STUDIO_API_KEY=6153c0cb926eb3e348a23288f8495f3993aba5c7
      - CUDA_VISIBLE_DEVICES=0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    ports:
      - "9090:9090"
    volumes:
      - "./data:/data:rw"

volumes:
  mydata:
  data:

I already tried adding 

environment:
   - ALLOWED_HOSTS=label_studio,localhost,127.0.0.1

But that also did not help anything. Can you give any pointers?