search

HumanSignal / label-studio

Label Studio is a multi-type data labeling and annotation tool with standardized output format
https://labelstud.io
Apache License 2.0
18.32k stars 2.3k forks source link

connect ML model, but show error "doesn't look like a valid ML backend" #818

Closed joul55320 closed 2 years ago

joul55320 commented 3 years ago

Describe the bug

I follow this guide https://labelstud.io/tutorials/object-detector.html let ML modelconnected to LS, but LS show "doesn't look like a valid ML backend"

To Reproduce Steps to reproduce the behavior:

  1. label-studio-ml init coco --from mmdetection.py
  2. python coco/_wsgi.py --with config_file=/path/to/mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_iou_1x_coco.py checkpoint_file=/path/to/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth device=gpu
  3. label-studio
  4. Setting -> Machine Learning -> Add model
  5. type "http://0.0.0.0:9090" in "URL"
  6. click "validate and save"

about step2, because when I type "label-studio-ml start [project]", then show ImportError: No module named flask so I run the python file that in project folder

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem. Screenshot from 2021-04-16 17-42-32

Environment (please complete the following information):

Additional context Add any other context about the problem here.

niklub commented 3 years ago

Hi! guess there are multiple problems I'd recommend checking:

  1. Before running ML backend, its environment should be installed first (that's why probably you've faced ImportError: No module named flask)
  2. Before connecting the model by adding ML backend URL, you have to specify the labeling configuration in your project settings (should be Browse Templates -> Object Detection with Bounding Boxes in your case)
  3. Also it's better to use localhost instead of 0.0.0.0 (but this problem might be more related to Windows setup)
joul55320 commented 3 years ago

Hi niklub:

Thanks, I think this problam is make by "flask".

Now, I can connect coco model to LS and open the button "Display ML-predicted annotations when labeling", but when I label datas, it always can not show any predictions.

My labeling configuration setting was follow guide: https://labelstud.io/tutorials/object-detector.html

The score threshold of model is 0.3

And my configuration code about object is:

christianjcc commented 2 years ago

Hi all,

I am experiencing the same error message above but have not been able to resolve given the information provided by others in this thread. My aim is to try to execute active learning with Label-studio (ideally with BaaL) on a simple example of simple images with vehicles to perform bound-box object detection using a pre-trained model (e.g. Faster-RCNN-based or YOLvX) trained on MS COCO dataset. Any clue that you be able to spot as to what I might be either incorrectly executing on missing would be greatly appreciated. Below I include as much information as to how I went about installing and executing the commands to provide context

Working with the following system specs:

Libraries were installed by taking the following steps ` // Create Conda Environment

conda create --name label-studio
conda activate label-studio

// Install label studio and its dependencies

conda install pytorch cudatoolkit=10.1 torchvision -c pytorch
pip install label-studio

// Install Open-MMLab MMDetection and its dependencies

pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu101/torch1.7.0/index.html
pip install mmdet==2.23.0

// Install Label-studio ML backend and its dependencies

git clone https://github.com/heartexlabs/label-studio-ml-backend
cd label-studio-ml-backend

// Install label-studio-ml and its dependencies pip install -U -e .

// Install example dependencies pip install -r label_studio_ml/examples/requirements.txt

// Export your active environment to a new file conda env export > environment.yml

// Create the environment from the environment.yml file conda env create -f environment.yml

// Install Bayesian active learning with Label-studio pip install baal==1.3.0

// Create and initialize the directory ./coco-detector label-studio-ml init coco-detector --from label_studio_ml/examples/mmdetection/mmdetection.py --force

// also tried (not sure when to use --from vs --script), label-studio-ml init coco-detector --script label-studio-ml-backend/label_studio_ml/examples/mmdetection/mmdetection.py --force

Checkpoint downloaded from: https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth

Config file https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn

// Start the ML backend server label-studio-ml start coco-detector --with config_file=label_studio_ml/examples/mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py checkpoint_file=label_studio_ml/examples/mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth device=gpu:0 score_threshold=0.5

// tried also, label-studio start coco-detector

// Start Label Studio label-studio start --ml-backends http://localhost:8080 `

Inserted and enabled smart to true, under Label Studio Labeling Inference tab, `

`

Added model URL: http://example.com:9090 (also tried http://0.0.0.0:9090/)

Conda environment installed libraries ` name: label-studio channels:

makseq commented 2 years ago
  1. Try this ML backend using docker-compose up - https://github.com/heartexlabs/label-studio-ml-backend/tree/master/label_studio_ml/examples/mmdetection
  2. Create a new project in LS, then add this ML backend via GUI (port 9090).
  3. Use this labeling config: 
<View>
  <Image name="image" value="$image"/>
  <RectangleLabels name="label" toName="image">
    <Label value="airplane" background="green"/>
    <Label value="car" background="blue"/>
  </RectangleLabels>
</View>

Note: mmdetection supports only rectangle labels.

The full list with labels are available here: https://github.com/heartexlabs/label-studio/blob/master/docs/source/tutorials/object-detector.md#the-full-list-of-coco-labels

christianjcc commented 2 years ago

@makseq, thank you for your suggestion. I executed the following,

Step 1. cd label-studio-ml-backend/label_studio_ml/examples/mmdetection/ docker-compose up -d after executing the last few lines gave,

Successfully built e75e22b88f4f Successfully tagged mmdetection_server:latest WARNING: Image for service server was built because it did not already exist. To rebuild this image you must usedocker-compose buildordocker-compose up --build. Creating redis ... done Creating server ... done Appeared fine. After following steps 2-3 you provided.

label-studio start coco-detector --init --ml-backends http://localhost:8080 --sampling=prediction-score-min Provided the list-of-coco-labels to the LS Labeling Interface tab, but ended up getting an error message not being able to connect to the ML backend.

check docker-compose ran,

docker-compose ps

// Name Command State Ports
// ------------------------------------------------------------------------------------------ // redis docker-entrypoint.sh redis ... Up 6379/tcp
// server /usr/local/bin/supervisord ... Up 0.0.0.0:9090->9090/tcp,:::9090->9090/tcp

Screenshot from 2022-04-06 20-47-56 Screenshot from 2022-04-06 20-47-28

KonstantinKorotaev commented 2 years ago

@christianjcc could you please check container server logs?

christianjcc commented 2 years ago

@KonstantinKorotaev, thank you for your suggestion. The solution was to export as environmental variables the paths pointing to the config_file and checkpoint_file (tried also hard coding these values in the mmdetection.py file) and then it worked. It appears the command line arguments are not being propagated to the python script correctly. When running,

label-studio-ml start coco-detector --with config_file=label_studio_ml/examples/mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py checkpoint_file=label_studio_ml/examples/mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth device=gpu:0 score_threshold=0.5