New: Ultralytics YOLO-Human

[Laughing-q]

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Introducing new YOLOHuman model for human attribute detection! 🚀

📊 Key Changes

• Added YOLOHuman class as part of the model imports.
• Introduced a new YAML configuration for the YOLOv8 human detection model.
• Implemented additional augmentations to include human attribute data handling.
• Established a new dataset class, HumanDataset, for loading and processing human-related datasets.
• Included Human object in results to encapsulate detected human attributes.
• Enriched model __init__.py to include YOLOHuman.
• Formulated HumanPredictor, HumanTrainer, and HumanValidator under the new YOLO human module for prediction, training, and validation.

🎯 Purpose & Impact

• Enhances Model Catalog: Expands Ultralytics' model offerings to include human-specific attribute detection.
• Improves Dataset Handling: Offers streamlined process for datasets involving human features.
• Facilitates Human-centric Applications: Paves the way for more sophisticated applications such as demographic analysis, security enhancements, and personalized customer experiences.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 86.46789% with 59 lines in your changes are missing coverage. Please review.

Project coverage is 70.68%. Comparing base (bddea17) to head (dfd63b5).

Files	Patch %	Lines
ultralytics/models/yolo/human/val.py	52.87%	41 Missing :warning:
ultralytics/utils/metrics.py	69.56%	7 Missing :warning:
ultralytics/nn/tasks.py	83.33%	3 Missing :warning:
ultralytics/utils/instance.py	72.72%	3 Missing :warning:
ultralytics/engine/results.py	95.83%	2 Missing :warning:
ultralytics/data/dataset.py	97.14%	1 Missing :warning:
ultralytics/models/yolo/human/train.py	96.87%	1 Missing :warning:
ultralytics/utils/loss.py	98.46%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #12702 +/- ## ========================================== + Coverage 70.27% 70.68% +0.41% ========================================== Files 124 128 +4 Lines 15681 16052 +371 ========================================== + Hits 11020 11347 +327 - Misses 4661 4705 +44 ``` | [Flag](https://app.codecov.io/gh/ultralytics/ultralytics/pull/12702/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics) | Coverage Δ | | |---|---|---| | [Benchmarks](https://app.codecov.io/gh/ultralytics/ultralytics/pull/12702/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics) | `35.18% <26.37%> (-0.23%)` | :arrow_down: | | [GPU](https://app.codecov.io/gh/ultralytics/ultralytics/pull/12702/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics) | `36.96% <29.81%> (-0.26%)` | :arrow_down: | | [Tests](https://app.codecov.io/gh/ultralytics/ultralytics/pull/12702/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics) | `66.89% <86.46%> (+0.50%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[glenn-jocher]

Awesome!!

[spacewalk01]

It would be better if you add clothing attributes such as size, color, and fabric type. Much more practical and useful!

[glenn-jocher]

Thank you for your suggestion! Adding clothing attributes like size, color, and fabric type is indeed a valuable enhancement for the YOLOHuman model, especially for applications in retail and fashion. We appreciate your input and will consider these features for future updates. Keep the great ideas coming! 😊👍

[glenn-jocher]

@Burhan-Q @Laughing-q guys we need to add the new task to the constants defined in cfg/init: https://github.com/ultralytics/ultralytics/blob/22d034da32b3386ff58b11eb33133d1aa4940824/ultralytics/cfg/__init__.py#L34-L55

This will automatically start testing it also, which will probably raise new errors.

[glenn-jocher]

@ambitious-octopus new test failures due to missing yolov8n-human.pt for autodownload. Can you upload this model to the 8.2 assets here? This is where all our models automatically download from. https://github.com/ultralytics/assets/releases

[glenn-jocher]

@Laughing-q @ambitious-octopus guess_model_task needs to be updated here for the new human task, right now it's incapable of guessing this task correctly:

https://github.com/ultralytics/ultralytics/blob/1a1102fd169bee6aa9c38ff61d17d00f9587bec6/ultralytics/nn/tasks.py#L1004-L1077

[glenn-jocher]

@Laughing-q @ambitious-octopus I've created an untrained YOLOv8n-Human model at https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-human.pt to get our CI going. Once we have trained all 5 sizes we can replace with a trained model!

[glenn-jocher]

@ambitious-octopus test_data_utils() is failing because we need to upload a HUB-format dataset to https://github.com/ultralytics/hub/tree/main/example_datasets to join the datasets for other tasks.

[glenn-jocher]

@Laughing-q @ambitious-octopus val test is also failing because our save_txt functionality is not adapted correctly (should output txt files in the same format as the labels).

yolo val human model=weights/yolov8n-human.pt data=human8.yaml imgsz=32 save_txt

Save JSON is working correct:

yolo val human model=weights/yolov8n-human.pt data=human8.yaml imgsz=32 save_json

[ambitious-octopus]

@glenn-jocher Updated https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-human.pt with a freshly trained model.

[Laughing-q]

@glenn-jocher @ambitious-octopus Guys I removed YOLOHuman class since it's not needed while we treat human as a new task of YOLO. Also I've fixed the save_one_txt issue for human task. There's actually another update made by me, is that I figured we can directly use the save_txt method in Results instead of recreating something similar/redundant for val mode of each task. https://github.com/ultralytics/ultralytics/blob/bac868c635699c27176960e9018b3ee268b8c28f/ultralytics/engine/results.py#L332-L366 https://github.com/ultralytics/ultralytics/blob/7593b4a301061c086e6e756245eef76ea466e170/ultralytics/models/yolo/detect/val.py#L256-L263 So I updated our detect/obb/human tasks with Results.save_txt. Other tasks i.e segment/pose actually have the save_one_txt part commented so I left it there for now and let's develop these two in another PR later. https://github.com/ultralytics/ultralytics/blob/7593b4a301061c086e6e756245eef76ea466e170/ultralytics/models/yolo/segment/val.py#L157-L158

[ambitious-octopus]

@ambitious-octopus test_data_utils() is failing because we need to upload a HUB-format dataset to https://github.com/ultralytics/hub/tree/main/example_datasets to join the datasets for other tasks.

@glenn-jocher Uploaded HUB-format datataset PR.

[Laughing-q]

@glenn-jocher Meanwhile I noticed that we have a lot duplicated code in Validator.update_metrics across different tasks. https://github.com/ultralytics/ultralytics/blob/7593b4a301061c086e6e756245eef76ea466e170/ultralytics/models/yolo/segment/val.py#L100 Looking into this PR: https://github.com/ultralytics/ultralytics/pull/12645, the author had to update each val.py to add the feature because we have multiple modified versions of this method. And that's why @ambitious-octopus encountered the target_img key missing issue after the PR merged(because I had to create another modified version for human task). I actually tried to refactor this part a bit in the OBB PR so we don't have a modified version of update_metrics lying in obb/val.py. But there's still a lot duplication with other tasks. I think I'll revisit this part of code and try to eliminate the duplicated part as much as possible later some day in another PR.

[Laughing-q]

@ambitious-octopus the CI tests are failing, seems to related to new updated yolov8n-human.pt

And that's because the model was trained withtask=detect but now since we're updating the logic here to treat human as a new task so the tests are failing with training from the new yolov8n-human.pt

[Laughing-q]

@ambitious-octopus let's reset the weight back, and later today I'll launch new training on our server with task=human to get all sizes of model. :)

[Laughing-q]

@glenn-jocher @ambitious-octopus ok I've re-uploaded the weight and now everything works properly in tests except the hub dataset, which I guess it'll be good when the PR that @ambitious-octopus opened merged. :)

And now there's several gpus freed on our server, I'll launch several training right now.

[ambitious-octopus]

docs image

[glenn-jocher]

@glenn-jocher @ambitious-octopus Guys I removed YOLOHuman class since it's not needed while we treat human as a new task of YOLO. Also I've fixed the save_one_txt issue for human task. There's actually another update made by me, is that I figured we can directly use the save_txt method in Results instead of recreating something similar/redundant for val mode of each task.

https://github.com/ultralytics/ultralytics/blob/bac868c635699c27176960e9018b3ee268b8c28f/ultralytics/engine/results.py#L332-L366

https://github.com/ultralytics/ultralytics/blob/7593b4a301061c086e6e756245eef76ea466e170/ultralytics/models/yolo/detect/val.py#L256-L263

So I updated our detect/obb/human tasks with Results.save_txt. Other tasks i.e segment/pose actually have the save_one_txt part commented so I left it there for now and let's develop these two in another PR later. https://github.com/ultralytics/ultralytics/blob/7593b4a301061c086e6e756245eef76ea466e170/ultralytics/models/yolo/segment/val.py#L157-L158

@Laughing-q really nice! I was thinking the same thing, I think some of the val methods are copied just because one line needs to be changed from the base class, its unfortunate.

We want to reduce copies/duplicates of classes as much as possible. When I was thinking about this some more YOLO-Human should really be a more flexible class that is essentially object-detection + features, with a parameterized way to define those features in the labels/loss/plots, but for now our hard-coded implementation is a good start.

[glenn-jocher]

@Laughing-q also yes I changed the task from detect to human in places because otherwise it was not guessing the task correctly for short commands like yolo predict model=yolov8n-human.pt

EDIT: Would be nice to be able to read the task directly from the model instead of guessing, maybe directly from the checkpoint dictionary or the torch model.

[glenn-jocher]

@Laughing-q maybe the regressions parameters are too many for the new attributes? We have significant overfitting on YOLO8x-human for attribute val losses, and I noticed the size difference is significant for YOLOv8n vs 8n-human:

• YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs
• YOLOv8n-human summary (fused): 225 layers, 4008539 parameters, 0 gradients, 12.0 GFLOPs

[Laughing-q]

@glenn-jocher yes I think you are right! In fact I've already made this part lighter once. Perhaps we can use group convs i.e set 3 groups for each regression part(weight/height/age). BTW I was intentionally using regular convs for them cuz I think there's connection between these three attributes in real life(at least height and weight are connected), and figured not spliting them as groups might get better results.

I also noticed the overfitting in my past experiments but It's hard to tell if the issue is from lack of dataset or the model side. As we were keeping generating more data so I didn't modify the head again yet.

Meanwhile I just realized that I didn't set stronger augmentation for large models, we could retrain L/X with stronger augmentation.

@glenn-jocher Sorry I'm not with my computer right now and won't be able to log in this weekend. Please feel free to modify the head part and retrain our large models with stronger augmentation if you feel necessary.:) FYI I trained all the models with its corresponding detection weight as pretrained weight for 100 epochs, and batch=128, other settings remained default. Training with pretrained weight got better results than from scratch in my experiments so I kept it.

[glenn-jocher]

@Laughing-q ah ok got it! Yes maybe I'll work on some updates. We have our new L40S server too this week as part of our NVIDIA launchpad trial so I can try training there.

[Laughing-q]

@glenn-jocher oh right before any updates please make sure any modifications on model structure won't break our current deployment pipeline on iOS side, or it might need extra time to develop iOS side. (Reminding as the deadline is close. :))