DuncanZauss
commented
2 years ago H, Thank you for your interest in Keypoint Communities! The shown video was processed with two separate models, one for the car keypoints and the other for the human keypoints. I don't have a model trained on cars and human keypoints. However, this project is based on Openpifpaf and it should be fairly straight forward to train a model with two heads/decoders. You could for instance use this command to train a model:
python3 -m openpifpaf.train --lr=0.0001 --momentum=0.95 --b-scale=3.0 --epochs=150 --lr-decay 130 140 --lr-decay-epochs=10 --batch-size=16 --weight-decay=1e-5 --dataset=wholebody-apollo --dataset-weights 0.5 0.5 --wholebody-upsample=2 --basenet=shufflenetv2k16 --loader-workers=16 --wholebody-train-annotations=<PathToYourMSCOCO>/data-mscoco/annotations/person_keypoints_train2017_wholebody_pifpaf_style.json --wholebody-val-annotations=<PathToYourMSCOCO>/data-mscoco/annotations/person_keypoints_val2017_wholebody_pifpaf_style.json --wholebody-train-image-dir=<COCO_train_image_dir> --wholebody-val-image-dir=<COCO_val_image_dir> --apollo-upsample 2 --apollo-bmin 2 --apollo-square-edge=769This command trains a single model with two different heads. At inference, the two decoders from both heads will be used and you should get a prediction for both human and keypoints. You might need to tweak some hyperparameters at training, i.e. --dataset-weights describes the weight between the two datasets/heads. You can also find more info here: https://openpifpaf.github.io/plugins_wholebody.html and here: https://openpifpaf.github.io/plugins_apollocar3d.html Let me know if you try the joint training. I would be curious to hear how it goes :)
HI, thanks for sharing! Can you please elaborate more, in your second demo video we can see you detected human and vehicle key points in each frame? Was it two different models of a single model? If it is a single model please let me know where to get it. Thank you in advance.