Closed amalnanavati closed 5 months ago
TODOs:
Mostly done. When playing back the above ROSbag, the predictions were quite unstable in the in-front-of-face configuration. I tried retraining with mean, but it didn't fix that. Worth looking into this more. But first test it in-person / play back the other ROSbags to not overfit to one case. Also because as of now, we won't be checking food-on-fork in the in-front-of-face configuration.
Did the in-person tests. It works...except that it is very sensitive to the fork being slightly rotated in the gripper. The thing is, this is a problem we have had before that causes the fork to not skewer items centered. So while there are some FoF changes I can do to make it less sensitive (e.g., only focus on deviation in the z direction, or maybe weight deviations in each direction differently), I think we should first try to come up with a proper fix to ensuring the forktip is actually aligned with the frame.
Description
In service of #43 , this PR implements automatic food-on-fork detection.
The core approach is to store a representative set of 3D points that correspond to the fork shape when there is no food on it. Then, given a new depth image, compute the average distance between the points in that image and the nearest point on the stored image. When there is no food-on-fork, that distances are on the order of
1e-4
m, whereas when there is food on the fork, they are on the order of1e-3
m. Train a classifier on those distances to predict the probability that there is food on the fork.As a concrete example, the specific model pushed in this PR stores 2426 points which look like the below image:
This approach achieves ~96% accuracy on bagged data of 8 real bites (acquisition and transfer) when we use an upper and lower threshold of 0.5. Inference takes a median of 0.016s (25th percentile 0.007s, 75th percentile 0.022s).
Testing procedure
~/colcon_ws/src/ada_feeding/ada_feeding_perception/data/food_on_fork/
.cd ~/colcon_ws/src/ada_feeding/ada_feeding_perception/ada_feeding_perception
python3 food_on_fork_train_test.py --model-classes '{"distance_no_fof_detector_with_filters": "ada_feeding_perception.food_on_fork_detectors.FoodOnForkDistanceToNoFOFDetector"}' --model-kwargs '{"distance_no_fof_detector_with_filters": {"camera_matrix": [614.5933227539062, 0.0, 312.1358947753906, 0.0, 614.6914672851562, 223.70831298828125, 0.0, 0.0, 1.0], "min_distance": 0.001, "verbose": true}}' --lower-thresh 0.5 --upper-thresh 0.5 --train-set-size 0.5 --crop-top-left 308 248 --crop-bottom-right 436 332 --depth-min-mm 310 --depth-max-mm 340 --rosbags-select 2024_03_01_two_bites 2024_03_01_two_bites_2 2024_03_01_two_bites_3 2024_02_29_two_bites --seed 42 --temporal-window-size 5 --spatial-num-pixels 10 --no-train
ros2 launch ada_feeding_perception ada_feeding_perception.launch.py
ros2 service call /toggle_food_on_fork_detection std_srvs/srv/SetBool "{data: true}"
ros2 topic echo /food_on_fork_detection
ros2 run rviz2 rviz2 --ros-args -p use_sim_time:=true
. Add an image for/local/camera/aligned_depth_to_color/image_raw
and/food_on_fork_detection_img
ros2 bag play 2024_03_01_two_bites --clock
. Note that this particular rosbag doesn't start streaming data until 16 secs in./food_on_fork_detection_img
. Verify that that view helps you re-align the fork.Documented Commands for Training
python3 food_on_fork_train_test.py --model-classes '{"distance_no_fof_detector_with_filters": "ada_feeding_perception.food_on_fork_detectors.FoodOnForkDistanceToNoFOFDetector"}' --model-kwargs '{"distance_no_fof_detector_with_filters": {"camera_matrix": [614.5933227539062, 0.0, 312.1358947753906, 0.0, 614.6914672851562, 223.70831298828125, 0.0, 0.0, 1.0], "min_distance": 0.001}}' --lower-thresh 0.25 --upper-thresh 0.75 --train-set-size 0.5 --crop-top-left 344 272 --crop-bottom-right 408 336 --depth-min-mm 310 --depth-max-mm 340 --rosbags-select 2024_03_01_no_fof 2024_03_01_no_fof_1 2024_03_01_no_fof_2 2024_03_01_no_fof_3 2024_03_01_no_fof_4 2024_03_01_fof_cantaloupe_1 2024_03_01_fof_cantaloupe_2 2024_03_01_fof_cantaloupe_3 2024_03_01_fof_strawberry_1 2024_03_01_fof_strawberry_2 2024_03_01_fof_strawberry_3 2024_02_29_no_fof 2024_02_29_fof_cantaloupe 2024_02_29_fof_strawberry --seed 42 --temporal-window-size 5 --spatial-num-pixels 10
"aggregator_name": null
to the kwargs and--viz-fit-save-dir "../model/results"
to the command line arguments to graph out the decision boundary for different aggregator functions.Ideas for Future Improvements
Before opening a pull request
python3 -m black .
ada_feeding
directory, run:pylint --recursive=y --rcfile=.pylintrc .
.Before Merging
Squash & Merge