For each run of the 3d_dataset, I insert the simple payload order algorithm in the integration unit test. To verify how reliable the payload order algorithm is, I relied on checking if the scores, a list of the number of times the description detected was close enough to the truth, was decreasing and if the list of the 5 target's track distance is increasing.
Running the payload order algorithm requires the letter and shape confusion matrices. I used the irl unit test generated confusion matrices for running the algorithm. When not using penalty, the order of the payload didn't appear to change. When using penalty, the order changes better, but the worst target was not last.
Test Plan
If the placement of the worst target detected was ignored, the track distance of the successfully tracked object were ordered in an increasing order. There might imply that it is the confusion matrix influencing the order of the payload.
Issues
Issue is that the unit test doesn't know how to handle a person test case. With letter and color, there is no specific probability value to assign when shape is a person. I will likely work on that after making this PR update.
Summary
For each run of the 3d_dataset, I insert the simple payload order algorithm in the integration unit test. To verify how reliable the payload order algorithm is, I relied on checking if the scores, a list of the number of times the description detected was close enough to the truth, was decreasing and if the list of the 5 target's track distance is increasing. Running the payload order algorithm requires the letter and shape confusion matrices. I used the irl unit test generated confusion matrices for running the algorithm. When not using penalty, the order of the payload didn't appear to change. When using penalty, the order changes better, but the worst target was not last.
Test Plan
If the placement of the worst target detected was ignored, the track distance of the successfully tracked object were ordered in an increasing order. There might imply that it is the confusion matrix influencing the order of the payload.
Issues
Issue is that the unit test doesn't know how to handle a person test case. With letter and color, there is no specific probability value to assign when shape is a person. I will likely work on that after making this PR update.