Include object detection to improve task recognition

[sakshambassi]

need to discuss this with @chenshaoyu1995 about this, doubts:

1. how are the inputs structured?
2. what are the types of the objects covered?
3. is there already a mapping from objects to recipes?

[chenshaoyu1995]

Below is an example of the output structure.

[{'pos': [-0.15625146866666473, -0.8371317528212436, -0.6927490014418084], 'id': 1, 'label': 'jar', 'last_seen': 133314156588973631, 'status': 'outside'}, {'pos': [-0.4858943354638425, -0.8379596287000366, -0.45763596621021796], 'id': 4, 'label': 'plate', 'last_seen': 133314156620960472, 'status': 'tracked', 'xyxyn': [0.9515625, 0.5416666666666666, 0.9984375, 0.625], 'state': {}}, {'pos': [-0.45783918563425385, -0.8485310446669867, -0.4034173386891438], 'id': 5, 'label': 'tortilla', 'last_seen': 133314156620960472, 'status': 'tracked', 'xyxyn': [0.89609375, 0.6541666666666667, 0.94296875, 0.7375], 'state': {'plain': 0.6100036977501946, 'cut': 0.20024783173920785, 'nutella': 0.07122405969726602, 'in-package': 0.08264436717609581, 'peanut-butter': 0.008253306691368097, 'nutella+banana': 0.032382052176811366, 'pb+jelly': 0.020797084049964557}}, {'pos': [-0.6377640109620134, -0.840088897205035, -0.11326367265112079], 'id': 6, 'label': 'tortilla', 'last_seen': 133314156620960472, 'status': 'tracked', 'xyxyn': [0.42578125, 0.4444444444444444, 0.47265625, 0.5277777777777778]}]

It should be the compatible as what Bea has mentioned before in the discord reasoning channel

For object classes that are being detected by Detic, you can find them at here

As far as I know, there is no mapping from objects to recipes for this vocabulary.

[roquelopez]

@chenshaoyu1995 what stream do we have to use to get these outputs? Thanks

[chenshaoyu1995]

detic:memory