search

dstl / Stone-Soup

A software project to provide the target tracking community with a framework for the development and testing of tracking algorithms.
https://stonesoup.rtfd.io
MIT License
384 stars 126 forks source link

Issues with the GOSPA metric #999

Open narykov opened 2 months ago

narykov commented 2 months ago

I ran into a couple of problems trying to use GOSPA in place of the OSPA metric:

  1. While compute_OSPA_distance returns a scalar, compute_gospa_metric is designed to return a dictionary of values (including that of distance). This leads to problems with visualising the GOSPA errors using built-in tools, as scalar values are anticipated.
  2. In the Metrics example, it is not possible to replace OSPA with GOSPA metric (e.g., via from stonesoup.metricgenerator.ospametric import GOSPAMetric as OSPAMetric) as it leads to issues when trying to extract a timestamp from a numpy array of Particle elements.

Perhaps, some of these issues are known (at least I heard about the second one from @A-acuto), but I couldn't find them explicitly outlined here.

sdhiscocks commented 2 months ago

I think issue 2. was fixed in #967

A-acuto commented 2 months ago

@sdhiscocks Thanks for pointing out that! I was not aware, I found that issue some time ago and I forgot about mentioning before. Glad it is fixed now

simonrmaskell commented 1 month ago

If we want a common visualisation process that is agnostic to whether we use GOSPA or OSPA, I think there is still an issue here. Do we want the OSPA output to look like GOSPA's or vice versa? I don't think we want to persist the current status quo

narykov commented 1 month ago

While compute_OSPA_distance returns a scalar, compute_gospa_metric is designed to return a dictionary of values (including that of distance). This leads to problems with visualising the GOSPA errors using built-in tools, as scalar values are anticipated.

To get any visualisation, I 'hacked' computed_over_time() in GOSPA by putting metric.value = metric.value['distance'] above line 196.

https://github.com/dstl/Stone-Soup/blob/13f7578f7a5d4238b00c52a07fc0b184506687ec/stonesoup/metricgenerator/ospametric.py#L192-L196

This way I only access 'distance' metric, which then can be visualised. By replacing 'distance' with 'localisation', 'missed', or 'false', it is (potentially) possible to get access to other information that is particular to GOSPA.

P.S. Visualisation of the GOSPA metric is accessed via title='GOSPA Metric', and of the OSPA metric via title='OSPA distances', which is not very intuitive.