Large spot IDs collapse to same value

[funkey]

When I tried to open an annotation XML created by another script (i.e., not by manual annotations within MaMuT), only one "Spot" per frame was shown, although the XML file contains a few hundred per frame. The problem seems to be the use of large IDs. My annotation XML contains spots with:

        <Spot ID="19152000001" ...  />
        <Spot ID="19152000002" ... />
        <Spot ID="19152000003" ...  /> 
        ...

By changing the IDs to

        <Spot ID="1000001" ...  />
        <Spot ID="1000002" ... />
        <Spot ID="1000003" ...  /> 
        ...

I can see all spots as expected. I suspect that somewhere during reading of the XML, the IDs get converted to float32, causing collapsing of similar large values:

In [3]: int(np.float32(19152000001))
Out[3]: 19152001024

In [4]: int(np.float32(19152000002))
Out[4]: 19152001024

In [5]: int(np.float32(19152000003))
Out[5]: 19152001024

In [6]: int(np.float32(1000001))
Out[6]: 1000001

In [7]: int(np.float32(1000002))
Out[7]: 1000002

In [8]: int(np.float32(1000003))
Out[8]: 1000003

[tinevez]

Hummm the IDs get converted to double (or float64?) normally, so we normally have access to the whole range of ints. Do your IDs go beyond 2 billions?

[axtimwalde]

Hi @tinevez, yes the ids are above 2bio, as shown in the example above where they are at 19bio. Is that illegal? I.e. how hard is it to use long for the ids instead of doubles?

[tinevez]

I think I would have to rewrite the TrackMate model, because all ids are int :(

And I am afraid we did not attend to that problem in Mastodon, where we have this cap too. Am I correct @tpietzsch ? I seem to remember you discuss using long instead of int in another use of mastodon-collection discussing with Philip/

[tpietzsch]

It should be easy to handle IDs as long. These are anyway just IDs from the file that are only used to establish identity for linking the spots. They are lost in the imported model. (You can add a new property if you want to preserve them.)

To change to long when importing, you would need to change this, https://github.com/fiji/TrackMate3/blob/4cec44c1e7a3183855d29d744fe65886692b4366/src/main/java/org/mastodon/revised/model/mamut/trackmate/TrackMateImporter.java#L219 this https://github.com/fiji/TrackMate3/blob/4cec44c1e7a3183855d29d744fe65886692b4366/src/main/java/org/mastodon/revised/model/mamut/trackmate/TrackMateImporter.java#L241 etc.

Accommodating > 2bio spots in total is a bigger task. But doesn't look like this is required here?

[tpietzsch]

just to clarify: I'm talking about Mastodon. Not sure what would need to happen in MaMuT.

[tinevez]

For MaMuT and TrackMate I made the mistake of indexing spots in lists. So their max number is bound by int values :/

[tinevez]

So yes I would go to Mastodon which in my humble opinion is superior.

[tinevez]

Actually we also use int IDs in Mastodon. See: https://github.com/bigdataviewer/mastodon-graph/blob/master/src/main/java/org/mastodon/graph/GraphIdBimap.java

[tpietzsch]

Yes, as I said accommodating > 2bio spots is a bigger task. But supporting long IDs in imported XML files is easy. They will anyway get new internal IDs assigned. It's just about making sure the long IDs survive until the import is complete (which is easy).

[tinevez]

I am closing this issue for now, as MaMuT will not be able to harness it.