Handle the case of multiple box experiment

[atuleu]

At this point in time, TrackingDataDirectory are stored into an Experiment such as none of them overlaps in time. The primarly goal is to be able to analyze with the same metadata an experiment that have been stopped and restarted multiple time.

However if we use multiple boxes to track the same colony, for eample one for the nest area and one for the foragging area with different climate condition (one staying in the dark not to distrub the ants).

We would like in that case to be able to use the same metadata, and we would overlapping TrackingDataDirectory in time.

This has several implication:

• We could not order FramePointer by using simply its TrackingDataDirectory start time and its frame number. Each frame should be represented by a Point in time to perform that kind of operation.
• We could have frame that happen at the same time.
• We have no way for the user to prevent to add two completely unrelated tracking data directory to the same project.

Right now every point in time is identified by a frame. But maybe it is not a good approach, and it should be done by passing an actual Time. This would solve the problem for ordering FramePointer and Identification as we are actually using timestamp. However this will make it harder to refer to a single frame in perticular by a timestamp, as for this we would need to open the tracking data files and read through all of them sequentially until we found the right frame

Having to deal with multiple boxes also adds a new problem with coordinates for trajectory and position. Now we have two "hermetic" world that share the same coordinate system. this implies:

• Ant should not collide between the two zone (or maybe they could if actually the two zone are overlapping,if we imagine an hypothetic setup when several camera track region of the same area
• Trajectory stops (no interpolation) when the Ant leave one zone and reappear to another.

This also implies that for the same experiment we do not have a single movie stream but multiple.

[atuleu]

bump @matthiasruegg , @nathaliestroeymeyt and @SeanMcGregor for request for comment

[nathaliestroeymeyt]

Maybe a naive comment, because I am not a programmer and don't know the details of the architecture of your programmes.

But wouldn't adding a 'box identity' information be helpful? For example if there was an option for the user to specify an option 'multiple box experiment', then to specify that in that experiment, say, Highgarden is connected to Casterly Rock, and maybe, that Highgarden is to the left of Casterly Rock. Then you could have two sets of non-overlapping coordinates, where the x-centre would be between on the right edge of Highgarden and on the left edge of Casterly Rock, and all x-coordinates for Highgarden are negative and those for Casterly Rock positive? And then different (non-overlapping) sets of zones could be defined for each? Sure, it would require the user to know what he is doing but if you are going to analyse tracking data you sort of need to know what you are doing anyway...

In a situation of multiple box experiment it would definitely be useful to be able to synchronise the two tracking sets in time. Not sure if this can be done by frame number (maybe not fool proof if they become unsynchronised) or whether you would need to use an actual Time.

Sorry if I am not more helpful, I was not sure whether you had specific questions or just wanted general comments

[atuleu]

Maybe a naive comment, because I am not a programmer and don't know the details of the architecture of your programmes.

But wouldn't adding a 'box identity' information be helpful? For example if there was an option for the user to specify an option 'multiple box experiment', then to specify that in that experiment, say, Highgarden is connected to Casterly Rock, and maybe, that Highgarden is to the left of Casterly Rock. Then you could have two sets of non-overlapping coordinates, where the x-centre would be between on the right edge of Highgarden and on the left edge of Casterly Rock, and all x-coordinates for Highgarden are negative and those for Casterly Rock positive? And then different (non-overlapping) sets of zones could be defined for each? Sure, it would require the user to know what he is doing but if you are going to analyse tracking data you sort of need to know what you are doing anyway...

Clearly to give each zone a name is an option. The user will add tracking data directory to a zone with a name and we would only enforce the non-overlapping condition for each zone. This what I thought at first

However I am clearly not in favor to merge the position of two different zone by adding some kind of offset to one of the zone. In the case of multiple camera tracking the same physical area, it makes sense. But in the use case at Keller's group (its two different physical area) it makes no sense, and is error prone:

• either we compute the offset for the user, but what if we get a 100MP camera in 5 or 10 years
  • either the user puts the offset, and on a mistake, we see a lot of ghost interaction between ants not in the same physical area because the zone overlaps in the computer virtual memory.

I believe anyway that the multiple camera setup for one physical is not going to show up in a long time, or ever. It should also certainly be dealt at the tracking level (artemis / leto) and just give one single super high resolution movie (or several one for each camera) but tracking data in one single reference frame and not multiple one.

So I would be more in favor of an interface that when you query for a trajectory between two time point, and give a list of trajectory segment. Every time the ant changes one zone, it creates a new segment. Each segment also store to which zone they are in. Therefore the position (0,0) could be in nest or foraging zone. Since the interaction detection will be done by myrmidon, you won't need to care for it in your analysis programs. If you need it by analyzing raw data in a frame, your program will need to have to check for (t,x,y,ZoneID) coordinates for an Ant.

In a situation of multiple box experiment it would definitely be useful to be able to synchronise the two tracking sets in time. Not sure if this can be done by frame number (maybe not fool proof if they become unsynchronised) or whether you would need to use an actual Time.

That would need a expensive piece of hardware with euresys framegrabber and lots of refactoring in leto for this to be possible (not unpossible, just lots of works). And it creates many issue, as what should we do when a computer drops a frame because it cannot process it, should the other do the same ? Its is also almost impossible without an extensive works to use the same FrameID for frame when we produce it. And what happen if a single computer crashes, what should we do... stop or continue ? Oups it was just a network failure it wasn't down, and now we have to resynchronized because their was some different number of frame dropped by the computers... This is really complex

Do you really need exactly frame synchronized trajectories for ants that are not in the same zone ? There is so many little gap when the tracking is lost for one or two frames, that I do not think putting in memory big vector with first column time and subsequent column position for all the ant is the right solution, as we will need to put lots of NaN or undefined data in many place in such a table. I would rather give a table for every ant, with their corresponding unique time vector for when we have tracking data for this ant. And with such an approach, we won't need time synchronization between the two boxes (they are more or less synchronized at a very few milliseconds intervals through the computer clocks. I don't think ants will blink form one box to the other in 10ms. Of course you will be able to query for two set of trajectory for two ants within the same time window, but you will never be sure that the times exactly matches at each index in these two result tables (and this even for the same physical zone because of mis-detection). But I would for myself rather have a dataset formatted this way and have to check time differences between rows than to deal with NaN or undefined value in a table. If you would need a big table then maybe we could make or use a function like https://www.mathworks.com/help/matlab/ref/timetable.synchronize.html (I am sure it exist for python and R too) To merge in a common time vector table all the data and put NaN when data is missing.

Sorry if I am not more helpful, I was not sure whether you had specific questions or just wanted general comments

No its a very general discussion as I need feedback to provide the best solution for you.

[atuleu]

RawFrame have been introduced in the dev/time and dev/refactor branch

[atuleu]

Wrong issue closed

[atuleu]

Done now with the introduction of priv::Time and priv::Space (formerly priv::Zone)