Getting duplicated trajectories with different obj_ids

[elhananby]

Hi Andrew,

I noticed something weird today when analyzing some data: quite often I see duplicated obj_ids. Meaning, two different obj_ids with the same (or extremely similar) trajectories. See this image for example: The trajectories come from two different obj_ids. One is slightly shifted with regard to the other, as well as a shift in the timestamps and frame number. Do you have any idea what could cause this? I can probably find a way to filter this out, although how do I know which obj is the 'true' trajectory? Is it possible this is related to the many image_tracker: ... acquisition duration negative? errors I have been seeing on a couple of my cameras?

Thanks, Elhanan

[astraw]

My first thought is that your cameras are not perfectly calibrated and that they fall out of sync so that you are getting two valid reconstructions but spatially and temporally offset. The spatial offset seems rather severe, however, and I think you would know if your calibration was so bad. The temporal offset is also severe but is entirely consistent with the warnings you mention.

If you look at the raw 2d data do you see one or more cameras lagging behind? Consistent with the temporal-offset-due-to-synchronization-errors hypothesis, this is what I would expect. For now, I think the best way to inspect all this is still to use the old flydra code. In particular, use the script convert_kalmanized_csv_to_flydra_h5.py to generate a flydra-compatible .h5 file and then run the script flydra_analysis_plot_timeseries_2d_3d (installed with flydra_analysis python pacakge) using the appropriate command-line arguments as shown in the flydra documentation gallery. This will give you an interactive viewer where you can zoom in on the 2d (and 3d) data while viewing object ids of the traces.

I am puzzled particularly by the spatial offset, so please let me know what you find when you look into the 2d data and also how good your calibration is.

[astraw]

By the way, I realized the name convert_kalmanized_csv_to_flydra_h5.py is outdated, so I will change it to convert_braidz_to_flydra_h5.py.

[elhananby]

My calibration reprojection errors are mean=0.35 and std=0.30, which I think is rather good. I did the analysis you suggested (on the same two objects I showed before), but I couldn't really find anything that seemed unusual? See image:

Also, I forgot to mention - I have six cameras total, two Basler ace acA800-510um and four Basler ace 2 a2A1920-160umBAS. I only get the acquisition duration error with the two acA800 cameras, the rest seem to be working fine. I may try to recalibrate and record only with the newer cameras, and see if I still get similar issues.

[astraw]

There is something clearly wrong in the spatial domain in your figure -- for cameras xx79, 53, 59 and 23, I see two back-projected 3D traces (the thin blue for X and yellow lines for Y coordinate) where there is clearly only a single object detected in the 2D data (the thicker red for X and green for Y). The offset of the projected 3d data to the 2d data indicates a spatial problem (if indeed there was only a single object which seems the case here). Weirdly for most of these cameras, the error looks like it is only in the X (cam 53) or Y (cam 79, 59) pixel coordinate. Only for cam 23 is there an offset for both pixel coordinates.

Although your data look temporally OK in these plots, the time scale on your axes is quite compressed. Perhaps if you zoom into an inflection point in the trajectory, you will be able to see a synchronization error. The cause of the issues you are facing is not at all clear to me, but I suggest to first fix potential synchronization problems as they make everything else more difficult.

Aside from a calibration issue, another possible spatial issue might be that the 2d detections are coming from more than one point on the tracked object. (Braid assumes the tracked object is an infinitely small 3D point.) Could this be an issue here? Indeed your reprojection distances are low -- is your calibration that good for all cameras? If so, then indeed the calibration would not seem to be contributing to a problem here.

You should just be able to remove the cameras from your calibration and then retrack it (with braid-offline-retrack) without collecting new data. I doubt there is anything fundamentally wrong with the new or old cameras - this should not be a problem.

[elhananby]

• Regarding the synchronization: is it possible that it's a hardware issue (triggerbox, cables)? Or is it more likely to be a software issue?

• Regarding the spatial issue: could it arise from a small change in the camera position or focus? (it happened to me in the past that I accidentally bumped one of the cameras, and it would screw up the whole calibration. Although then I would just get an empty kalman_estimates csv).

• I've also attached a more zoomed-in figure of the previous plot, right at the time of detection, although I'm not completely sure what to make of it.

• I also doubt that it comes from more than one object, since I have just one fly in the arena, and I'm limiting Braid to detect only a single object.

• The reprojection error for all cameras is between 0.3-0.4 for both mean and std.

[astraw]

It seems unlikely 3 or 4 of your cameras were bumped simultaneously, but if they were, this could indeed cause the spatial issue.

Regarding the synchronization - I doubt it is hardware per se but rather my suspicion is that the computer may simply be too busy to handle everything and may fall behind. I also cannot exclude software bugs that perhaps only are relevant in such overload scenarios. You could try limiting the framerate from the triggerbox just to check if this helps. Also the code could certainly be improved, such as made more computationally efficient.

Regarding the plot - could you also zoom in on the Y axes at a time when the fly switches direction? We want to look at the time at which the direction changes in the different cameras and see if it is exactly synchronous across them. It might help to blink or rapidly move an LED instead of looking at a real fly trajectory. Regarding multiple points - how large (in pixels) is the fly in your camera? If it is only a few I agree, but if the fly fills many pixels, some cameras may be detecting the head while others the abdomen.

Maybe another suggestion is to save MKV videos and to plot the tracked data over them. This may help understand and diagnose the issue. (I think this should already work with the released braid-process-video command but I have a some improvements I am working on which will be available as soon as possible.)

[elhananby]

Hi Andrew,

So I'm still checking some things, but I did figure out what caused the acquisition duration negative errors: the exposure time of the two Ace1 cameras was lower than the other cameras (1500us vs. 5000us). Once I increased it to around 3000us, the messages stopped completely. I'm not completely sure why the exposure time had such an effect – since the cameras are triggered externally, I assumed it shouldn't change the amount of incoming frames?

[astraw]

I have an idea. Were you getting these acquisition duration negative errors only in the initial seconds of operation? It could be that the clocks in the Braid computer and in the triggerbox (and - more importantly - Braid's model of the triggerbox clock) have not settled down yet. In that case, though, things should settle down after a minute or two. Furthermore, I don't think this would not explain the weird temporal and spatial shifts you are seeing.

Typically when changing the exposure time causes synchronization problems it is when the exposure time is too long and the camera cannot acquire at the rate the trigger pulses are coming. In your case it seems to be backwards - that increasing the exposure time helped. Just to be sure, can you confirm that your trigger rate from the triggerbox is slower than ~5000 us (i.e. less than 200 fps)?

Does your trigger source provide enough current so that it can reliably power the trigger of all the cameras you have? IIRC, the current that a digital out on an Arduino can source is only something like 20 mA and whereas the opto-coupled trigger inputs on Basler cameras take a good part of that. (This is why the triggerbox design has a buffer amplifier.)

[elhananby]

Previously I had constant negative duration errors, throughout the entire recordings. Although now instead I am getting a lot (a few per minute) of mean reprojection 100x distance value 1060657 out of expected range, though this might be due to a calibration issue that popped up for some reason?

I record at 100fps, and I use the original triggerbox design (https://github.com/strawlab/triggerbox/blob/main/hardware_v1/circuit_board/triggerbox-schematic.pdf). I haven't had any issues with it until recently – I looked back a few recordings, and it seems like the issue only started last month or so.

[astraw]

Yes, the mean reprojection distance warning is saying that the reprojection errors are very high, which would be consistent with a bad calibration.

[elhananby]

Hi Andrew, Sorry, I kind of forgot about this issue. Eventually I had to reinstall Ubuntu anyway, and now it seems like the constant acquisition duration negative are solved. They still happen occasionally, and only in the two Basler Ace1 cameras, but the tracking works fine apart from that.

Closing this for now, thanks