tuning parameters

[aryarm]

After working to fully understand morphological operations for the past couple weeks, I was able to solve the blob coalescence problem. The current solution only performs erosion (and nothing else), leading to blobs that are actually smaller than the ants themselves. The blob coalescence problem is also helped by the Kalman filter, which accurately rescues the identification of the ants after they've coalesced by keeping a semi-accurate location of the disappeared ant in memory as a prediction and then correctly reassigning the ants after they've separated sufficiently.

Unfortunately, there are still a few errors with the pipeline.

1. The pipeline still runs into trouble when ants deviate even slightly from a straight path across the bridge. In these cases, the blob continues in the direction predicted by the constant velocity model instead of tracking the ants. Increasing the Kalman filter's MotionNoise and decreasing the MeasurementNoise helped only a little.
2. When an ant disappears off screen and another ant appears in its place, the pipeline recognizes them as the same ant (and assigns them the same ID). My guess is that this has to do with the invisibleForTooLong parameter. We should try decreasing it.
3. Ants that move particularly quickly are sometimes not tracked until they reach half-way across the frame. I currently have no ideas for what may be causing this or how I could fix it. In the future, we may want to crop out the full length of a bridge instead of simply a segment. I don't see any reason why we wouldn't want to, and it will allow the pipeline to have more information about whether the ant actually crosses a bridge or stops 2/3 of the way and turns back, for example.
4. Ants that travel underneath a bridge are still tracked. This problem was made a lot better by the addition of fluon underneath the bridges. The fluon also helped keep ants from climbing underneath the bridges. However, the Kalman filter allows the pipeline to recognize ants that have disappeared, including those that disappear underneath a bridge. The simplest solution to this problem is to extract only the bridge during cropping and nothing below or above it, unlike what was previously done. Unfortunately, bridges are rarely exactly horizontal in a video, so they probably won't fit perfectly. Future versions of the pipeline may include a "rotation" step, in which a video is rotated so that bridges are exactly horizontal before cropping.
5. Some ants are simply never tracked. This occurs rarely but may still be a cause for concern. You should determine whether this is due to the erosion or the speed of the ants.

[aryarm]

Problems 3 and 5 seem to be because of the erosion, not the speed of the ants. Some of the blobs appeared to be very small, even single-pixel-width sticks. The erosion was so strong that some frames didn't even have any foreground pixels.

Rather than decreasing the effect of the erosion, I first sought to try setting the MinimumBlobArea parameter to 0, so that even single pixel foreground pixels would be detected as ants. This actually seems to have resolved the problem because for the occasional frames in which no foreground pixels are detected, the Kalman filter kicks in and predicts the ant locations. A byproduct of this method may be that ant tracks will be more readily deleted. We may have to increase the visibility.

One thing to note: problem 3 improved after this step but didn't completely resolve.

[aryarm]

After reducing the MinimumBlobArea, problem 1 became arguably worse. Now, two blobs are detected instead of one. The first blob is the prediction under the constant velocity model and the second blob is the actual ant! This occurs at 00:44 in Ctest2_h1_mask.mp4

[aryarm]

It's important to have a MinimumBlobArea that is non-zero. Otherwise, not-ants will appear in the tracking output. See 5ddcee9

[aryarm]

Problem 3 has become a very real problem. The large majority of NA values in the tracking output are because ants don't start getting tracked until they reach the center of the bridge. But we determine which direction they are moving in only by looking at whether they move from one side of the frame to the other. Either come up with a new strategy for determining direction or fix the problem of late appearing ants.

Also, you should crop out everything but the bridge to see whether that helps things.

[aryarm]

Cropping out everything but the bridge has helped confirm that some ants (approximately 8 in the first test video) are not getting tracked at all. Interestingly, the output doesn't include ants with IDs 6, 9, and 11. Perhaps these three ants are part of the 8 that are missing. Maybe the pipeline is deleting ant tracks before it returns them to us?

[aryarm]

I also changed the output of the pipeline. Now, it outputs the sum of the distances that the ant travels in each frame. Negative distances represent moving right to left (instead of left to right). I have yet to verify this data manually.

Update: there's no point to summing the distances between frames. just do it from start to finish (see commit f9b9137)

[aryarm]

I think the ants that aren't appearing are because the pipeline is continuing to track ants after they move out of frame and then assigning different ants to the same track. Evidence in support of this hypothesis includes that some of the coordinates of ants are negative or larger than the dimensions of the video. Graphs of the ant track coordinates are the best indicator of this phenomenon. Here is a plot of ant 4, which I suspect is actually 3 different ants: The plot above is surprising not least because it shows ant 4 teleporting nearly 40 px in the span of one frame. I can't understand why the Kalman filter (or Hungarian algorithm?) would do something like this. I'll have to look at the parameters.

[aryarm]

I made some visualizations to aid in further debugging and committed the code for generating them in b89aaba. It appears as though I've more or less solved the problem of ant tracks getting combined (see e93287d), but we're still missing 1 ant out of the 17 that I've manually verified are crossing the bridge in Ctest2_h1.mp4. The plots seem to indicate that ant 9 might actually be two ants. In fact, I think ant 11 became ant 9! If so, it might be harder to separate them, since their tracks don't both start and end at the edge of the frame. Perhaps we could reduce the cost to the Hungarian algorithm of not finding a matching ant?

[aryarm]

Update: the missing ant appears at 0:48 in Ctest2_h1.mp4 between ants 17 and 19. It appears in the mask video for 5 consecutive frames (as a 3-5 pixel blob) and then another two separated frames later (both as only a single pixel). At the time, the minimumBlobArea was 5, so I reduced it to 4. This led, surprisingly, to several of the ant tracks getting broken up into discontinuous segments. Unfortunately, our missing ant still isn't appearing!

Also, reducing the minimumBlobArea seems to maybe slow down the tracking pipeline?

[aryarm]

I found the missing ant! Hoorah! We now have 17/17 of the ants tracked in Ctest2_h1.mp4. The solution was to reduce the minVisibleCount (see ffc800a), but I kept the minimumBlobArea at 5. One problem that still remains to be solved is why the track for ant 18 (the missing ant) was so short to begin with. Why isn't the ant blob ever bigger than 5 pixels and why is it only 3-5 pixels for so few frames? Is there a way to improve this without touching the erosion?

[aryarm]

I can confirm that ant 11 does actually become ant 9 at the end of its lifetime. Reducing the costOfNonAssignment for the Hungarian Algorithm did not help. As you reduce the costOfNonAssignment, other tracks start to break up into fragments before ant 9 does; it's quite stubborn. I also tried splitting the unassignedTrackCost and unassignedDetectionCost by calling the assignDetectionsToTracks() function in its second form, but to no avail.

[aryarm]

I tried reducing the minimumBackgroundRatio so that ant blobs would be larger and ants would be tracked for more of their time on screen. However, this led to some of the ant tracks getting fragmented and others getting merged. While fragmented tracks could get filtered out using the minVisibleCount parameter, changing the Kalman Filter parameters didn't help the merged tracks. For example, here's a plot using a MinimumBackgroundRatio of 0.7 instead of 0.75: To generate this plot, I also used a costOfNonAssignment of 15 (instead of 20) and the following Kalman Filter parameters for MotionNoise and MeasurementNoise: [100, 15], 100 (instead of [150, 50], 50).

[aryarm]

things I was going to do next but never got the chance to

(but that @JarredAllen might want to do)

1. Make the ants be tracked for the full width of the frame. Before I left, I achieved relative success in this area by lowering the MinimumBackgroundRatio (see this). However, this led both to fragmentation of some tracks and merging of some other tracks. Is there a way to have the best of both worlds?
2. Tune the parameters to maximize tracking accuracy on all test videos, instead of just Ctest2_h1.mp4. Although the current parameters work great for Ctest2_h1.mp4, they probably aren't ideal for all videos. In particular, I had to lower the minVisibleCount to get 17/17 accuracy in Ctest2_h1.mp4, but I would imagine this might lead to unwanted fragments in other videos (and may actually help in point 1 above?). In fact, a cursory glance over the results of running the pipeline on other test videos was, indeed, a bit disappointing. To be fair, this may have in part been because I rotated Ctest2_h1.mp4 to be perfectly horizontal (so that I could crop out non-bridge parts of the video) but didn't rotate most of the other videos.
3. Test the tracking parameters on other types of test videos (and with other ant colonies) than the ones I was working with. I was thinking of documenting how the tracking parameters might need to be altered depending on the ants and the type of pipeline. For example, most of the ants in the videos I was working with had grey RFID tags on their backs. These allowed the ants to blend in with the bridge they were walking on and made it particularly hard to track them in some cases (since less pixels would be interpreted as foreground). Ants without RFID tags might allow even more aggressive erosion, but changing the erosion might necessitate changing all of the other parameters; the morphological operations are performed so early in the pipeline that the rest of the parameters are usually pretty sensitive to them.
4. All of the other issues in the repository.
5. Try to make the pipeline smarter. For example, the pipeline currently doesn't use knowledge about ants detected in a previous frame to control morphological operations in the next frame. This might help us balance the amount of erosion performed so that it increases when it expects ants to get close to each other and decreases otherwise. Another thing I wanted to do is allow the pipeline to somehow automate the tuning of parameters for a set of videos that use the same types of ants and the same camera. Basically, do point 3 above automatically. I was imagining a GUI where the user could manually annotate a couple training videos for this purpose.

[aryarm]

Hey @JarredAllen,

I was looking around the repo and noticed that you had pushed some changes to ant_tracking.m that I had left uncommitted at the end of the semester.

In fact, I had actually left those uncommitted because I considered those parameters to be overall worse than the ones I had been using before. As I explained in the comment above, I thought there might be a way to improve upon those new parameters even though I considered them to be worse at the time. I didn't want you to have to copy them from that comment if you wanted to use them, so I left them there instead of removing them.

If it was truly your intention to commit them because you decided they were actually better than the original parameters, then please disregard this message. I just wanted to make sure you weren't using them solely because you thought that was the "best" version of the pipeline before I left. I actually considered the "best" version to have been the one right before your commit.

Sorry for the confusion. I should have communicated this to you earlier. My bad.

[JarredAllen]

Thanks for letting me know about that. I have since changed a couple of the values there to other things, but I also have not yet gotten around to all of the values which were changed in that commit, because I have been working on a fair number of other things, as well.

I'll keep this in mind when looking at the rest of the values which I haven't tweaked yet.