Video misalignment

[gainbrew]

I have encountered an error where the frames of a video are inconsistent. For example, when trying to label landmarks on fly larvae, I will label the desired parts but then when going back to that frame the larvae appears in a different position than when I labelled the first time. It seems that depending on if I am scrubbing forward or backward, the movie is actually on a different frame than what is being displayed. I have attached a video for clarity.

These videos are shot with the H.264 codec, 30fps. When using manage movies to add unprocessed videos, they never load. I have been using VLC to 'convert' to H.264, MP4/MOV even though I thought they should be in that format already. That enables APT to open the video file, but now I have this problem. Hoping to clarify the proper format for videos and if the wrong format or codec could lead to this problem. Thanks!

https://user-images.githubusercontent.com/67379938/117871389-53140d80-b26b-11eb-86a5-36e437f1c5a6.mp4

[allenleetc]

Hi, thanks for your report. What operating system and MATLAB version are you running? We have seen this on rare occasions with certain videos, but always on Linux I believe (it looks like you are running Windows?).

Yes it's possible there is an issue with the encoding. APT supports some specialized movie formats, but for general purposes uses MATLAB's VideoReader(). The idea is that if MATLAB can read it, then it should work with APT. The flipside though is that if MATLAB/VideoReader somehow has trouble reading a video (eg in a random-access, forwards-then-backwards fashion), then APT will inherit the same behavior.

Are you able to share your original and/or re-encoded videos? If so I'd be happy to play with it a bit on our end to debug.

[gainbrew]

Thanks for your help with this. I'm running Windows 10 and MATLAB 2019a.

I can confirm that MATLAB's VideoReader() function cannot read the source video, but after re-encoding has no problem. Below you can find a google drive link to the two videos. I shortened them to save space, but the behavior in MATLAB's VideoReader() persisted so figure it should still be fine for troubleshooting purposes.

https://drive.google.com/drive/folders/1gLaH0-gcYzS62pKMci6FSHdj0y7rVD7W?usp=sharing

[allenleetc]

I tested your two videos and confirmed that I could open the 'reencoded' mp4 in MATLAB, but not the 'source' mp4.

However, so far I can't reproduce the jumpy/history-dependent reading of the reencoded movie within APT. I made a new project, added the reencoded movie, labeled a few frames, and tried scrolling back and forth. So far things seem ok for me. It may be significantly slower to scroll backwards, but the frames are not wildly different scrolling backwards vs forwards as in your demo video.

When you say that the behavior of VideoReader() persisted with the shortened movies, do you mean that you reproduced the same jumpy/history-dependent movie-reading when using these shorter videos in APT? Just checking in case shortening the movies may have had an effect.

Here is something else to try that may give a clue. After you have started APT, your MATLAB path will include various command-line functions. The following commands will try some diagnostic tests of a movie you specify. These are internal development utilities, so not polished for production use, but maybe they are useful in this case.

res = VideoTest.test1gen('c:\path\to\movie_with_issues.mp4','nsamp',20);
VideoTest.test1internal(res); 
% If any frame does not print as "OK", this could be a clue

Just from studying your demo video, I am inclined to think that it is indeed a codec issue. If it is not difficult to experiment with a different encoding that may be worth trying to see if it has any effect.

[gainbrew]

What I meant about the VideoReader() behavior was that MATLAB still wasn't able to read the 'source' video after being trimmed down. I didn't try running the videos in APT, but I just went ahead and tried reproducing the jumpy behavior with the trimmed video and wasn't able to replicate what I saw before.

When running the diagnostic test on the problematic video it printed errors for 20 frames. Ex: 'Frame 8, inconsistent RAR vs SR.'

I am going to go ahead and try shooting the videos either uncompressed or in JPEG format to see if that helps with this problem. The only reason I even noticed that something was awry was because occasionally training seemed to drastically reduce the performance of the tracker, which makes me suspect that I was actually training the landmark tracker to recognize the background rather than a part of the larva.

Hope this helps clear things up a bit! Thanks.

[allenleetc]

Cool ok. Yes it sounds like we have a consistent picture then. The diagnostic test indicates that frames read from the problematic movie in Sequential order (SR) differ from when read in random-access fashion (RAR), as might be the case when scrolling a movie forwards-then-backwards. This confirms what you observed in APT, and the diagnostic test calls VideoReader() directly (without involving any APT application code) so we know the problem exists at the level of MATLAB/VideoReader() or its dependencies.

Changing the encoding sounds good -- hopefully this resolves the issue. Let us know if you can get going!