Weird Skipping in Video

[GrahamboJangles]

I'm using Colab to run this.

Whenever I use the video completion, the video output is very skippy. A couple of frames will play nicely, but then the video skips to a couple frames back and then quickly skips back. I can tell the video is being extrapolated, but this skipping makes playback not smooth.

[gaochen315]

Can you first check if the output images are in the right order?

[jelenaaaaa]

Can you check in what order sorted function is returning your images?

[GrahamboJangles]

Can you check in what order sorted function is returning your images?

Yup you guys might be right, I will invest investigate that and report back.

[GrahamboJangles]

@gaochen315 @jelenaaaaa - So the frames are in the right order when I open each jpeg in numerical order. But in Colab, it looks like this:

I don't know if this is the problem though, because no matter how the files are arranged on screen they are still in order.

[jelenaaaaa]

@gaochen315 @jelenaaaaa - So the frames are in the right order when I open each jpeg in numerical order. But in Colab, it looks like this:

I don't know if this is the problem though, because no matter how the files are arranged on screen they are still in order.

Try to use natsorted instead of sorted function (you can install natsort from pip and then from natsort import natsorted)

[GrahamboJangles]

@jelenaaaaa

I sorted used natsort, sorry for the noob question but then am I supposed to shutil.move() according to the sorted list of items?

I zipped up the files in Colab and downloaded them onto my Windows PC, and they were ordered correctly. Then I put them into Colab again, unzipped it, and they are in the wrong order again. I think it's just the way that Colab shows the order of files, but as long as the files are numerically in order it shouldn't be a problem.

Is this a Colab problem?

[jelenaaaaa]

@jelenaaaaa

I sorted used natsort, sorry for the noob question but then am I supposed to shutil.move() according to the sorted list of items?

I zipped up the files in Colab and downloaded them onto my Windows PC, and they were ordered correctly. Then I put them into Colab again, unzipped it, and they are in the wrong order again. I think it's just the way that Colab shows the order of files, but as long as the files are numerically in order it shouldn't be a problem.

Is this a Colab problem?

I had a similar problem myself and I just used natsorted instead of sorted (I changed it everywhere) and It worked just fine. I did not use shutil.move(). The problem was with reading images and masks, if images are not appended in video in the right order (and also the corresponding masks) the final video will look like it's skipping frames. I did not use Colab so I really don't know if the problem is with that or something else.

[GrahamboJangles]

@jelenaaaaa Where did you natsort, exactly? It just seems to make an ordered list, I'm not sure what to do with the list.

[jelenaaaaa]

@GrahamboJangles wherever you have sorted function in the code just replace it with natsorted. It does make an ordered list and the rest of the code will just use a list that is sorted in that way, you don't have to do anything else with it. For example, here (lines 395,396):

for filename in sorted(filename_list): video.append(torch.from_numpy(np.array(Image.open(filename)).astype(np.uint8)).permute(2, 0, 1).float())

This loads your files from a list and appends them to a video. If the sorted function doesn't sort images as it should, it will append it to a video in a irregular way so your video final video will skip (because the rest of the code will use images that were not sorted correctly). Try to print filename_list when you use sorted and also when you use natsorted, if you have the same issue that I had the problem will be obvious. Also, if the masks are loaded irregularly you will have a similar problem.

[gaochen315]

A naive solution is to zero pad the filename. For example, 0.png --> 000.png. You can do str(idx).zfill(3) + '.png'.

[GrahamboJangles]

@jelenaaaaa I replaced the instances in the code of sorted(filename_list) with natsorted(filename_list) and it seemed to help a little bit but I still had some skipping for some reason. Maybe I missed a line. But then I implemented what @gaochen315 said with my frame extracting code and the padding of the filenames fixed the issue. Thank you guys for your help!