Trying to run Inference on new video, but no instances found in any frames

[Idavr]

Hi there!

I've gotten a model really (really!) trained and wanted to test it on a video not in the training set. However, when I run Inference on the entire video, I get this output (this is the last snippet of the code as it repeats for - I assume - every frame):

0 successful operations. 0 derived errors ignored. INFO:sleap.nn.inference: 2020-09-27 14:18:39.923791: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at iterator_ops.cc:941 : Invalid argument: 2 root error(s) found. (0) Invalid argument: {{function_node inference_Dataset_map_predict_1560}} ConcatOp : Dimensions of inputs should match: shape[0] = [1,24,30,128] vs. shape[1] = [1,30,40,256] [[{{node model/stack0_dec0_s16_to_s8_skip_concat/concat}}]] (1) Invalid argument: {{function_node inference_Dataset_map_predict_1560}} ConcatOp : Dimensions of inputs should match: shape[0] = [1,24,30,128] vs. shape[1] = [1,30,40,256] [[{{node model/stack0_dec0_s16_to_s8_skip_concat/concat}}]] [[model/stack0_dec2_s4_to_s2_refine_conv1/Conv2D/ReadVariableOp/_25]] 0 successful operations. 0 derived errors ignored. INFO:sleap.nn.inference:ERROR in sample index 53354 INFO:sleap.nn.inference:2 root error(s) found. (0) Invalid argument: {{function_node inference_Dataset_map_predict_1560}} ConcatOp : Dimensions of inputs should match: shape[0] = [1,24,30,128] vs. shape[1] = [1,30,40,256] [[{{node model/stack0_dec0_s16_to_s8_skip_concat/concat}}]] (1) Invalid argument: {{function_node inference_Dataset_map_predict_1560}} ConcatOp : Dimensions of inputs should match: shape[0] = [1,24,30,128] vs. shape[1] = [1,30,40,256] [[{{node model/stack0_dec0_s16_to_s8_skip_concat/concat}}]] [[model/stack0_dec2_s4_to_s2_refine_conv1/Conv2D/ReadVariableOp/_25]] 0 successful operations. 0 derived errors ignored. INFO:sleap.nn.inference: INFO:sleap.nn.inference:Finished 53355 examples in 1005.22 seconds (inference + postprocessing) INFO:sleap.nn.inference:examples/s = 53.0781825900967 INFO:sleap.nn.inference:Predicted 0 labeled frames in 1009.379 secs [0.0 FPS] Saving: C:/Users/idavalik\predictions\Obs30_DroCol1.mp4.200927_140142.predictions.slp Total Time: 1009.3928165435791 Process return code: 0

And the message I get in the GUI is that inference were run on all frame but inference were run on 0 frames since no instances were found in any of them. I thought this might have been due to the video being unseekable, so I converted it with ffmpeg but still get the same errors.

Hope you can help me with this, so excited to see how the model might work on the rest of my videos.

Best regards,

Ida

[talmo]

Hi @Idavr,

It looks like the videos were of different sizes so it's failing when trying to apply the model.

This should be fixed automatically in most cases, but for now the easiest might be to attempt to increase the padding. You can do this by editing the training_config.json in the model folder and setting the pad_to_stride setting to 32. This will be under:

"data": {
    ...
    "preprocessing": {
        ...
        "pad_to_stride": 32

If that doesn't work, let me know what sizes your original and new videos are at and we'll dig into it.

Cheers,

Talmo

[Idavr]

Hi @talmo,

Wow, this is weird. I wasn't aware that the videos were of different dimensions as I've used the exact same camera and set-up for all videos, so I am quite confused as to how this might have happened. Unfortunately, adjusting the "pad_to_stride" did not work on either training_config's...

The videos the model has been trained on is 640x480, and the new (?) videos are 480x360.

Cheers,

Ida

[talmo]

I see -- are these videos smaller due to cropping or are they scaled down as well?

[Idavr]

Neither on purpose. I’ve used the exact same camera set-up that saves the files in .h264 format, then I convert them to .mp4 with VLC. Done the same thing with the videos the model has been trained on.

29. sep. 2020 kl. 22:39 skrev Talmo Pereira notifications@github.com:



I see -- are these videos smaller due to cropping or are they scaled down as well?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/murthylab/sleap/issues/407#issuecomment-700973375, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALMPW35L6ABDF5NM32YOYYDSIJAX3ANCNFSM4R3UQBJA.

[Idavr]

Alright, so, I went back and messed with the conversion of the videos to get the correct dimensions and now the inference runs without problems :) There has been a change in both frame rate and size of the newer original videos compared to the older ones (that have been used for training), but I have no idea what has happened and why... Regardless, I've found a workaround!

Side-note: The model definitely needs to be trained more after the results from this video, so if following the instructions on sleap.ai then correcting the entire video and then merging the corrected predictions into the original project is the way to go, right? Or would it be better to add the video(s) I want to use SLEAP on into the same project as the model has been trained on and then just run another training session with all the videos together?

[talmo]

Hi @Idavr,

Great! Weird that the dimensions and framerate changed... If you're using FLIR/Point Grey cameras this can sometimes happen when the USB bus bandwidth (maybe called packet size) is reduced in the camera settings, so it limits the amount of data the camera can acquire per second. It's happened to us a couple of times before for some reason.

Regarding further training: you definitely don't need to correct the entire video -- just a few frames that appear to be doing poorly should be a good start. You can merge the predictions into the original project but only the labeled frames will be used for training. And yes, you'll want to train with all of the labels from all of the videos.

Going to close this issue for now but feel free to re-open or create a new one if you're still having issues!

[Idavr]

Hi @talmo,

Hmm... Alright, thank you, I'll keep that in mind and take a look to see if something like this has happened with our camera. We are using a Raspberry Pi camera, and I do have a suspicion as to where this reduction might have occurred if that is the case. Thanks for the hint, it is at least a start as to where to look :)

And thanks once again for your swift help!