remove shake in video by marking fixed reference points?

[mathog]

(Also posted in a slightly different form here: https://superuser.com/questions/1677463/remove-shake-in-video-by-marking-fixed-reference-points )

I have a video of a mechanical object which was made with a cell phone. Most of the parts of the device in the video are fixed, a few move (a little), and a tripod was not used so the camera itself moves a little. The device is almost flat and the camera is directly above it, but not always in exactly the same place since it was handheld. Already attempted to stabilize it using ffmpeg and vid.stab like so:

ffmpeg -i VID.mp4 -vf vidstabdetect=shakiness=10:accuracy=15 -f null -
ffmpeg -i VID.mp4 -vf vidstabtransform=input=transforms.trf VID_stable.mp4

Unfortunately this did not work very well and the image still dances around in the frame.

What I would like to do is mark, manually if necessary, an object (or objects) in the scene which are known to be immobile (there are numerous bolts, bits of the frame, and so forth, which fit this criterion) and then remove all of the camera shake by fixing the position in the image of those, for instance at their position in the first frame. I do not see how to do that with ffmpeg or vid.stab. If I am wrong please provide the commands.

Thank you.

[georgmartius]

If the camera moves only a little then you can try with shakiness=1-5. Also, use the visualization option during motion detection (show=1 or 2) to get an idea of what the system is looking at. Unfortunately, specific objects or areas are not yet supported.

[mathog]

Tried it with shakiness 1,5, and 15 and they all look pretty much the same to me. I don't know how to interpret the "show" output but there are dots in squares that seem to indicate some attempt at correction. Is there some way to filter by color? The center of the device is orange/red from rust, and the rest isn't rusty, and that rusty section has no moving parts, so if vid.stab did nothing but center it that would correct most of the motion.

[martius-lab]

I have not tried it, but you should be able to use ffmpegs filtering on the first pass (motion detection). You can apply color filtering or maybe even paint parts with a rectangle or so. Let me know if that worked.