search

cvhciKIT / sloth

Sloth is a tool for labeling image and video data for computer vision research.
Other
608 stars 198 forks source link

Updated support for annotating video frames #87

Closed alexbw closed 7 years ago

alexbw commented 9 years ago

Using the actively maintained project moviepy (https://github.com/Zulko/moviepy), I have added support for annotating video frames, with a simple example of how to write the JSON specification for including frames. I also included moviepy as a pip-installable dependency, so users should automatically have it installed, and not need to fish for dependencies. A caveat is it requires an installed version of ffmpeg, but all modern Linux systems include this, and it is easily installable on OSX. If the maintainers of sloth felt it was required, I could write a small bit of documentation highlighting how to troubleshoot the installation of ffmpeg.

I also did not use the _video_cache object at all in my code, instead opting for a seek-and-read strategy for single frames. It performs decently well on the videos (all tens of minutes long) that I have tried. However, caching might be desirable in the future.

Please let me know any comments or tips for improvement. Thanks for the great tool, would be excellent to contribute to it.

You should also advertise! It took me forever to find out this tool exists, and it's very useful for my community.

baeuml commented 9 years ago

Thanks for the PR. I think it would be good for now for our internal users that the okapy way still worked, therefore I am leaving this unmerged for now.

Do you have any experience on how accurate and repeatable the seeking to a certain timestamp is with moviepy? We found that ffmpeg (which I understand moviepy uses under the hood) has some problems to repeatably seek to the exact same frame if the video does not have an index (i.e. if you seek to the same timestamp/frame from different locations in the video you actually end up at different frames, and sometimes with more than just a few frames offset). This would obviously be bad for labeling since you can't be sure that you actually labeled the correct frame :(

Did you notice anything in this direction so far?

alexbw commented 9 years ago

I had not heard of this bug, and have not found it in my own work.

In my code, I always seek from the beginning to the desired frame, so this should mitigate any potential problem.

Definitely understand re: migration of your users. Would be good for external folks if there was a clear way to install okapy, which I couldn't find. On Wed, Nov 12, 2014 at 3:51 AM baeuml notifications@github.com wrote:

Thanks for the PR. I think it would be good for now for our internal users that the okapy way still worked, therefore I am leaving this unmerged for now.

Do you have any experience on how accurate and repeatable the seeking to a certain timestamp is with moviepy? We found that ffmpeg (which I understand moviepy uses under the hood) has some problems to repeatably seek to the exact same frame if the video does not have an index (i.e. if you seek to the same timestamp/frame from different locations in the video you actually end up at different frames, and sometimes with more than just a few frames offset). This would obviously be bad for labeling since you can't be sure that you actually labeled the correct frame :(

Did you notice anything in this direction so far?

— Reply to this email directly or view it on GitHub https://github.com/cvhciKIT/sloth/pull/87#issuecomment-62687095.

Zulko commented 9 years ago

Hey there,

Just found this discussion through Github.

I believe MoviePy is very accurate now. See this script for details on how I checked time accuracy.

For the technical details: fetching a given timestamp with FFMPEG can be very fast and precise if you use combined seeking. Basically, you first make an unprecise seek to the nearest keyframe, then you crawl to the wanted frame. See also this blog post for the basic python implementation with Python/FFMPEG.

alexbw commented 9 years ago

@Zulko, while you're here, is there functionality through moviepy to automatically install a decent version of ffmpeg? Just maybe as an included binary? A small headache I've run into while deploying my code to end-users is they either don't have ffmpeg, or it's so old as to be unusable. Would be very useful in the context of sloth, for my use cases.

Zulko commented 9 years ago

I guess it would be possible but I am very bad at these things (maybe it will be platform-specific, too). @srid proposed to include a Docker file which I believe answers this problem, but I didn't have time to really understand how that works. If you find a nice "standard" way of solving this, I'll happily merge it.

Maybe the solution would be system-specific packaging. I have seen that there is a package for Arch-Linux, maybe we can make a package for Ubuntu/Debian, Windows, MacOS. I have no Idea how to do these.

Can you open an issue so we can continue the discussion on the MoviePy page and keep track of this request ?

nilsonholger commented 7 years ago

@alexbw Is there still interest in this to be merged. I'm actually on @baeuml's side to not break the existing functionality with OKAPI which is used internally. I'm really sorry OKAPI is not widely available. Without looking too detailed at your commits, is there any way to alternate between OKAPI and moviepy?

alexbw commented 7 years ago

I no longer use this tool, so this request can be closed if it's too much trouble.

On Fri, Aug 5, 2016 at 6:58 AM nilsonholger notifications@github.com wrote:

@alexbw https://github.com/alexbw Is there still interest in this to be merged. I'm actually on @baeuml https://github.com/baeuml's side to not break the existing functionality with OKAPI which is used internally. I'm really sorry OKAPI is not widely available. Without looking too detailed at your commits, is there any way to alternate between OKAPI and moviepy?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cvhciKIT/sloth/pull/87#issuecomment-237822605, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ4jwWzHmWAs98cFGIrf-Q1sbirCW0uks5qcxc8gaJpZM4C2cS3 .

nilsonholger commented 7 years ago

Ok, then I'm closing this, may someday somebody will pick this up as a base for improvement, so thank you for your effort. Are you using any other tool?

alexbw commented 7 years ago

No, I don't use frame-by-frame video annotation anymore for research, that phase is finished.

On Fri, Aug 5, 2016 at 7:23 AM nilsonholger notifications@github.com wrote:

Closed #87 https://github.com/cvhciKIT/sloth/pull/87.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cvhciKIT/sloth/pull/87#event-746517042, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ4j71VHmTu7VkGxCYlSQzfPoesO22Qks5qcx0cgaJpZM4C2cS3 .