Implement an efficient way to create and compress clips to upload to Zooniverse (using ALVIS)

[victor-wildlife]

I tried to upload clips from one of the GU movies to Zooniverse using tut#3 and creating the first 10 clips took ~20 minutes. Is there any way to use Alvis GPU resources or find a more efficient way to generate clips

[jannesgg]

@victor-wildlife The only way to use the GPU for FFMpeg with Alvis is to build our kso-data-management Docker image with ffmpeg from source. I can create such a version, perhaps with a special tag so that we only use it for Alvis. It will probably take 5-10 times longer to build the image than before though, since this is a major compilation job.

Another option includes looking at optimisations without building ffmpeg from source, e.g. as they do in this article https://jeromewu.github.io/speedup-ffmpeg-without-compiling-from-source-code/. In the example, they achieve a speedup of about 2x, which would bring our extraction time to 1min per clip - not optimal, but better.

What are your thoughts on this?

[victor-wildlife]

@jannesgg I think we should go ahead with building ffmpeg from source. It might pay off to ask someone from SNIC/Alvis to provide their suggestions on this as they might have encountered a similar challenge before?

[jannesgg]

@victor-wildlife I have created a branch with ffmpeg built from source (ffmpeg-cuda). It took 1h10 to build the image. I have also sent a support request to SNIC, so we can see what they suggest.

Edit: The latest build only took 10 minutes. Perhaps the connection to the ubuntu server was just slow before.

[victor-wildlife]

that's awesome @jannesgg I will try running it on my end and report back

[victor-wildlife]

@jannesgg How can I run the ffmpeg-cuda branch in Alvis? jupyter1.sh get the latest docker image from the main branch right?