Could FFmpegInteropX play back image sequences?

[jjouppi]

Hey all!

Thanks once again for this project. Was wondering if it would be possible to play back an image sequence using FFMpegInteropX? Ideally I'd like to do so without rendering out the sequence as a video file beforehand.

Right now I have image sequences working in my UWP project using Windows.Media.Editing;. I pass a Media Composition to the source of the MediaPlayerElement I use to control FFmpeg playback:

private async void OpenImageSequence(object sender, RoutedEventArgs e)
{
    files = null;

    FileOpenPicker filePicker = new FileOpenPicker();
    filePicker.ViewMode = PickerViewMode.List;
    filePicker.FileTypeFilter.Add(".png");
    filePicker.FileTypeFilter.Add(".jpg");
    filePicker.FileTypeFilter.Add(".tif");

    files = await filePicker.PickMultipleFilesAsync();

    foreach (StorageFile file in files)
        {
            var clip = await MediaClip.CreateFromImageFileAsync(file, TimeSpan.FromMilliseconds(1000 / mediaFrameRate));
            composition.Clips.Add(clip);
        }

    MediaStreamSource newImageSequence = composition.GenerateMediaStreamSource(defaultProfile);

    windowsSource = MediaSource.CreateFromMediaStreamSource(newImageSequence);

    mediaElement.MediaPlayer.Source = windowsSource;

}

This works great for TIFF, PNG, and JPEG files but I need to be able to playback EXR, TGA, and PSD files as well. I have it working where I convert the images from EXR/TGA/PSD to PNG using ImageMagick before compiling them into a sequence but this is a prohibitively slow process.

Would it be possible to achieve image sequence playback with FFmepgInteropX? Based upon what I've found it seems that FFmpeg can compile a decoded video from a sequence of images and also has support for EXR, PSD, and TGA images. If this currently isn't possible but could be added to FFMpegInteropX I'd love to work on it but would just need to be pointed in the right direction as I'm unsure of where to start.

Thanks!

More documentation: https://trac.ffmpeg.org/wiki/Slideshow http://www.ffmpeg.org/ffmpeg-formats.html#image2-1 http://www.ffmpeg.org/ffmpeg-formats.html#image2-2

[brabebhin]

Hi

You could start by using the samples and open an image file and see how it goes around demuxing the streams.

We currently do have a case for handling image streams, but we basically assume they are album art streams in music files. You can take a look at that.

[lukasf]

Nope, actually, image data in image files is not treated like album art, but like a normal video stream in a video file. So when you open a JPEG file with FFmpegInteropX normally (like you'd open a video file), it will work, it will have one video stream, and when you play it, it will display that one frame it has. So this all works basically. What we'd have to add on top is decoding and returning these image frames in a sequence.

One thing I noticed: FFmpeg does not report EXIF rotation like it does for video files. So files with image rotation are currently displayed incorrectly. But that should be easy to fix.

[lukasf]

Oh, and before doing work here, please verify that the current FFmpegInteropX can decode the image formats you need. Sometimes, a format will require special libs to be linked with FFmpeg. We never really focused on image files, so you should first verify that the formats can be decoded at all, by opening some files with the sample app.

[brabebhin]

IIRC album art was a video stream that had a very specific attribute that designated it as album art, and this was basically the only instance in which we used an "image stream". That's what I meant :)

I would assume that this is similar with other image files, so they would be treated like a video stream with one frame but will have an attribute somewhere denoting it as an image stream.

[lukasf]

What I tried to say is that when you open a real image, it will be handled differently than album art in a video or audio file. Its video stream also does not have that flag, so FFmpegInteropX will treat it like a normal video file. So if you want to decode image files, you have to look at how video file decoding works, not how album art decoding works.

[lukasf]

I have built a simple implementation of an ImageListSource. Check branch "image-list". Run the C# sample, press "I" on your keyboard, select a bunch of images (currently all must have same size/orientation). It will play, but if resolution is high, playback won't be very fast, because decoding is single threaded. I get about 6fps on 24MP high quality jpeg files. Smaller files will be considerably faster. Larger files or more complex formats will be even slower.

I don't know which performance requirements you have. Decoding larger images is always time consuming. We could speedup things by decoding multiple images at once, this would give you higher speed, based on numbers of CPU cores.

At least on newer Windows versions, it should be possible to support different resolutions.