seek functionality?

[timholy]

How feasibly would it be to implement seek functionality? If it's feasible, then it would be possible to use ImageView as a full video player, including dragging a slider bar.

Relatedly, https://github.com/kmsquire/VideoIO.jl/blob/0d6b34e0a30f30d8868c79ee141143693ee6403e/src/avio.jl#L598 assumes that decompression & display take no time. Suppose instead that these fill the entire interframe interval, then you shouldn't be sleeping at all. ImageView uses a Timer mechanism to try to keep the frame rate correct no matter what (although I think it doesn't drop frames, even if that would be needed to keep up).

[kmsquire]

Quite feasible. There are function calls for seeking available in the wrapped libraries already, so they would really just need a nicer interface.

The line you point to is a very naive way to implement timing. A timer based implementation would be a bit nicer way to go.

I've seen at least one ffmpeg tutorial which discusses dropping frames, which supposedly is needed for proper syncing of audio sometimes. Since we don't have audio support yet (#7), that probably isn't something we need to worry about yet.

I'll look into the seeking API and post back.

[lucasb-eyer]

I wanted to look into seeking, mainly following tutorial 7, but got stuck when I couldn't find av_rescale_q. (Note that while I played with it, I probably won't be able to finish it.)

[mbauman]

Here's some monkey-see-monkey-do code that works for my configuration, but I haven't submitted it as a PR because I'm uncertain how to verify that it will work in all the supported configurations and I don't really know what I'm doing.

If this looks right to you, I'll happily submit it as a pull request, or anyone can feel free to take it and run.

function Base.seek(s::VideoIO.VideoReader, time, video_stream=1)
    pCodecContext = s.pVideoCodecContext
    seek(s.avin, time, video_stream)
    VideoIO.avcodec_flush_buffers(pCodecContext)
    s
end
function Base.seek(avin::VideoIO.AVInput, time, video_stream = 1)
    # AVFormatContext
    fc = avin.apFormatContext[1]

    # Get stream information
    stream_info = avin.video_info[video_stream]
    seek_stream_index = stream_info.stream_index0
    stream = stream_info.stream
    time_base = stream_info.codec_ctx.time_base
    ticks_per_frame = stream_info.codec_ctx.ticks_per_frame

    # Seek
    ret = VideoIO.av_seek_frame(fc, seek_stream_index, convert(Int, div(time*time_base.den, time_base.num*ticks_per_frame)), VideoIO.AVSEEK_FLAG_ANY)

    ret < 0 && throw(ErrorException("Could not seek to start of stream"))

    return avin
end

[jehiah]

@mbauman I did a little digging because this seek function didn't work as expected on different video streams for me. Based on my research the correct way (which works for me on streams w/ different time bases) is below (comments included w/ references for this approach).

I'm still very new to Julia so please let me know if you see anything wrong. I'm a little unsure where this code would land in the library or i'd open a PR.

import VideoIO

# The VideoIO library is really great, but it's missing a random access seeking API.
# This should eventually be pushed upstream (https://github.com/kmsquire/VideoIO.jl/issues/30)
function Base.seek(s::VideoIO.VideoReader, time, video_stream=1)
    pCodecContext = s.pVideoCodecContext
    seek(s.avin, time, video_stream)
    VideoIO.avcodec_flush_buffers(pCodecContext)
    s
end

function av_rescale_q(a, bq::VideoIO.AVRational, cq::VideoIO.AVRational)
    b = bq.num * cq.den
    c = cq.num * bq.den
    return a * b / c
end

function Base.seek(avin::VideoIO.AVInput, time, video_stream = 1)
    # AVFormatContext
    fc = avin.apFormatContext[1]
    stream_info = avin.video_info[video_stream]

    # https://www.ffmpeg.org/doxygen/2.3/group__lavu__time.html
    seek_target = time * VideoIO.AV_TIME_BASE

    # http://dranger.com/ffmpeg/functions.html#av_rescale_q
    # seek_target= av_rescale_q(seek_target, AV_TIME_BASE_Q, pFormatCtx->streams[stream_index]->time_base);
    seek_target = floor(Int, av_rescale_q(seek_target, VideoIO.AVUtil.AV_TIME_BASE_Q, stream_info.stream.time_base))

    # pos (aka Timestamp) is in AVStream.time_base units or, if no stream is specified, in AV_TIME_BASE units.
    # https://www.ffmpeg.org/doxygen/2.5/group__lavf__decoding.html#gaa23f7619d8d4ea0857065d9979c75ac8
    # http://dranger.com/ffmpeg/functions.html#av_seek_frame
    ret = VideoIO.av_seek_frame(fc, stream_info.stream_index0, seek_target, VideoIO.AVSEEK_FLAG_ANY)

    ret < 0 && throw(ErrorException("Could not seek to position of stream"))

    return avin
end

[tlnagy]

Any update on this?

[kmsquire]

Actually, yes! Seek functionality was added very recently in #102.

The main issue right now is that VideoIO.jl doesn't work with any version of ffmpeg greater than 3.02.9. I've been working on it when I can (see the feature/fix_ffmpeg3+ branch), but still haven't gotten the incantation right for converting from YUV to RGB (after making other necessary changes). (There may still be other issues as well, although I think it's close.)

[tlnagy]

Awesome. Good to hear 👍