Seeking with FFmpegFrameGrabber.setTimeStamp(long timestamp)

[iexavl]

So I am pulling a video from a network stream and am making something of a player. Thus I want to implement seeking. I noticed though that seeking for some reason happened quite slowly and I noticed why when I looked through the code. So as far as I understand when avformat_seek_file is called with AVSEEK_FLAG_BACKWARD it seeks to the nearest key frame (so called I frames in the case of the video stream) and than decodes up to the desired frame. That makes sense but not in my case since I can just seek to the exact point where the frame I need is via opening a new connection and putting a header that specifies at exactly what byte the server needs to start sending me data. Anyway, from what I understand I do still need the key frame for the decoder otherwise it doesn't really know what it's supposed to do. Thus I can just seek once with AVSEEK_FLAG_BACKWARD, give the decoder the keyframe and than seek again with AVSEEK_FLAG_ANY to get the actual frame I need instead of decoding every frame up until the frame I need. (I have switched out the seekCallback with my own so this works). So good so far except it doesn't work. I checked things out with the debugger and noticed that when I grab a frame right after a call to avformat_seek_file with AVSEEK_FLAG_BACKWARD I am not actually getting a keyframe the next frame I read. This I think could be a reason why this isn't working, is it supposed to be like that?

edit: forgot to mention (as far as I understand) an avformat_seek_file with AVSEEK_FLAG_ANY gives me any frame regardless of whether it's a keyframe or not. Please correct me if I am wrong in any of this.

[anotherche]

You need more than just a keyframe to decode an arbitrary next frame. You need ALL the frames since the previous keyframe to decode it. That's why avformat_seek_file normally called with AVSEEK_FLAG_BACKWARD. See, for example, the conversation on a similar question. You will also find there that sometimes calling avformat_seek_file with AVSEEK_FLAG_BACKWARD returns not the previous keyframe but the next one. This is "by design" in the underlying ffmpeg code. That's why we use a workaround for this case - we seek to a more distant keyframe (0.5 sec earlier).

                /*  Sometimes the ffmpeg's avformat_seek_file(...) function brings us not to a position before
                 *  the desired but few frames after. In case we need a frame-precision seek we may
                 *  try to request an earlier timestamp.
                 */
                early_ts -= 500000L;

All we can do is to change the seek code so that we don't always add this half-second ahead, but first check that the correct (the previous one) keyframe has been received, and only ask for an earlier keyframe otherwise. But in my opinion, this will not give a noticeable difference in speed, since in most cases we will get the nearest of the previous keyframes, since they do not occur more than once a second normally.

[iexavl]

ll we can do is to change the seek code so that we don't always add this half-second ahead, but first check that the correct (the previous one) keyframe has been received, and only ask for an earlier keyframe otherwise. But in my opinion, this will not give a noticeable difference in speed, since in most cases we will get the nearest of the previous keyframes, since they do not occur more than once a second normally.

Yeah, I realized that I need all the frames since the key frame shortly after I shut down my pc that night. After all every frame that isn't a keyframe is derived from the frame before it so I don't know what I was on thinking that would work. Also about the keyframes being normally one second apart I do quite often run into situations where that isn't the case. Sometimes when I seek I see that the timestamp is set like 4 or 5 seconds before the timestamp I desire which isn't a problem if you are reading from a file considering how quick today's computers and phones are but when it comes to internet speed this pretty insignificant otherwise difference turns into 4-5 seconds of waiting to decode packets up to the desired timestamp. Something good came out of this though, namely the thing you mentioned with requesting an earlier timestamp. As I don't need super high precision frame seeking I will probably remove that and it might help a bit. Thanks!

[iexavl]

By the way another thing I was wondering, I built my program and it's like a gigabyte and something. The reason for that is mostly because of the libraries for just about every single operating system and every single processor architecture in existence. Is there a way to exclude them from the build as I don't need most of them?

[anotherche]

By the way another thing I was wondering, I built my program and it's like a gigabyte and something. ...

Yes, you can specify needed platform (e.g. javacpp.platform=linux-x86_64) in the build options

[iexavl]

@anotherche alright , thanks. Also another thing is I think the decoder is using a bit too much of my cpu. Is there a way to enable some kind of hardware acceleration ? edit : I couldn't do it at the time but I checked the thing with the platforms, where am I supposed to put that? I have these put in my pom.xml

<dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>javacv-platform</artifactId>

            <version>1.5.8</version>
        </dependency>

        <!-- Additional dependencies required to use CUDA and cuDNN -->
        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>opencv-platform-gpu</artifactId>
            <version>4.6.0-1.5.8</version>
        </dependency>

        <!-- Optional GPL builds with (almost) everything enabled -->
        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>ffmpeg-platform-gpl</artifactId>
            <version>5.1.2-1.5.8</version>
        </dependency>

but it doesn't really let me specify a version

[anotherche]

1. As far as I know, ffmpeg uses hw acceleration automatically, if supported by system/decoder.

2. javacpp.platform=... is not specified in pom, it should be a build option "-Djavacpp.platform=..." for the tool you are using (mvn, javac, IDE)

[saudet]

For hardware acceleration, it doesn't do it automatically, no. For more information, see https://trac.ffmpeg.org/wiki/HWAccelIntro

[iexavl]

@anotherche I'm not sure how to not specify it in pom because I am using intelliJ with maven build system but this looks like a VM argument and I tried passing it to maven via this:

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.11.0</version>
                <configuration>
                    <compilerArgs>
                        <arg>-Djavacpp.platform=windows-x86_64</arg>
                    </compilerArgs>
                    <source>20</source>
                    <target>20</target>
                </configuration>
            </plugin>

more specifically the compilerArgs tag. It gave me this : java: error: invalid flag: -Djavacpp.platform=windows-x86_64

[iexavl]

For hardware acceleration, it doesn't do it automatically, no. For more information, see https://trac.ffmpeg.org/wiki/HWAccelIntro

I will check it out

[anotherche]

For hardware acceleration, it doesn't do it automatically, no. For more information, see https://trac.ffmpeg.org/wiki/HWAccelIntro

OK, you are right. I just confused it with automatic switch to software mode when hw acceleration is not supported.. BTW here is what ffmpeg documentation says about hw acceleration "advantages": "Note that most acceleration methods are intended for playback and will not be faster than software decoding on modern CPUs. In addition, ffmpeg will usually need to copy the decoded frames from the GPU memory into the system memory, resulting in further performance loss. This option is thus mainly useful for testing." So, it is better to find out the reason why the performance is insufficient in your case. In what conditions high decoding CPU load is observed?

I'm not sure how to not specify it in pom because I am using intelliJ with maven build system

I've also never been able to specify the platform in pom.xml (I tried many options, none worked). In IDEs, the platform is specified in the project build/run settings. For example, in Eclipse it is the Parameter/Value settings in "Run Configurations" dialogue. And there is similar way in IntelliJ

[iexavl]

For hardware acceleration, it doesn't do it automatically, no. For more information, see https://trac.ffmpeg.org/wiki/HWAccelIntro

OK, you are right. I just confused it with automatic switch to software mode when hw acceleration is not supported.. BTW here is what ffmpeg documentation says about hw acceleration "advantages": "Note that most acceleration methods are intended for playback and will not be faster than software decoding on modern CPUs. In addition, ffmpeg will usually need to copy the decoded frames from the GPU memory into the system memory, resulting in further performance loss. This option is thus mainly useful for testing." So, it is better to find out the reason why the performance is insufficient in your case. In what conditions high decoding CPU load is observed?

I'm not sure how to not specify it in pom because I am using intelliJ with maven build system

I've also never been able to specify the platform in pom.xml (I tried many options, none worked). In IDEs, the platform is specified in the project build/run settings. For example, in Eclipse it is the Parameter/Value settings in "Run Configurations" dialogue. And there is similar way in IntelliJ

okay so I checked out the hardware acceleration and opted out of using it considering what my application will be doing most of the time. Although I am still a little bit suspicious of the CPU usage it isn't that detrimental. Second off yeah I didn't really find a way to specify it in the pom.xml either but I used the maven shade plugin in my pom.xml for maven to package all my dependencies along with my jar and ran mvn -Djavacpp.platform=windows-x86_64 clean install and it worked. I honestly am not all too sure why it didn't work with compilerargs but in any case that's what I did and in my case it worked. Went down from 1.2 gigabytes to 130 megabytes, pretty sweet. Also the run configurations is a thing, yes but its only for running the program in the IDE, not building it.

[iexavl]

Anyway, with this the issues are solved. Thanks a lot for the help!