Cannot play ffmpeg in JavaFX MediaPlayer

[Adi-18]

What I have to set for recording video stream. I can play result in VLC but not in JavaFX MediaPlayer. There is no exception just no playing. Code:

        recorder = new FFmpegFrameRecorder(file, captureWidth, captureHeight);
        recorder.setFormat("mp4");
        recorder.setVideoCodecName("libx264");
        recorder.setAudioChannels(0);

I dried with different options:

        recorder = new FFmpegFrameRecorder(file, captureWidth, captureHeight);
        recorder.setVideoBitrate(10 * captureWidth * captureHeight);
        recorder.setFormat("mp4");
        recorder.setVideoCodecName("libx264");
        recorder.setVideoOption("crf", "23");
        recorder.setPixelFormat(avutil.AV_PIX_FMT_YUV420P);
        recorder.setFrameRate(30);
        recorder.setGopSize(60);
        recorder.setAudioChannels(0);

[saudet]

@johanvos @kevinrushforth Would you guys know exactly what JavaFX supports and doesn't support from the H.264 standard?

[Adi-18]

I can limit the problem to one point: In my try above , I received video stream with resolution 1600bx x 1200bx. When I change camera resolution (or FFmpegFrameRecorder setting) to 1280px x 960px, it works.

I attempted to use 1920px x 1080px to have compare with an Video-Movie in that resolution. I see javacv coded movie starts with big delay, Video-Movie starts immediately. What are the best settings to have the same result?

Codec information, javacv coded movie:

General
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
Overall bit rate                         : 2 405 kb/s
Writing application                      : Lavf58.20.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings, CABAC                   : Ja
Format settings, ReFrames                : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Bit rate                                 : 2 402 kb/s
Width                                    : 1 920 Pixel
Height                                   : 1 080 Pixel

Codec information, video movie:

General
Format                                   : MPEG-4
Format profile                           : Base Media / Version 2
Codec ID                                 : mp42 (isom/mp42)
Overall bit rate mode                    : variabel
Overall bit rate                         : 2 661 kb/s

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings, CABAC                   : Ja
Format settings, ReFrames                : 1 frame
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Bit rate                                 : 2 466 kb/s
Maximum bit rate                         : 10,6 Mb/s
Width                                    : 1 920 Pixel
Height                                   : 1 080 Pixel

[saudet]

The "big delay" occurs only with JavaFX, right? It's working fine with VLC, correct? If so, that's something related to JavaFX. If we don't hear from @johanvos or @kevinrushforth here, you'll probably get more help about this over there: https://github.com/javafxports/openjdk-jfx/issues

[johanvos]

Can you attach the JavaFX code where you see this delay?

[Adi-18]

The code is easy. From element click event i start load file and play video (delay from event to play 6 sec):

            String f = new File(file).toURI().toString();
            player = new MediaPlayer(new Media(f));
            player.setAutoPlay(true);
            mediaView.setMediaPlayer(player);

I also have an opencv recorded file. Those also starts immediately (there is another known bug with blocking if streams goes lost). Codec information, opencv coded movie:

General
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 25.0 MiB
Duration                                 : 26 s 400 ms
Overall bit rate                         : 7 945 kb/s
Writing application                      : Lavf57.83.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Baseline@L4.1
Format settings, CABAC                   : No
Format settings, ReFrames                : 1 frame
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 26 s 400 ms
Bit rate                                 : 7 943 kb/s
Width                                    : 1 280 pixels
Height                                   : 960 pixels

[saudet]

@Adi-18 If you could attach a sample failing video file, I'm sure the JavaFX developers would appreciate.

[Adi-18]

I have attached a zip file contained an 11 sec recording.

cap_UHBEO_Int_DOM-Cam1549482463694.zip

[saudet]

That video starts at 37 seconds, and although VLC skips that portion, another player like JavaFX can very well choose to wait 37 seconds:

$ ffprobe cap_UHBEO_Int_DOM-Cam1549482463694.mp4 
...
  Duration: 00:00:11.23, start: 37.733008, bitrate: 1407 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x1280, 1403 kb/s, 28.04 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler

[Adi-18]

Ok I see... But Why the movie starts on that time (btw, JavaFX don't starts with that gap)? Could it be somethings with timestamp?

Edit: I tried a new record for exactly 10 sec. The result starts at 5 seconds in VLC with a duration of 10 seconds (with 5 seconds overtime = total 10 seconds).

[Adi-18]

My capture code (timestamp correction found in internet):

                while (live && (capturedFrame = grabber.grabImage()) != null) {
                    Image image = jconverter.convert(capturedFrame);
                    Platform.runLater(() -> {
                        imageView.setImage(image);
                    });

                    // Let's define our start time...
                    // This needs to be initialized as close to when we'll use it as
                    // possible,
                    // as the delta from assignment to computed time could be too high
                    if (startTime == 0) {
                        startTime = System.currentTimeMillis();
                    }

                    // Create timestamp for this frame
                    videoTS = 1000 * (System.currentTimeMillis() - startTime);

                    if (doRecord) { // Check for AV drift
                        if (videoTS > recorder.getTimestamp()) {
                            System.out.println(
                                    "Lip-flap correction: "
                                    + videoTS + " : "
                                    + recorder.getTimestamp() + " -> "
                                    + (videoTS - recorder.getTimestamp()));

                            // We tell the recorder to write this frame at this timestamp
                            recorder.setTimestamp(videoTS);
                        }

                        // Send the frame to the org.bytedeco.javacv.FFmpegFrameRecorder
                        recorder.record(capturedFrame);
                    }
                }
                stopRecord();

[Adi-18]

Ok I removed the part with recorder.setTimestamp(videoTS); and it works as expected. Is the correction needed or why do I found this in many examples? An if needed (I think so because of many lip-flap corrections), what is the correct way?

[saudet]

We can put whatever timestamp we want! If we don't set them, it will put the first frame at 0, and increment it according to the frame rate for subsequent frames, that's all.

[Adi-18]

Ah, this is the the information that solved my problem. I set startTime for timestamp before I start recording. Now I moved this after if (doRecord) and it works.

[saudet]

Great! I'll keep this issue open as an "enhancement" for the documentation. Contributions welcome!

[johanvos]

So just for my understanding, there is no JavaFX issue here that needs to be solved?

[Adi-18]

@saudet Thank you for the quick response. @johanvos: maybe the question of "delayed" or no start if the timestamp starts higher then 0? Probably if you can compensate the time start at 0 internally. On other hand, if its only my mistake, then its not an highlevel issue.

[kevinrushforth]

I guess if the JavaFX media player is the only player that doesn't treat the initial time stamp as the "0" point, then we could consider changing this behavior, although then there might be compatibility issues to consider if some apps are relying on this to effect a delay in starting.

[teocci]

@Adi-18 can you share the code that solved your issue?? so another people can be referenced here.

[Adi-18]

While capturing the stream, i check if recording is required.

       if (doRecord) {
               // code for recording
       {

but I set the startTime = System.currentTimeMillis() (for AV drift correction) before independently. This war a failure. When I started recording after I started capturing 2 minutes later, I had a time shift from this value (delayed start).

So I must move

               if (startTime == 0) {
                   startTime = System.currentTimeMillis();
               }

inside: if (doRecord) { if (startTime == 0) startTime = System.currentTimeMillis(); // code for recording {

[saudet]

I see, so you're probably just not encoding fast enough and losing packets. You'll need to find a way to encode faster on your machine.

[Adi-18]

No I'm fast enough. This works for me.

[Adi-18]

Thank you for reopen this issue. In this issue, I didn't realized there are two problems in one file. So I attached a short file with high resolution (2592 x 1520, strange but I get this from rtsp camera). Timestamp should be correct now.

maz_UHBEO_IntDOM-Cam(neu)1563814288483.zip

[saudet]

And you're saying JavaFX can't play that file? It wasn't created with JavaCV, so the issue is with JavaFX, and should be reported over there.