Embedding SDL window to java swing or javafx

[tahaarian]

hello sir

The ffmpeg plugin is a great solution for java. Actually we have used your sdl player with java solution and it is great. As i have detected that you use SDL Window for playing it in C++ project. we need to embedded the sdl player in javafx or swing java based window. Also according to our unit test, the java player has lagged for udp stream and it had lip sync problem totally

The main goal is playing video with sdl lib in java swing or javafx windows. We will appreciate if you could help us.

Thank you in advance

Taha Arian

[TheoWolf]

Hello @tahaarian

Thank you for your interest in our FFmpeg plugin.

The goal of the FFmpeg plugin is to provide a fast and accurate player for Java, our first initiative was to decode frames on the Native (C++) side and send byte buffers to swing/JavaFx to be rendered, and we faced the issues that you mentioned (UDP stream and lip sync)

The lip synchronization problem is fixed in version 0.21 and it was not related to Java, whatsoever. On the other hand, the slow udp upload is, Java AWT rendering is slow which causes the player to skip frames. The only way to have a fast rendering is to use/control the native SDL window from Java, this approach has pros and cons but works.

We tried embedding SDL in a Java Swing/JavaFX window but didn't suit our needs especially on Mac OS (Crashes, lags, resizing issues...), this could be done through this https://wiki.libsdl.org/SDL_CreateWindowFrom, all that you will need to do is to get the Java window handle and create JNI bridge to create a window with the Java Handle (our plugin doesn't super this feature).

We are releasing an SDL/FFmpeg player soon that could be used and controller via Java on both Windows and Mac OS, providing good performance and frame accuracy.

If you or your team feel that you can contribute to this project, you are more than welcome to fork and submit changes. we really appreciate any feedback or help.

Let me know if you have any other questions.

Best Reda

[tahaarian]

Hello again sir

Thank you for your quick and great answer.

Its useful so we will work on it that we could create a java window handle and communicate with your C++ solution. I will appreciate if you could send me any helpful links or content.

Actually i have found some issues on calling pause , stop and some other api that you have used. It was related to catching "ref_media" when we call from java and jni. Below part i mean for example:

` dlMediaPlayer_ffmpegSetWindowSize (JNIEnv env, jobject obj, jlong ref_media, jint jiWidth, jint jiHeight) { CMedia pMedia = (CMedia *)jlong_to_ptr(ref_media); if (NULL == pMedia) return ERROR_MEDIA_NULL;

CPipeline *pPipeline = (CPipeline *)pMedia->GetPipeline();`

it could not found appropriate instance of CMedia and got ERROR_MEDIA_NULL. So i have created a global CMedia param and initialize it on your init function and then use it all over (maybe its wrong solution but it works) I have found 3 other issues that i will fork your solution and commit. One of the important issues was related to playing udp stream from java solution that i have fixed with removing extra thread in C++ which was created when you call from java

we will appreciate if you could help us with sdl based java Swing/JavaFX player which it can handle window from java source

Best Regards,

Taha Arian

[tahaarian]

Hello again

I have created an JFrame and pass to Init function as below(For implementing java handle and sending to JNI) but i have got EXCEPTION_ACCESS_VIOLATION. Actually we need to load sdl player in JFrame May i ask you to check these codes.

// SimpleSDlFxMediaPlayer.java

  windowPlayer = new JFrame();

  windowPlayer.setMinimumSize(new Dimension(1000, 800));

  windowPlayer.setLocationRelativeTo(null);

  windowPlayer.setVisible(true);

  mediaPlayer.init(windowPlayer);

// FfmpegSdlMediaPlayer.java

//// Native methods

protected native int ffmpegInitPlayer(long[] newNativeMedia, String sourcePath,JFrame window);

// FfmpegSdlMediaPlayer.cpp

JNIEXPORT jint JNICALL Java_org_datavyu_plugins_ffmpeg_FfmpegSdlMediaPlayer_ffmpegInitPlayer( JNIEnv *env, jobject obj, jlongArray jlMediaHandle, jstring sourcePath, jobject jWindow) {

CPipelineOptions *pOptions = new (nothrow) CPipelineOptions(); if (NULL == pOptions) { return ERROR_SYSTEM_ENOMEM; }

CPipeline *pPipeline = new (std::nothrow) FfmpegSdlAvPlaybackPipeline(pOptions);

if (NULL == pPipeline) { return ERROR_PIPELINE_NULL; }

pMedia = NULL; CMedia **ppMedia = &pMedia;

*ppMedia = new (nothrow) CMedia(pPipeline);

if (NULL == *ppMedia) { delete pOptions; delete pPipeline; return ERROR_MEDIA_CREATION; }

if (CMedia::IsValid(pMedia)) { jlong lMediaHandle = (jlong)ptr_to_jlong(pMedia); env->SetLongArrayRegion(jlMediaHandle, 0, 1, &lMediaHandle); } else { delete pOptions; delete pPipeline; return ERROR_MEDIA_INVALID; }

CJavaPlayerEventDispatcher *pEventDispatcher = new (nothrow) CJavaPlayerEventDispatcher(); if (NULL == pEventDispatcher) return ERROR_SYSTEM_ENOMEM;

pEventDispatcher->Init(env, jWindow, pMedia); pPipeline->SetEventDispatcher(pEventDispatcher);

const char *filename = env->GetStringUTFChars(sourcePath, 0);

jint iRet = (jint)pPipeline->Init(filename);

env->ReleaseStringUTFChars(sourcePath, filename);

return iRet; }

Then i have got EXCEPTION_ACCESS_VIOLATION when i have debugged java/cpp solution. It was crashed at "CJavaPlayerEventDispatcher::Init" function and below line :

send_player_media_error_event_method_ =
    env->GetMethodID(klass, "sendPlayerMediaErrorEvent", "(I)V");
hasException = javaEnv.ReportException();

Whats my wrong point ? is it correct for java handling and related stuffs ?!

I have attached error log.

We will appreciate if you could help us.

Best Regards, Taha Arian hs_err_pid13120.log

[majidbagherzadeh]

Hello sir

Such a great thread...

we need to develop a java player based on sdl_ffmpeg with jframe style which could handle sdl functions in java code.

any solution ?

Best

M Saadatmand

[TheoWolf]

@tahaarian I will need more details on which plugin version you are using and on which platform you are having issues? Note that the Readme is outdated and the latest version is 0.21

We have three JNI bridges: Java, SDL and MPV if you need to see a working example on how to send a JFrame window handle to the native side look at the MPV bridge. passing the JFrame Object like you have in your code won't work, you will need to extract the reference of that JFrame in the Java world and pass it to the native side (extracting this reference is different for Swing and JavaFX).

Code to get the window handle

  public static long getHWnd(Object obj) throws InvocationTargetException, IllegalAccessException {
    if (obj instanceof Container) {
      return getContainerHWnd((Container) obj);
    }
    if (obj instanceof Stage) {
      return getStageHWnd((Stage) obj);
    }
    return 0;
  }

  static long getContainerHWnd(Container container)
      throws InvocationTargetException, IllegalAccessException, IllegalArgumentException {

    if (!container.isVisible()) {
      container.setVisible(true);
    }
    // The reflection code below does the same as this
    // long handle = container.getPeer() != null ? ((WComponentPeer) container.getPeer()).getHWnd()
    // : 0;

    Long hwnd = Long.valueOf(0);
    if (isWindows) {
      Object wComponentPeer = invokeMethod(container, "getPeer");
      hwnd = (Long) invokeMethod(wComponentPeer, "getHWnd");
    } else if (isMac) {
      throw new NotImplementedException();
    }
    if (hwnd == 0) {
      new IllegalArgumentException("Unable to retrieve Container Handler");
    }
    return hwnd;
  }

  /**
   * Get the window id for a stage
   *
   * <p>This seems only to work on windows
   *
   * <p>Code adopted from https://github.com/java-native-access/jna/issues/706
   *
   * @param stage The stage
   * @return The window Hwnd
   * @throws RuntimeException if the window id can't be retrieved
   */
  static long getStageHWnd(Stage stage) {
    Class<?> stageClazz = stage.getClass();
    try {
      Method tkStageGetter =
          JAVA_VERSION == 9
              ? stageClazz.getSuperclass().getDeclaredMethod("getPeer")
              : stageClazz.getMethod("impl_getPeer");
      tkStageGetter.setAccessible(true);
      TKStage tkStage = (TKStage) tkStageGetter.invoke(stage);

      Method getPlatformWindow = tkStage.getClass().getDeclaredMethod("getPlatformWindow");
      getPlatformWindow.setAccessible(true);
      Object platformWindow = getPlatformWindow.invoke(tkStage);

      Method getNativeHandle = platformWindow.getClass().getMethod("getNativeHandle");
      getNativeHandle.setAccessible(true);

      Object nativeHandle = getNativeHandle.invoke(platformWindow);
      return (long) nativeHandle;
    } catch (IllegalAccessException | InvocationTargetException | NoSuchMethodException ex) {
      logger.error("Unable to retrieve window id, due to Error: " + ex);
      throw new RuntimeException("Unable to retrieve window id");
    }
  }

  static Object invokeMethod(Object o, String methodName)
      throws IllegalAccessException, IllegalArgumentException, InvocationTargetException {
    Class c = o.getClass();
    for (Method m : c.getMethods()) {
      if (m.getName().equals(methodName)) {
        Object ret = m.invoke(o);
        return ret;
      }
    }
    throw new RuntimeException("Could not find method named '"+methodName+"' on class " + c);
  }
}

You can adjust the SDL bridge like in the MPV one (https://github.com/databrary/datavyu-ffmpegplugin/blob/dev/src/main/cpp/MpvMediaPlayer.cpp#L22) and use the JFrame reference when creating the SDL window here via SDL_CreateWindowFrom

If you are working on the Mac OS platform any call to the SDL2 library need to be done on the main thread!

You mentioned extra c++ threads created with FfmpegJavaMediaPlayer bridge, and removing them fixed the UDP lags during the playback, could you please point me to the code where this is happening?

Reda

[TheoWolf]

@saadatmand You are in the right place, the objective of this repo is to provide an FFmpeg media player for Java Applications, I'm working on 0.22 version that will provide to Java applications an SDL/FFmpeg player fully controlled via Java on both Windows and Mac OS. Having a standalone SDL window was the only way to have good rendering performance. If your issue is different or you need assistance to embed the plugin in your Java Application please open a new thread and I will be more than happy to help.

Best Reda

[TheoWolf]

@saadatmand @tahaarian I will close this ticket, please re-open it or feel free to open a new one if you have any question. Thanks