search

HumbleUI / JWM

Cross-platform window management and OS integration library for Java
Apache License 2.0
552 stars 44 forks source link

Possible to attach to existing OS-specific window handle given to the process (not spawn new window + handle)? #150

Closed GavinRay97 closed 2 years ago

GavinRay97 commented 2 years ago

Hello, first I'd like to thank the contributors of this library -- it opens up very exciting possibilities and is sorely needed in the ecosystem. So thank you all 😃

My question:

I've been browsing the source, trying to understand if the following scenario is possible:

  1. A process is created (let's call it ParentApp) and launches it's own window/UI
  2. ParentApp creates a new window + handle, let's call this ChildWindowHandle, and passes it to our JWM program
  3. We would like to receive this ChildWindowHandle pointer and use it to render/attach our JWM app UI to
    • On Windows this would be an HWND
    • On Linux, this would be an XWindowID
    • On Mac OSx, this would be an NSView or HIView

If this sounds strange to you/you can't imagine why you would want to do this:

A popular example (my usecase): VST Audio Plugins for DAW's

It would be amazing to be able to author audio plugins (or at least the UI's) in say, Kotlin. General availability to JVM languages would be revolutionary.

For a code short example, essentially what I'm asking is if something like this is possible (minus the low-level X11 stuff, just the bit about receiving and using the parentWindowHandle to render):

import kotlinx.cinterop.*
import Xlib.*

fun vstOnAttached(parentWindowHandle: COpaquePointer, platform: String) {
    val msg = "Hello, World!"

    val d = XOpenDisplay(null)
    if (d == null) {
        println("Cannot open display")
        return
    }

    val s = XDefaultScreen(d)

  /* Window XCreateSimpleWindow(display, parent, x, y, width, height, border_width, 
                                border, background) */
    val w = XCreateSimpleWindow(d, parentWindowHandle.reinterpret<Window>, 10, 10, 160, 160, 1,
                                XBlackPixel(d, s), XWhitePixel(d, s))
    XSelectInput(d, w, ExposureMask or KeyPressMask)
    XMapWindow(d, w)
    val e = nativeHeap.alloc<XEvent>()

    while (true) {
        XNextEvent(d, e.ptr)
        if (e.type == Expose) {
            XFillRectangle(d, w, XDefaultGC(d, s), 55, 40, 50, 50)
            XDrawString(d, w, XDefaultGC(d, s), 45, 120, msg, msg.length)
        }
        else if (e.type == KeyPress) break
    }

    XCloseDisplay(d)
    nativeHeap.free(e)
}

Curious if there's any way to accomplish this currently? Thank you

Boring Contextual Details Below

The way audio plugins work, is that the host application (generally a DAW) allocates a window handle and gives the plugin the window to render into:

image

tonsky commented 2 years ago

Sounds like a great use case. Do you think it should support drawing only or window events too? E.g. mouse move or window resize?

GavinRay97 commented 2 years ago

Sounds like a great use case. Do you think it should support drawing only or window events too? E.g. mouse move or window resize?

IMO the ability to respond to those would be really useful -- particularly the mouse movement (a lot of FX use cursor-movement based UI updates), but I'd be grateful for whatever is possible.

Many audio plugins don't handle resizing 😅

Here's a video example of "Responds to mouse movement in UI, but you're SOL if you want to resize the thing"

https://user-images.githubusercontent.com/26604994/132262119-b44c3537-81aa-4854-a420-22afcd77b24a.mp4

GavinRay97 commented 2 years ago

Looking at the ctor of Window, it looks like it just takes a pointer? Is this pointer just a void* to whatever the platform representation of a Window handle is?

https://github.com/HumbleUI/JWM/blob/328040db418894de306c944df5ea96784e478237/shared/java/org/jetbrains/jwm/Window.java#L9-L19

https://github.com/HumbleUI/JWM/blob/328040db418894de306c944df5ea96784e478237/windows/java/org/jetbrains/jwm/WindowWin32.java#L11-L16

https://github.com/HumbleUI/JWM/blob/328040db418894de306c944df5ea96784e478237/windows/cc/WindowWin32.cc#L885-L893

So from the looks of it, it seems like MAYBE this should already work if there were just a secondary constructor on each platform-specific Window implementation from the Java side, like this:

public class WindowWin32 extends Window {
    @ApiStatus.Internal
    public WindowWin32() {
        // Make a new window
        super(_nMake());
    }

   public WindowWin32(long HWND) {
        // Attach to an existing window, using platform-specific pointer
        super(HWND);
   }

   // Or maybe a static method, probably better to be put on "Window" but whatever
   public static Window fromPlatformSpecificWindowPointer(long ptr) {
        super(ptr);
   }
}

And then, I don't really know C++ very well (or Java for that matter lmao 😅) and I've never used JNI, but I think usage would look roughly something like this?

package com.acme;
import org.jetbrains.jwm.*;

class ExampleApp {
  public static void initFromNativeHWND(long HWND) {
    Window window = new Window(HWND);
    // or
    Window window = new Window.fromPlatformSpecificWindowPointer(HWND);
    // Now, write the rest of the app ;^)
    // "Step 2. Draw the rest of the owl"   
  }
}
// Stuff to init JVM here
bool initJVM() {}

// Invoke our JWN app, handing it the HWND/window handle (pointer) we've already allocated
bool invokeJWMAppWithHWND() {
  jclass cls = env->FindClass("com/acme/ExampleApp");
  jclass kCls = static_cast<jclass>(env->NewGlobalRef(cls));

  // public static void initFromNativeHWND(long HWND)
  jmethodID kInitFromNativeHWND = env->GetStaticMethodID(kCls, "initFromNativeHWND", "Ljava/lang/void");
  return true;
}

int main() {
  // Register the window class.
  const wchar_t CLASS_NAME[]  = L"Sample Window Class";

  WNDCLASS wc = { };
  wc.lpfnWndProc   = WindowProc;
  wc.hInstance     = hInstance;
  wc.lpszClassName = CLASS_NAME;

  RegisterClass(&wc);

  // Create the window
  HWND hwnd = CreateWindowEx(
      0,                              // Optional window styles.
      CLASS_NAME,                     // Window class
      L"Learn to Program Windows",    // Window text
      WS_OVERLAPPEDWINDOW,            // Window style
      // Size and position
      CW_USEDEFAULT, CW_USEDEFAULT, CW_USEDEFAULT, CW_USEDEFAULT,
      NULL,       // Parent window    
      NULL,       // Menu
      hInstance,  // Instance handle
      NULL        // Additional application data
  );

  if (hwnd == NULL)
      return 0;

  // ====================================
  // Initialize JVM & environment
  // then call "ExampleApp.initFromNativeHWND()" and give it our new window's HWND
  if (initJVM()) {
    invokeJWMAppWithHWND(hwnd);
    return 1;
  }

  return 0;
}

Also @tonsky, may as well mention it: I'm a big fan of your work =) Thoroughly enjoy reading your content on The Orange Website 🟧 Didn't know you worked for JB, neato

GavinRay97 commented 2 years ago

Though I should prolly clarify -- I don't intend to create a JVM in native code (C/C++/Rust etc) and call it this way. (I will if I have to and it's the only way to get it to run in a .dll/.so 😅)

If this functionality is added, I would try to work on getting GraalVM to compile JWM so that it's useable with native binaries and shared-libs, or Kotlin Native (if that's an option).

And I would use either Graal's @CFunction or Kotlin Native's equivalent, to create the underlying native window pointer

tonsky commented 2 years ago

Interesting! I see some obstacles on your plan:

May I ask, what kind of features do you plan to get access to by using JWM instead of native Win32 APIs? Especially if you are planning to write the rest of your plugin in native code (if I understand you correctly)?

Re: Java Window ctor, it takes a pointer to our own native C++ Window class, e.g. WindowWin32

GavinRay97 commented 2 years ago

Re: Java Window ctor, it takes a pointer to our own native C++ Window class, e.g. WindowWin32

Ahh, well it would not be so simple then! Nothing ever is, aye? 😅

May I ask, what kind of features do you plan to get access to by using JWM instead of native Win32 APIs? Especially if you are planning to write the rest of your plugin in native code (if I understand you correctly)?

I want to open up audio plugin development to the JVM, as a platform.

Personally I am not a fan of C++, and I can imagine how nice it would be to be able to write plugins in a "nicer" (sorry) language like Kotlin or Java.

There's no getting around the fact that you need native code to do this -- but that native code doesn't ACTUALLY have to be C/C++.

What I mean by that is, for example there's someone who has ported the VST C++ API's to a C FFI wrapper and then done Kotlin Native on top of that.

Here's a Kotlin Native binding to the EditController C++ class of the VST SDK:

So my intention was to:

Compiling to GraalVM might get tricky

Yeah definitely -- I see there are issues with it currently and have checked out your notes and build scripts.

I have some familiarity with it, and had planned to troubleshoot getting it working + reach out to the Graal devs on Slack for ideas (I've done this a good number of times and they're really helpful) if I absolutely couldn't make headway on it.

It's possible that:

Which would suck, but that's a "cross that bridge when I get to it -- with a positive attitude" sort of thing haha.

Window events would not probably work as nice when using only part of the window (we didn’t planned for this)

Just to double check, could you elaborate more on this?

I don't actually know very much about windowing API's, does this mean that child windows are more like partial windows rather than regular whole windows?

JWM was designed to control rendering itself (buffer swaps, vsync, see Layer interface). It might get in the way if your host app is doing it for you

I think the host app essentially hands a "blank canvas" window, where you're free to do to the window what you wish. Someone had asked about using VSTGUI (A UI framework for VST plugins) with OpenGL custom rendering, and the VST devs answered: "Just give VST the HWND and then have your OpenGL app do the rendering on the window":

image

GavinRay97 commented 2 years ago

After speaking with some folks who are more experienced at Win32 dev, it seems the expected design pattern is to simply re-parent windows.

So if you are given a HWND to attach to, it's expected you do this by calling SetParent(hWndChild, hWndNewParent);

"Yes, you can reparent literally any window in the system to yourself." "If you're given an HWND, reparenting is basically the expected usage pattern, as you can't do much with that HWND except reparenting to it."

So maybe something roughly like this is an easier implementation?

interface Reparentable {
   public void reparentWindowTo(long parentPlatformSpecificWindowHandle);
}

abstract class Window implements Reparentable {
  private long platformSpecificWindowHandle; // HWND, XWindowID, NSView, etc
}

class Win32Window extends Window {
  // https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setparent
  private native SetParent(long hWndChild, long hWndNewParent);

  @Override
  public void reparentWindowTo(long parentPlatformSpecificWindowHandle) {
    this.SetParent(this.platformSpecificWindowHandle, parentPlatformSpecificWindowHandle);
  }
}

And then trying to get JWM built as a native resource with Graal or Kotlin Native should be the only barrier left if re-parenting is also available (and an acceptable design pattern) on other platforms I think?

tonsky commented 2 years ago

Window events would not probably work as nice when using only part of the window (we didn’t planned for this)

Just to double check, could you elaborate more on this? I don't actually know very much about windowing API's, does this mean that child windows are more like partial windows rather than regular whole windows?

Me neither :)

The host app essentially hands a "blank canvas" window, where you're free to do to the window what you wish

Just to reiterate: host app creates an empty window for you, and your plugin is running in a different process? And from that process you want to reparent the original window to yourself? So we need something like Window::makeFromHWND(long hwnd) that would do reparenting internally. Does that sounds like what you need?

GavinRay97 commented 2 years ago

host app creates an empty window for you, and your plugin is running in a different process?

Yeah, correct. The architecture of the audio plugin is:

Digital Audio Workstation process
          |                                
  VST3 Interfaces (C++) 
          |                         
      MyPlugin.dll

The DAW host process will allocate an OS-specific empty window, and then call: IPlugView::attached(void* windowHandle, enum kPlatformName)

And then the plugin is responsible for taking the window handle and figuring out how to draw itself on that window.

Which I guess apparently means: "Set this window as your app's parent/container window and then spawn your own window as it's child."

On Windows that's HWND and SetParent() and on Linux, I believe that's an XID ("Window") and XReparentWindow():

XReparentWindow(display, w, parent, x, y)
      Display *display;
      Window w;
      Window parent;
      int x, y;

image

image

And from that process you want to reparent the original window to yourself? So we need something like Window::makeFromHWND(long hwnd) that would do reparenting internally. Does that sounds like what you need?

Yeah that sounds fantastic 💯

tonsky commented 2 years ago

Ok, I think it’s getting more clearer now. Before, I though that SetParent() has something to with process separation. Now I see they are all in the same process, and SetParent actually requires two windows: parent and child. So it’s a way to put one window inside another.

In light of that, I think the original proposal of Window::setParent(long hwnd) would serve this purpose better. I am not sure how would one window inside another look or work, but we can start with this method only and you can report what else do you need.

tonsky commented 2 years ago

See https://github.com/HumbleUI/JWM/commit/8417f29175c11568a5ec8531e0336bcb0a4f43f9, let me know if it works

GavinRay97 commented 2 years ago

Awesome -- thank you!

Going to build this and then try to make a dummy Win32 app that spawns one Window, and then fires up JVM and sends across the HWND to a Java class method that calls this new function 👍

First time I've hosted the JVM in native code so may take me a couple of hours, hopefully this works and then I can start seeing if there's any headway I can make on getting JWM to work with Graal. Because at that point, it's just a matter of writing (roughly) this:

@CContext(Main.Directives.class)
class JwmNativeExample {

    interface Callback extends CFunctionPointer {
        @InvokeCFunctionPointer void invoke(long hwnd);
    }

    // Exposes "void myEntrypoint(long hwnd);" as a C function when built as static/shared lib w/ Graal
    @CEntryPoint
    @CEntryPointOptions(prologue = CEntryPointSetup.EnterCreateIsolatePrologue.class,
                        epilogue = CEntryPointSetup.LeaveTearDownIsolateEpilogue.class)
    private static void myEntrypoint(long hwnd) {

    }

    private static final CEntryPointLiteral<JwmNativeExample.Callback> myEntrypointCallback =
          CEntryPointLiteral.create(Main.class, "myEntrypoint");
}

And then now in CXX you can do:

int main()
{
  auto jwmLibrary = LoadLibrary("my-jwm-library.dll");

  using myEntrypointFn = void(*)(long hwnd);
  auto myEntrypoint = (myEntrypointFn)GetProcAddress(hModule, "myEntrypoint");

  // pass HWND here
  myEntrypoint(hwnd);
}

Of course the whole point of this is to avoid writing any C/C++, so the only thing that has to be done is to expose the proper @CEntryPoint names (pluginFactory) from Java/Kotlin/Clojure whatever, and then the VST3 framework will automatically invoke them for you when it imports your plugin library =)

The net result is that you can write audio plugins that export and interface with, native C(++) methods, without writing any C(++) yourself -- able to use Graal to implement the native handlers!

GavinRay97 commented 2 years ago

@tonsky It works! Thanks a ton!! ❤️

Check it out: how neat is this? 🙌

https://user-images.githubusercontent.com/26604994/134979997-018af42c-d6b5-49d5-9cbb-2f90d7500ac1.mp4

tonsky commented 2 years ago

Mind-blowing!