[Question] React components as UI elements

AntzeloBRKK commented 1 year ago

First of all hello and a big shoutout for the amazing work done in the library, it has proven very helpful so far and relatively easy to integrate babylonjs with react.

I'm currently working on a Nextjs project where I have basic scene setup with a ground grid and a model with some animations and some playback controls.

What I'm trying to achieve is have the Scene and model/s render inside but the UI I've built using react components and attach the respective functions to their event handlers.

It's far easier and more consistent for me to use react to built the UI components hence the need for help/clarification as to how I go about this.

The current structure is as follows:


const MeshComponent = () => {
    return (
        <Suspense fallback={<box name='fallback' position={position} />}>
            <Model />
        </Suspense>
    )
}

const SceneView = () => {
    return (
        <div>
            <Engine>
                <Scene>
                    <arcRotateCamera />
                    <hemisphericLight />
                    <directionalLight />
                    <ground />
                    <MeshComponent />
                </Scene>
            </Engine>
        </div>
    )
}

const PlayerControls = () => {
    return (
        <div>
            <PlayBtn />
            <LoopBtn />
            <Slider />
            <SpeedBtn />
            <CameraLockBtn />
        </div>
    )
}

const PlayerComponent = () => {
    return (
        <div>
            <SceneView />
            <PlayerControls />
        </div>
    )
}

One of the issues I had in the beginning was of the hooks useScene and useEngine but I quickly realized that they were correctly null due to them used inside the PlayerControls which is sitting on the same level as the SceneView component and not inside the Scene itself.

I circumvented most of the things by using the onSceneMount callback and storing a ref to them in a zustand store, I was using either way, like this:

  function onSceneMount(sceneArgs: any) {
    const { scene } = sceneArgs;
    const engine = scene.getEngine();
    if (scene && engine) {
      setSceneRef(scene);
      setEngineRef(engine);
    } 
  }

Unfortunately, this seems to not completely solve the issue I was having. I tried to add some onSeek event when the slider moves in order to set the frame and then use .goToFrame() (kind of inspecting the animation) but the ref to scene in that function is undefined although in the same component as play/pause.

I also tried to use a <Html> component inside the scene (under <adtFullscreenUi> and render the controls as {props.children}. The html element (although I can add a className to it and style it) it is wrapped in another div with some inline styles and messes up the rest of styles.

To sum it up my question is what do you think would be a proper way to wire something like this together? Is there a possibility of using the context provider in the parent component but the Scene itself render lower in the tree?

p.s. sorry if this is not the right place for it but it felt directly corelated to some of the features here (useScene hook, <Html /> component as scene child)

brianzinn commented 1 year ago

This is the right place - good question.

Firstly that's a good way to grab a scene ref and make it available outside. There is an issue with React and when switching renderers (reactDOM and react-babylonjs) that they do not share context, but zustand gets around that so is also the right choice. Keep in mind also StrictMode will run twice in dev mode, but that should be fine.

I think in zustand you just need to call const { scene } = get()?

The Html component is wrapped in a div, so that it can control with CSS to present it in screen space with rotation/scaling.

It's hard to see exactly where you need help. Maybe you can make a concrete example like a codesandbox?

AntzeloBRKK commented 1 year ago

Thanks for the feedback will try to make some time to produce a small subset of the thing I'm doing to better illustrate my case, i've been a bit busy these days but will definitely come back with an update.

AntzeloBRKK commented 1 year ago

@brianzinn Hello, I've come back with a codesandbox example of my question here: https://codesandbox.io/s/blue-moon-uwk4kt

I'm having an issue to load the asset with the Model component (which works locally, so I think this part of codesandbox's problem) but the general structure is the same as to what I'm trying to achieve with the use of the scene outside the canvas to implement some basic playback controls.

brianzinn commented 1 year ago

~do i need a pro account or something to run that?~ nevermind that error - the next error I got was that you are missing import @babylonjs/loaders and then the model loaded - it's a bit unresponsive. i'll check it out tonight - I think I see the issue, so will be good to discuss ways to accomodate that scenario. cheers.

AntzeloBRKK commented 1 year ago

Yea, sorry about that. I'm not sure what is causing that error on codesandbox because locally my model works correctly with the use of the <Suspense> component as it waits until it completes loading.

My point is beyond the issue of loading the asset though (always helpful to know if that's a good practice or not) but the feedback Im mainly looking for is on how I am using zustand to store a the scene Reference and use that inside other components to implement some of the playback controls (like updating the slider value when the animation is playing).

Specifically, if I uncomment the useBeforeRender hook on the Model component to update on each call the slider value (which is a DOM element) makes a huge fps drop which I understand is not because of babylonjs or the hook itself but the fact that a DOM element changes props almost 60 times per second 🤣 .

brianzinn commented 1 year ago

i'll try locally as well! i see your point - if you look at libraries like react-spring they do animations outside of the react render loop - that's what i do as well generally. i agree the sync is not in state, so cannot display numbers without some dhtml, which is what we are trying to get away from in the first place with React... let me have a better look and report back.

brianzinn commented 1 year ago

sorry been super sick lately - haven't forgotten. will look again soon...

brianzinn commented 1 month ago

This issue ended up being forgotten - it's not alone - sorry. It looks like the solution is not to use state to update in the render loop - as it causes perf drop. If there is something more specific with performance or this is unresolved - kindly re-open and we can investigate. Cheers.

brianzinn / react-babylonjs

[Question] React components as UI elements #247