How to use pixi3d with pixi-viewport?

[Friksel]

Hi,

First of all; great work! This is a very nice piece of tool to be able to use 3d inside pixi! Thanks a lot for this!

At the moment I'm learning all features of it. I'm stumbling on one thing though I can't find the answer to so far: When adding the Model as a child of a pixi-viewport Container (which does scaling, dragging and all kind of cool stuff to navigate a 2d scene), the Model isn't moving with it. This does work with other display objects, but unfortunately not with a pixi3d model. At least I can't get it to work here.

How can we make the Model move and scale with the viewport (preferrably by not changing anything in the 3d object/scene nor the camera, so only its 2d 'render' output)

[jnsmalm]

Hello and thanks!

A 2D object (in regular PixiJS) is placed using screen coordinates, and 3D objects (Pixi3D) is using world coordinates. That is why the positioning is not really compatible, it's by design. But you can accomplish what you want in a number of ways.

1. Render your 3D object as a sprite using PostProcessingSprite. This will turn you 3D object into a regular sprite, and will behave like any other 2D sprite in PixiJS. See example at https://codesandbox.io/s/github/jnsmalm/pixi3d-sandbox/tree/master/post-processing-sprite. API at https://api.pixi3d.org/classes/PostProcessingSprite.html. This is easiest but might not be a good idea if you have many 3D objects in different positions on the screen.
2. Set Camera orthographic = true, then calculate the position of your 3D object using PIXI.Container.toGlobal and PIXI3D.Camera.screenToWorld.
3. If you don't want to use a orthographic camera, you can use the obliqueness property of the camera. See example below.

class ScreenSpaceCamera extends PIXI3D.Camera {
    constructor(renderer) {
      super(renderer)
    }

    update(container) {
      let { x, y } = container.toGlobal(new PIXI.Point())
      this.obliqueness.x = 1 + (x / window.innerWidth) * -2
      this.obliqueness.y = -1 + (y / window.innerHeight) * 2
      this.fieldOfView = window.innerHeight / 15
    }
  }

 let sprite = app.stage.addChild(new PIXI.Sprite(PIXI.Texture.WHITE))
  sprite.scale.set(20)
  sprite.anchor.set(0.5)

  let screenSpaceCamera = new ScreenSpaceCamera(app.renderer)

  let model = app.stage.addChild(PIXI3D.Model.from(resources["assets/teapot/teapot.gltf"].gltf))
  model.meshes.forEach(mesh => {
    mesh.material.camera = screenSpaceCamera
  })

  app.ticker.add(() => {
  // This will make the model be rendered at sprite's position
    screenSpaceCamera.update(sprite)
  })

Hope it helps!

[Friksel]

Thanks a lot for your quick and extensive helpful answer @jnsmalm ! With your code I'm going to dive into it. Have a nice weekend!

[Friksel]

@jnsmalm At the moment I'm using your code to synchronise. It's syncronising the positions fine and I'm working on the scale. I have two questions tho:

I don't get why the camera needs to be set on each material. Probably this is because of the uniforms in its shaders or something like that, but I would expect the camera to be set on a scene/model instead. Am I missing something here? Is this the only way?

How can we reach the default camera object for a model/scene when no camera is explicitly set after loading the model?

Thanks in advance!

[edit] Unfortunately it's starting to look like when syncing up both the scaling and positioning of the 3d model with the pixi-viewport it's so much work I'm basically creating another viewport just to display the 3d model in pixijs. Than it's a small step and easier to just use ThreeJs. Which is a pitty as this plugin got a great potential and it would be absolutely ideal to be able to use this plugin inside pixijs to have the best of two worlds when focused on 2d or flat isometric.

I really appreciate all your hard work and love the idea. But IMO what this plugin is missing now is a 2D representation of the output that's seemingly integrating with pixijs the way all other displayobjects do. I would expect the Model to be a 2D plane with all known pixi properties which is the result of an internal 3d render. If that would be the case we could just use this Model object like every other pixi object and add it as a child to other containers, like pixi-container.

When working with pixi I don't want to move the camera to position something on screen. I want to work with 3d and have shaders, materials, animations and all in 3d. But when it comes to positioning that model in the pixi world I want to just use pixi coordinates and scaling and would like the Model to be a true pixijs container with all its known methods in place.

IMO there's too big of a gap now between the 2d and 3d world making it difficult to use in a 2d environment. We now don't have all the things threejs can do, but also don't have the benefits of being able to treat the 3d output as a regular (but 2d renderered) pixi displayobject we are familiar with. So now we still need to worry about camera's and all when just moving things in 2d, while I would expect to just move the output and not having to worry about scaling, perspective and all when moving things around. Because if I would mind perspective relative to the screen I would have picked a 3d lib like threejs to begin with.

So basically what I would expect and would dream about:

• The Model is a displayobject which is a 2d representation (render) of a 3d world. We can treat this fully as every other 2d displayobject in pixi. So making it easy to scale and position without worrying about lights and camera's.
• Inside that model there is the 3d World with its 3d coordinates. When we move stuff around inside there, than we can change camera's, lights and all that in 3d space. But the result of this would always be the 2d plane which is the model itself.
• That way we could have best of 2 worlds: move the Model (= 2d render of 3d world) just like every other pixi object AND we can create a full 3d scene and treat everything 3d like we're used to in libs like threejs within the 3d world.
• We also have the benefit of scaling the world as the 3d model is fully built with scalable geometries so can be re-rendered when changed so the 2d output used in pixi is always sharp.
• We don't need to worry about scaling or positioning when adding a Model to a pixi container as it 'just works' like any other pixi object and scales and moves with its parent.

Sorry for this long post. I was reading on the gamedev forum you're open to feedback and you'd like to create a plugin that's easy to use and works the pixi way so I'm hoping you're still open to this feedback.

This plugin definitely got great potential and I'm very happy to start using it in projects, but I'm missing this link at the moment. I'm hoping there will be a easier pixi 2d connection in the future. Keep up the great work!

[jnsmalm]

Thanks for the feedback, can you help me understand how ThreeJS solves your problem? Also, pretty much all your issues can be address by using PostProcessingSprite, did you try that?

[Friksel]

@jnsmalm thanks for your quick response. First of all I'd like to emphasize that I in no way critisize you nor your code. It's obviously fantastic work and you put a lot of effort in this and it shows. It's also quite impressive what's possible in 3d with pixi3d.

can you help me understand how ThreeJS solves your problem?

Threejs doesn't solve the problem of having 3d object animated in 3d and rendered in a 2d world of pixi. That's exactly why I think this plugin is such a great idea and has such a great potential. What I meant was that if we still need to think in 3d to move and scale the output than it feels like a layer on top of the 2d world which feels disconnected to pixis 2d world. Than for me it would be easier to just setup everything in a 3d world and at least have one consistent coordinate system where everything is synced up.

Also, pretty much all your issues can be address by using PostProcessingSprite, did you try that?

No, I didn't know about that. Because of your answer I thought this would only be possible by moving either the camera or object in 3d space. I tried an orthographic camera, but it didn't make it easier to scale with the viewport. Especially because the viewports scaling center can be everywhere on the screen/world.

If PostProcessingSprite can be used without postprocessing, can be used with realtime rendering (for 3d animations), keeps the 3d coordinates connected to that sprite (instead of to the screen) so it moves with the sprite and doesn't take performance down that looks exactly what I'm after indeed. It would be fantastic if that's the case! Thanks a lot, I'll give it a try.

Did you also notice my questions above?

Please know I definitely appreciate your hard work!

[jnsmalm]

PostProcessingSprite should work well for what you want to do, it renders the 3d object to a regular PixiJS texture/sprite - that sprite behaves exactly as you would expect. The position of the model rendered to PostProcessingSprite should be (0,0,0) - but the PostProcessingSprite itself can have the position you want on screen (screen coords).

When loading Pixi3D, there is a default camera created at PIXI3D.Camera.main. That camera is used by default when no other camera is set on the material. The camera set on material is only needed in special cases (like the one in the example above). Normally, you don't need to do this. Note that when using PostProcessingSprite, you can define the size of the sprite. The aspect ratio of the camera used when rendering should match the aspect ratio of that sprite. So if you create PostProcessingSprites which have different aspect ratios, you need to set different cameras (with different aspect ratios) on the materials you are rendering. This is because the camera doesn't know you are rendering to a texture, so it assumes you are rendering to the entire screen (and will use the aspect ratio of the screen if not set).

Hopefully this answers your questions :-)

[jnsmalm]

Also, PostProcessingSprite doesn't use any actual processing if you don't give it any filters.

[Friksel]

@jnsmalm I need your help. Using a PostProcessingSprite on the pixi-viewport it synchronises with the viewport, but it never updates the resolution when the scale changes. So when the pixi-viewport is zoomed in the 3d graphics render is pixelated.

I want to correct this when the scale of pixi-viewport changes, but whatever I do I can't get this to work.

When the PostProcessingSprite is initialized we need to set a size for the sprite. When using higher values for the size this creates a higher resolution sprite. But when things are zoomed out I don't want this high resolution for performance reasons. So this needs to be changed on the fly while zooming.

So I set the width and the height of the sprite according to the relative scalefactor. And than I need to correct that scale with a container wrapper to keep it in sync with the other elements in the viewport. This should than up the resolution, but keep the size.

But I found out that when setting the width and the height of the sprite at initializing stage is doing something else than setting the width and height at runtime after initializing is done. When initializing is done it's still resizing the sprite, but isn't highering the resolution. It just scales the initial render output. So this doesn't make the sprite having a higher resolution.

I've tried a few other things, like trying to change the internal RenderTextures resolution with .setResolution, but this also doesn't seem to up the resolution, only changes the size of the sprite.

Whatever I do I can't seem to change the resolution of the sprite. After lots of trying I'm stuck. In an ideal world we would have the setResolution() and resize() methods of a renderSprite on the PostProcessingSprite to make it possible to up the resolution of the sprite without changing its size. But I'm probably missing something crucial here as I can't find anything that looks like doing that or making it possible.

I need your help: How could I make sure when the viewport is zooming in the sprite adjusts its resolution to the screen so the 3d graphics inside are always rendered sharp (also when using 3d animations)?

[jnsmalm]

Right now, the creation of the PPSprite doesn't support setting the resolution of the render texture. I can add this though, it should be very easy. This might improve your problem to some degree.

But, if you are scaling the PPSprite too much - the problem will still be there. Remember, the PPSprite renders the 3d object to a texture which becomes like a regular image, if you scale up any image too much the quality will suffer.

One thing you could try is to instead of scaling the PPSprite, you scale the 3d object that it rendered to the PPSprite, that will improve the quality for sure. This may or may not work depending on your specific requirements.

[jnsmalm]

Added a optional parameter when creating PostProcessingSprite called "resolution" in v1.2.1

[Friksel]

One thing you could try is to instead of scaling the PPSprite, you scale the 3d object that it rendered to the PPSprite, that will improve the quality for sure. This may or may not work depending on your specific requirements.

Thanks. Not sure how you imagine this tho. If I scale the 3d object it doesn't fit the size of the PPSprite anymore and the sprite needs to fit the object as that's the most performant. And there will be more PPSprites with 3d models inside the viewport doing the same so it adds up.

When scaling the 3d object we also need to change the PPSprite size and then need to correct that again to keep the same size of the PPSprite inside the viewport as the viewport is doing the scaling, not the PPSPrite. The sprite only needs to adjust its resolution to the new viewport scaling. So basically we need something like setResolution(). Although that's normally used for the screen resolution to adapt to retina/HiDPI screens we can use a factor on that to use it for when scaling the viewport.

Added a optional parameter when creating PostProcessingSprite called "resolution" in v1.2.1

That's great and one step forward. And nice to be able to enter the screen resolution in there. But for the pixi-viewport issue this doesn't solve things. For performance reasons we don't want to render the PPSprite at a higher resolution then needed when initializing as the pixi-viewport is zoomed out at that stage to show everything (so the sprites don't need huge render sizes). The initializing isn't the big issue. It's the runtime part AFTER the initializing that can't change the resolution of the sprite. When changing the width and height of the sprite after initializing it's not changing the content, only the already renderered sprite so we have great quality loss after zooming in.

To be able to zoom without quality loss is needed when zooming in and out the viewport (= the parent container of the sprite). When we zoom in (= scaling the viewport container, so the parent) the sprite goes with it so shows bigger while being the same size, but the content resolution of the sprite doesn't change, so the more we zoom in the more we see jagged edges and pixelated graphics. So while zooming the size of the PPSprite needs to stay the same (as the viewport is already scaling it with itself), but the resolution of the inner graphics need to change.

If the PPSprite would have a setResolution() and that method would underwater change the render size and scale the camera view to that without changing the actual size of the PPSprite, this would fix it and I'm sure it would be very helpful for a lot of use cases. We then really have scalable 3d graphics and are compatible with pixi-viewport when using the PPSprite.

It would also help to solve this if the width and height setters of the PPSprite would not blow up or shrink current renderpixels down, what it seems to do now, to fit the new size, but re-rendering the camera with the new resolution/sizes instead (so it fits the new size of the PPSprite with a fresh resolution). Otherwise we could never scale up without quality loss and the nice thing about 3d, like vector, is that it is endlessly scalable. So we'd like to use that when zooming in on the viewport. But then we should also scale the size of the PPSprite to keep it the same inside the viewport after changing the 'rendersize'. So this method is not ideal.

But scaling the resolution could for some use cases be an unwanted performance thing tho. For instance when animating something in or out just to pop something in view (so it starts with a scaling lower than 1 and ends with scale 1). So perhaps instead of changing the way width and height are working, a resize() method (like the render texture's resize()) would be better. And would be inline with pixi's own api.

Scaling the resolution instead of the size would be easier to use tho and more intuitive to use for this use case IMO. But being able to resize with the best possible resolution inside that is something that's very, very welcome too (and what I expected from setting the width and height to be honest)!

So setResolution() or (but preferably also) resize() in the PPSprite would solve it. Hope this makes sense.

[Friksel]

Just made an image to illustrate this a little better, hope this helps:

BTW It just popped into my mind that setResolution() should be able to handle subpixels in order to use it for this purpose.

[jnsmalm]

I fiddled around with the functions for texture in PixiJS. Might have found a solution for this. Let me know if it works for you, when maybe I can add it to the API. Also not entirely sure what you mean with PixiJS viewport, is this the same as the canvas?

In this example I render a 3d object using PostProcessingSprite. The sprite is scaled 4 times which makes it blurry by default. I can press A/Z to increase/decrease the resolution which changes the quality of the rendered output without changing the size of the sprite.

let sprite = app.stage.addChild(new PIXI3D.PostProcessingSprite(app.renderer, {
    objectToRender: world,
    width: 128,
    height: 128
  }))

  sprite.scale.set(4)

  function setResolution(resolution) {
    sprite.texture.setResolution(resolution)
    sprite.texture.resize(128, 128, true)
  }

  let resolution = 1
  document.addEventListener("keydown", e => {
    if (e.key === "a") {
      resolution = resolution * 2
      setResolution(resolution)
    }
    if (e.key === "z") {
      resolution /= 2
      setResolution(resolution)
    }
  })

[Friksel]

@jnsmalm I just see your reaction coming in, so not yet had time to look at it, but wanted to tell you that with 'viewport' I mean pixi-viewport (https://github.com/davidfig/pixi-viewport). Which is basically an interactive pixi container to drag and zoom on everything that's inside. Somewhat like SVG Viewbox, but already with extensions to drag, zoom from a point (with the mouse wheel for instance) etc. build in. Advisable to check out as it's a pretty useful 'plugin' to know!

I'll get back to you for the rest later. Thanks a lot and for now have a nice weekend!

[Friksel]

Hi @jnsmalm I just tested your code and this works perfectly! It's changing the resolution while keeping the size! Also with animations. Great!

When scaling the pixi-viewport with the mousewheel the scale grows exponentially so it results in huge sprite sizes quickly and makes the viewport slower, but that's just the way it technically is and we can limit the max resolution of the sprite or limit the max scale of pixi-viewport so that's fine. It's way better than before and it's great we can now scale up the sprite without quality loss!

And also when using pixi3d without pixi-viewport this is very benificial as we can now scale without quality loss AND adapt to hi-dpi/retina screens with both the initial resolution setting and the setResolution() method!

Thanks a lot for all your help and effort! I'd say the setResolution() method is approved to be in the API ;)

[jnsmalm]

Glad it worked out!

[Friksel]

@jnsmalm Thanks for adding the new method!