Added a Method (`panorama(img)`) which adds a sphereMapped Background.

[perminder-17]

Resolves #6752

Changes:

• I have developed a function that integrates a sphere-mapped background into the scene. The fragment shader is responsible for accurately rendering the reflections of the background.

https://github.com/processing/p5.js/assets/127239756/199de001-a790-4bee-b4c1-908db23e3fbe

Screenshots of the change:

PR Checklist

• [x] npm run lint passes
• x is included / updated
• are included / updated

[perminder-17]

The PR is ready for your review, @davepagurek. Please share your thoughts on it. Additionally, I'm unsure whether to store the sphereMapping file separately or use it directly within a function.

Also calling @haroon10725 as you were intrested in solving this. Please share your thoughts on this one.

[perminder-17]

I think uModelViewMatrix here is going to be made for the filter shader's rectangle, so I'm not sure it's what we want for doing reflections. Does it work if you set vViewPosition to all zeros, since we're assuming the position of the camera is negligible, assuming the texture is on an infinitely large sphere? And if so, can we just do n = normalize(vGlovalNormal.xyz) directly without doing a reflect?

You're right about that, your logic seems correct. However, despite following this approach, we aren't achieving the desired outcome. Take a look at my code for reference.

#define PI 3.141592

precision highp float;

uniform sampler2D uSampler;
uniform mat3 uNewNormalMatrix;
uniform mat3 uCameraRotation;
uniform mat4 uNewModelViewMatrix;
uniform mat4 uModelViewMatrix;

varying vec2 vTexCoord;
varying vec3 faNormal;
varying vec3 faPosition;

void main() {
 // vec4 viewModelPosition  = uModelViewMatrix  * vec4(faPosition, 1.0);
  vec3 vViewPosition  = vec3(0.0);
  vec4 newTexColor = texture2D(uSampler, vTexCoord);
  vec3 vGlobalNormal = uNewNormalMatrix  * faNormal ;
  vec3 n = normalize(vGlobalNormal.xyz);
  vec2 suv;
  suv.y = 0.5 + 0.5 * n.y;
  suv.x = atan(n.z, n.x) / (2.0 * PI) + 0.5;
  newTexColor = texture2D(uSampler, suv.xy);
  vec4 baseColor = newTexColor;
  gl_FragColor = baseColor;
}

we are getting something like:-

https://github.com/processing/p5.js/assets/127239756/a8818620-aa7a-42ea-8618-8892439d2c5e

[davepagurek]

Good point, if we take out that matrix, we need to account for the different perspective from each pixel in a different way. Using uMVMatrix uses the filter camera's perspective to do that, where we ideally would want the active camera from before the filter shader is called. Maybe the thing to do is just to add new uniforms for the camera's field of view and the camera's aspect ratio? Then in the fragment shader, the initial normal before applying the normal rotation matrix would be something like:

• start as vec3(0., 0., 1.)
• rotate about the X axis by the angle mix(-fovy/2., fovy/2., vTexCoord.y)
• rotate about the Y axis by the angle mix(-fovx/2., fox/2., vTexCoord.x) where fovx = fovy * aspect

[perminder-17]

Good point, if we take out that matrix, we need to account for the different perspective from each pixel in a different way. Using uMVMatrix uses the filter camera's perspective to do that, where we ideally would want the active camera from before the filter shader is called. Maybe the thing to do is just to add new uniforms for the camera's field of view and the camera's aspect ratio? Then in the fragment shader, the initial normal before applying the normal rotation matrix would be something like:

• start as vec3(0., 0., 1.)
• rotate about the X axis by the angle mix(-fovy/2., fovy/2., vTexCoord.y)
• rotate about the Y axis by the angle mix(-fovx/2., fox/2., vTexCoord.x) where fovx = fovy * aspect

Thanks...Previously, I was confused about the rotation matrix. I believe I understand your point now and have made the necessary updates. The code is still functioning as it was before. Let me know if any thing still needs to be changed. I'll look into the minor cleanups, indentations, examples once the code looks good to you.

[perminder-17]

I also notice what seems to be a tilt happening sometimes you rotate the view. Do you know if this happens because of the new way we're calculating the initial normal, or did this happen before too? It could be that it's something about how we're applying the normal matrix, not sure yet.

Ah, I just overlooked this one. I see where the issue lies. It's not with the new Normal or old Normal matrix, but rather with how we're calculating it. Specifically, we're performing an inverse transpose of the uMVMatrix. The problem arises when we rotate the scene by 180 degrees; this alters the orientation of the normals. For instance, if we drag the scene downwards by clicking the left button and pulling down, after a 180-degree rotation, dragging the cursor downwards causes the scene to move upwards. The orientations is getting changed, so in between when you try to drag the scene downward, it's tilting funny - like half of it's going up and half of it's going down!

The result after fixing this have a look:-

https://editor.p5js.org/aman12345/sketches/AwsE_YTLa

[davepagurek]

The rotation of the background looks good now! Although I noticed in the sketch that the tilt behaviour we saw in the background before now seems to be present in the reflections in imageLight: https://editor.p5js.org/davepagurek/sketches/HjrSsZN2n

https://github.com/processing/p5.js/assets/5315059/a5c78aef-05fa-4585-a403-afb0a22da5ff

This doesn't seem to be the case using the 1.9.1 beta build though: https://editor.p5js.org/davepagurek/sketches/edkYxkJAa

I'm not sure what might have changed to cause that yet, do you have any ideas?

[perminder-17]

The rotation of the background looks good now! Although I noticed in the sketch that the tilt behaviour we saw in the background before now seems to be present in the reflections in imageLight: https://editor.p5js.org/davepagurek/sketches/HjrSsZN2n

Screen.Recording.2024-02-20.at.12.22.25.PM.mov This doesn't seem to be the case using the 1.9.1 beta build though: https://editor.p5js.org/davepagurek/sketches/edkYxkJAa

I'm not sure what might have changed to cause that yet, do you have any ideas?

Oh, no, this isn't a bug. I apologize. I was actually working on fixing the code above and testing it with my current camera matrix. So, I just reverted those changes. In this pull request, it's functioning somewhat like this: https://editor.p5js.org/aman12345/sketches/TAWBmKR-5

[davepagurek]

Ah ok makes sense, thanks for clarifying!

[perminder-17]

Please let me know if we can enhance the readability and user-friendliness of this document.

[perminder-17]

Sorry for the delay working on this. I have made the changes.

[perminder-17]

Thanks for your continued work on this, I think this is good to go now!

Thanks. Excited to see this feature how it goes.....and really thanks for your constant review on this. Love to contribute more like this❤