are `u_tex` working right with videos?

[konsumer]

I have a shader like this:

#ifdef GL_ES
precision mediump float;
#endif

uniform vec2 u_resolution;

uniform sampler2D u_tex0; // britney.mp4
uniform sampler2D u_tex1; // hypnocat.png

void main() {
    vec2 uv = gl_FragCoord.xy / u_resolution.xy;
    gl_FragColor = texture2D(u_tex1, uv);
}

I am working on a live-editor with uniform-inputs.

Initially, britney.mp4 does not load, but after calling update() (with new shader, from prisma-live editor) it's in u_tex1 (where I would expect hypnocat.)

It seems like it has some sort of video support, but it's putting it in the wrong uniform.

If I set it up like this, initially, it works as expected:

uniform sampler2D u_tex0; // hypnocat.png
uniform sampler2D u_tex1; // https://prismjs.com/assets/img/spectrum.png

Possibly related:

• 34

• 10

Maybe 2 features I would like to add is working video-support and WEBCAM as URL. Is there interest in a PR?

[konsumer]

Additionally, I tried injecting a base64 URL (on user file-browse) and it does similar mixed-up texture buffers.

#ifdef GL_ES
precision mediump float;
#endif

uniform vec2 u_resolution;

uniform sampler2D u_tex0; // data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAAXNSR0IArs4c6QAACFRJREFUWEc9l8uuXEcVhr+1qnZfzsU2thM5iREIJkAACcEYGDJiyIgBT8EM8QbwErwBkygjBkgIMSCTCGVCUIgSHMeXc3wu3b2r1kJ/7XaObZ129+6qWmv9t7LPHpXEDAdWblSHHoUXxXGMWqB6Ui1xNxLDHDwhA/RdDIIZa0akcbBgNiO1VgIJlpXeje6H8V5rTvaK/ePxW7mrF3TrhBm3Wq0He6BjNC1WbGyohbDlhQ6XYRwS9sTYuIfKcDyNmhAWVDMKSfE+ipxshWenmN5z7HHZ5ht07nDGRc68tB1mSc1kZcYqE09nMjhxuFuMjcHKoOKc6ExuTGokSbcYx0v9MafrYGnM6XSCEnommcdxEvt53eRt7jmkc6KFtSHrcXrLPloclpzhnJmx1bhGoTqYqoczhzMrHIBLa6zNuGeJVWipEdloXDdINXA0Min6/eeTbVqueVXOuNECvbMzeBGFl3rEg7thPCzqgHNB50WfOKTq2ZF+wgMa972yz8IXrXGdGuEOz0t2wE4VZ9ARVpJA3djQuIf97exeblxnD1wnDR/Vp8eY06RppyYbTAGtqCK1WN9QI4xiAVHoGkEmszY0KBiafKawBHvBaw6aOdcdDqyxD8+3ufXGuhhXPXh2BJZ2Feo9knXC3QonpYyFd12LBb3YmOkqg3VxKD6qf9GC6zQuD6kecRPJPk0I4Cr13ih3GcX+3fuZXhZ0557uqjCJgdoA/V9VeENkzaKK2qCj/vmqkJ5YLrRDYNTvCKI7HjbmHglVM0inNqc1w9oa+8XJuVjKnI19tOWLwEH7jk80M60rZgj9+mBhpHRgTGQcXqNL8jXKgCZtGcdfxpVilaHGc6MipSk/KZ56SLM+MecuydacSWSy4KEV3rbCo+rccTifYD051TsnK6gTR1Z0oqhnhQgxfzlkDx1KHZiYZydDHUzmaLTesH9ufpelnDHVD7HpKZN/jtUvobygRKOq5S5VTKqWHxXa6Ixn+Ypee1sNxuhHSC9t6dzV7PxPFet1D2aNaoA32Wil/5bneVmCD+pHrHjOk/IXPso/cVkuIAW65Nm+c5PBypK1weRVeknJ79Dtp7xo7/O1/A/f9IescVa+58Q6ncJNOM8D5nGsHdXqYEfQ2NkN9pvNH/Jp7PiwH7j0L7nNp/R8nygXA3QSC41CdNOM+zABKYoOcIfgTTyf8iBveMuCKZNTfT6UMGkmKr5GjV4nky0+Ixm371nNrSUbS94x5/vV+Pa2crpy1mtjvRaqu0jN5CusFyQpXQRxZ0XnUnyxA3dWPsbUWydnmCpITGOAVSamQRrzIdgfgqtesL8/2OSZJ+demRI2myZb5MKNlz14tU8+3RuvWvKwdh6tk9PtxFRkPkmPoGXFM3jnJLm3SqwlKQq4mGJkk3Et2BEAIyR4Q8+wz7e/TuontPIJFzzh48OefzX4oCdPTJUZJYPpKB5ZjG0YE8GrCK6ATTp3gJ+58avziQdnQakr8Qn8jUUl+xqLA2G7Yfdd5jS/wvY/vJdmEpYYltlLIdxpdUF4FImSqonF+s0xL/Q28dEXD/j3HOxLx+IFKwt+PM18Q8ZRJiLPiVxjbQ/diT5jeUALZndoHfvtyZRtIFRevZB35AwhVZqfmhrUxdQGp5tkmhMqj+nm7POKsCdkNNZ6QDYcyRUyoRyjVYmHMG5zPbTCbU/T2p++e5LVnaI0ZH2Iiqp2XwKD9H0ooux3JCKRYOH7sKSBcluMTJsPzV+R83bRX4lSq/QMYt5w3R8xxxmn+TGRz7G/vr3O2+FWzkFGoY2l2yN6FQ4jAS26OzRevn50Tm06t0QuoR+FGPOJQztjL4SPQFLZd3VJqExKTHRljIQdz7Ddj84yp77ouYatLCf107ISgZFtDLO6PGMxgDmMalhAHbIg14wmvdAakG0iY5m3MqCQHylrF0sqqYK08u8ffj0995ScSMmprbimUdPYlC0tKje24WV05rgeFjOZaHjGVtkuG3MK8Xsig51tWfmGk4CbDlftlst+zWydg23oOQ2xWhGc1BX23rceZ7VXVNmk9N3XRGz47HAYEjqFaUvFEU6nl0OOdZj0DVNObGPPja+ZfMcjO7CLNc9m40L5yxe+j6R8DKkDO+qcBEpecvjBVi6JyUhMGXD8HdYpIdH7w3pGLDhmg6MhjdVHeta8l9cajo0sbqPtEiLr8v6+UEgLNeX1JYLYH998N9OfEvbqOO/Fx7WV0m4s+CKEmvGmjGVRNK2373Dbc5xlrH8MH+Ng0he5nyrFRg5YSTOGTR9T8anXlHu7lRE41q5kfMzyupgIuSxeoaxwvyodl6GOW4N7xbg/waoUrJbBe9M9QiHrta4o5OjA6m4sm2sskV1Mc+VPJiuD15MH2+F2SxsnMyZXLvSRDeXhG108dFuSfhxvAXruIO4MaTBq1cSO6Wk0b3kt0YsurdAUFN2lMK8jllo4blvHdo9PFlAsN7BByJHzpJ1HV15mNLi3aMbY6jiu8TKFjCWELvcjbb/EUrtvRfUMnvfcMMsV7ArXFWokOs3aqakvyIpnmi/ibalngkmXEOqiVxR2Q9xvx7VsHEq5YPBeny86IpqPPPne3VPZBROVlw0+a8Yu9+PCkaUyx4znGed+zjY3rOzAFc+5pbPyFTX7yAQb1kxTHXR7FldcxgXXEi0pZzqHHnhtY/ZXR4bowmo3372TXnS6GFG8CWAhgTl2VRpvmrR4I19oeD8ibFBvNPIrGgr1gzmKzEoir9Npyit0M1morBTdwrFfnp6nqT0WtFDWWW416leJhRECleLVMr3RxIFq4eF1SB+gktSOochPRL+ZcoxfI6KbsxnRXN6weMj/AQeD020iIxBGAAAAAElFTkSuQmCC
uniform sampler2D u_tex1; // https://prismjs.com/assets/img/spectrum.png

// these are input-knobs (0.0 - 1.0)
uniform float u_x0;
uniform float u_x1;
uniform float u_x2;
uniform float u_x3;

void main() {
  // scaled pixel-position
  vec2 uv = gl_FragCoord.xy / u_resolution.xy;

  // input 1 is mix-fader for tex0/1
  gl_FragColor = mix(texture2D(u_tex0, uv), texture2D(u_tex1, uv), u_x0);
}

No error, but the texture is not updated (even though other shader code is.)

If I set the initial shader to that, it also doesn't seem to use it (loads 2 copies of spectrum image.)

Update: I managed to get it sort of working with images, with this:

function setTex (index, url) {
  const r = new RegExp(`^[ \t]*uniform *sampler2D *u_tex${index} *;(.*)`, 'gm')
  const m = r.exec($code.value)
  const newline = `uniform sampler2D u_tex${index}; // ${url}`
  if (m && m[0]) {
    $code.value = $code.value.replace(r, newline)
    $code.dispatchEvent(new Event('input'))
    preview.loadTexture(`u_tex${index}`, url, {})
  } else {
    $code.value = `${newline}\n\n${$code.value}`
    $code.dispatchEvent(new Event('input'))
    preview.loadTexture(`u_tex${index}`, url, {})
  }
}

Basically, I need to update the shader first (adding the base64 URL comment, for code-sharing) and then force it to update texture with preview.loadTexture.

Initial base64-url comment is not working still, and videos still fail, and they are still swapped around, but this gets it loading them.

I also switched to objectURLs over base64, to make the code lighter:

// this works initially
uniform sampler2D u_tex0; // hypnocat.png
uniform sampler2D u_tex1; // https://prismjs.com/assets/img/spectrum.png

// this sort of works
// but hypnocat moved to u_tex0, and new objectURL is on u_tex1
uniform sampler2D u_tex0; // blob:http://localhost:5173/81e99c3d-046a-4ca7-8b34-b71887138af0
uniform sampler2D u_tex1; // https://prismjs.com/assets/img/spectrum.png

[konsumer]

I think the buggy image stuff is actually related to video. It bugs out the texture buffers, and swaps them around, but if I stick to just uploaded images, it seems fine.

Here the first image is the skull thing, and the second is the clown thing, so these are correct, so i think the issue is just videos.

You can play with it here, if you want to.

[konsumer]

I ended up just making my own much simpler glsl-canvas with threejs. Seems to work really well.

For video I am doing this:

  loadTexture (name, url, type) {
    if (type.startsWith('image')) {
      this.material.uniforms[name].value = new THREE.TextureLoader().load(url)
    } else if (type.startsWith('video')) {
      const video = document.createElement('video')
      video.src = url
      video.muted = true
      video.loop = true
      video.play()
      this.material.uniforms[name].value = new THREE.VideoTexture(video)
    }
  }