OnResize and other functions

[kalwalt]

Probably the code need an onResize() function and maybe onReady().

[albjeremias]

it is not very clear what needs to be done in this issue.

[kalwalt]

I was refering to some AR.js internal functions and that maybe could be implemented. But i don't know if it worth to develop a onResize function as in AR.js: https://github.com/AR-js-org/AR.js/blob/aaad9847f67a4738b00724c38d9f501d78e0a8af/three.js/src/threex/threex-artoolkitsource.js#L392 . That should resize the domElement when the window is resized. But i think not sure if it make sense in ARnft.

[kalwalt]

The problem arise when you switch on a mobile device from portrait mode to landscape mode. Without resizing the mesh/model is off or not placed correctly onto the marker. We shouold provide a onResize function maybe with a callback, within you can customize your code.

[konopimi]

Hello, I'am working on a project using this amazing tool, I have been strugling to finishing it due to the offset + video texture pixelation that arises from rotating the device screen from portrait to landscape.

Ultimately I will try to wipe off everything "onorientationchange" and try to restart the worker, the renderer and the scene, I dont want to go this way, it seems "hacky" as I think I could tweak some configuration or run some "refresh" piece of code to make the matrix match. This is the last "toDo" I have waiting to be solved, please guide me on how can I aproach this problem. Thank you very much for your work.

[kalwalt]

Hello, I'am working on a project using this amazing tool, I have been strugling to finishing it due to the offset + video texture pixelation that arises from rotating the device screen from portrait to landscape.

Ultimately I will try to wipe off everything "onorientationchange" and try to restart the worker, the renderer and the scene, I dont want to go this way, it seems "hacky" as I think I could tweak some configuration or run some "refresh" piece of code to make the matrix match. This is the last "toDo" I have waiting to be solved, please guide me on how can I aproach this problem. Thank you very much for your work.

Hi @konopimi welcome to webarkit! I think i will add the onResize feature, but i don't know yet exactly how and when. For sure we should also let developer to use his/her own onResize function or use a predefined resizing.

What do you think?

[konopimi]

Amazing @kalwalt , thanks. I have actually achieved to develop a solution yesterday. Now my video has consistent dimensions and position while changing device orientation on mobile. Managed to fix it by updating some values on renderer, camera, arnft, an extra message, plus updating matrix, seems the order of the operations is important. Restarting the worker was not necesary. Additionally I have a normalized dimensions on markers with different dpi, as well as a calculated video ratio for different video sources. I will share my approach to this solution soon.

[kalwalt]

@konopimi have you done some progresses on this topic?

[konopimi]

Hey, hello, @kalwalt. I've been doing some testing, and came to some conclusions about my solution. Sorry for the late response. Most of my solution code was a bloat, I found the minimal piece of code to solve this issue.

I've really been working on a fork of this project based on the experiments of: https://github.com/ThomasRutzer/joefix-ar-radio

The reason is, I found his repository to be more stable on the recognition of the marker, after some test comparing both solutions.

What I did was to take some initializing part of "arnft" code and isolated some part of the process to a function Then I re-run it using the event "onResize" or "onOrientationChange". This part needs to be refreshed for the algorithm to asume new screen dimensions. Here we are using 'this' notation. But it is the same logic for your example.

fixRotation() {
    this.inputWidth = this.video.videoWidth;
    this.inputHeight = this.video.videoHeight;
    const pScale = 320 / Math.max(this.inputWidth, (this.inputHeight / 3) * 4);
    const sScale = isMobile() ? window.outerWidth / this.inputWidth : 1;
    const sw = this.inputWidth * sScale;
    const sh = this.inputHeight * sScale;
    this.w = this.inputWidth * pScale;
    this.h = this.inputHeight * pScale;
    this.pw = Math.max(this.w, (this.h / 3) * 4);
    this.ph = Math.max(this.h, (this.w / 4) * 3);
    this.ox = (this.pw - this.w) / 2;
    this.oy = (this.ph - this.h) / 2;
    this.canvasProcess.style.clientWidth = this.pw + 'px';
    this.canvasProcess.style.clientHeight = this.ph + 'px';
    this.canvasProcess.width = this.pw;
    this.canvasProcess.height = this.ph;
    this.renderer.setSize(sw, sh, false);
  }

I've found an issue on on Safari. When you change orientation, the values provided by window are not inmediately refreshed. I am speaking about information on window size. So, in order to fix it. I've made a conditional, which checks if the platform is Safari and then, acordingly I use a timeout to make it work. In this way:

  try {
    let userAgentString = navigator.userAgent;
    let safariAgent = userAgentString.indexOf('Safari') > -1;
    let chromeAgent = userAgentString.indexOf('Chrome') > -1;
    if (chromeAgent && safariAgent) safariAgent = false ;  
 window.addEventListener(
      'orientationchange',
      // 'resize',
      safariAgent ? () => setTimeout(fixRotation, 200) : fixRotation
    );
  } catch (e) {
    alert(e);
  }

It is important to use the timeout on Safari, but is also important to not use it on chrome, otherwise it will not work.

If in doubt Ping me I'll be up to explain further. https://konopimi.xyz wapp or signal: +57 315 3410282

You can test my solution here, try to turn the device:

https://magia.blessaccesorios.com/6Hj80QUxtX

You will need to use this picture in order to test it:

https://imgs.search.brave.com/5ut07zy7Xf3u0_XjSwIzdowQQWK_Tr2HK3z6ep1ZKg8/rs:fit:508:225:1/g:ce/aHR0cHM6Ly90c2Uz/Lm1tLmJpbmcubmV0/L3RoP2lkPU9JUC5o/VU1VUklHSE9SVHdi/dDdpdWoza29BSGFH/NiZwaWQ9QXBp

Thanks to @ThomasRutzer for sharing his project aswell

[konopimi]

Thank you so much. This project has been a huge help for me to deploy a solution for my friend Shop. https://blessaccesorios.com/products/foto-magica?_pos=1&_sid=a887f92ad&_ss=r

[kalwalt]

Thank you @konopimi for this, i will try that when i will have time. Didn't know about that repository and react-arnft. 🙂

[kalwalt]

I tested your solution with my Oppo A72 (android device) and works perfectly!

What I did was to take some initializing part of "arnft" code and isolated some part of the process to a function Then I re-run it using the event "onResize" or "onOrientationChange". This part needs to be refreshed for the algorithm to asume new screen dimensions. Here we are using 'this' notation. But it is the same logic for your example.

I think you are refering to the arNFT_initialize_raw_example.html right?

[konopimi]

Yes pinball

[kalwalt]

I've really been working on a fork of this project based on the experiments of: https://github.com/ThomasRutzer/joefix-ar-radio

The reason is, I found his repository to be more stable on the recognition of the marker, after some test comparing both solutions.

I checked his code, the fact is the linear interpolation routine is inside the Worker, instead ARnft send raw data to ARnft-threejs where is done the interpolation routine. I could move this part of the code inside the worker and test the difference. Anyway i'm trying to make a simple example based on your snippets, i hope to show you soon. :smile:

[konopimi]

Ooh nice, I was not aware of this difference. For this part of the code, at evaluating the matrix and interpolating, I have modded it to use Promise.all(...). I am not sure if this is doing much, but I think it may give some boost. Also, I tried to lower the interpolation factor to make the video more 'sticky' but this introduces some trembling on some devices, more on the rotation than the position, so I am trying to have two separate interpolation factors for a better tunning, one for the rotational parts of the matrix and other for the position. Would you somehow indicate me for some ease, how the matrix is composed, so I can properly assign the coefficients? Thanks.

[konopimi]

@kalwalt I am delivering the markers using an admin panel. There, using an iframe I am using the arnft creator, I wan to directly embed it to the app without an iframe, how should I do it.