How to Capture the screen of 3D object with camera (or webcam) content as the background?

[mrizkiaiman]

Hello,

I want to ask, is it possible to take a screenshot of 3D with the camera content as a background? I manage to find a way to screenshot the "3D only" programmatically thanks to A-frame documentation, but I wonder if it is possible to take a screenshot with the content from the camera/webcam as the background.

Thanks^^

[demisquare]

In HTML file:

<button (click)="screenshot()"> Screenshot </button>
<canvas #canvas id="canvas" width="640" height="480" style="display:none"></canvas>

<div class="fullscreen" #screenRef>
  <a-scene #scene
     ...
   </a-scene>
</div>

In the script (it's Angular, but you can translate in js easily):

public source?: string;

@ViewChild('scene', { read: ElementRef }) sceneRef!: ElementRef;
@ViewChild('screenRef', { static: true }) screenRef: any;
@ViewChild("canvas") public canvas!: ElementRef;

  public screenShot(){

    const video = Array.from((this.screenRef.nativeElement as HTMLDivElement).children)
    .find(child => child.tagName == 'VIDEO')! as HTMLVideoElement

    video.pause();
    const screen = this.sceneRef.nativeElement.components.screenshot.getCanvas('perspective') as HTMLCanvasElement;

    const videoWidth = (video.srcObject! as MediaStream).getVideoTracks()[0].getSettings().width ?? 640;
    const videoHeight = (video.srcObject! as MediaStream).getVideoTracks()[0].getSettings().height ?? 480;
    const context = this.canvas.nativeElement.getContext("2d")
    const canvas = this.canvas.nativeElement as HTMLCanvasElement;

    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;

    let outputRatio = canvas.width/canvas.height

    if (outputRatio > 1 && window.innerWidth > videoWidth) {
      canvas.width = videoWidth;
      canvas.height = canvas.height*(videoWidth/window.innerWidth)
    } else if (outputRatio < 1 && window.innerHeight > videoHeight) {
      canvas.height = videoHeight;
      canvas.width = canvas.width*(videoHeight/window.innerHeight)
    }

    context.drawImage(video,
      (videoWidth-canvas.width)/2, (videoHeight-canvas.height)/2,
      canvas.width, canvas.height,
      0, 0,
      canvas.width, canvas.height
      );
      context.drawImage(screen,
        0, 0,
        canvas.width, canvas.height
        )

        let base64 = this.canvas.nativeElement.toDataURL("image/png")
        this.source = base64;
        video.play()

}

  public downloadImg(){
    let a = document.createElement("a");
    a.href = this.source!;
    a.download = "Mind_AR_screenshot.png";
    a.click();
    this.deleteSource()
  }

  public deleteSource(){
    this.source = undefined
  }

[mrizkiaiman]

In HTML file:

<button (click)="screenshot()"> Screenshot </button>
<canvas #canvas id="canvas" width="640" height="480" style="display:none"></canvas>

<div class="fullscreen" #screenRef>
  <a-scene #scene
     ...
   </a-scene>
</div>

In the script (it's Angular, but you can translate in js easily):

public source?: string;

@ViewChild('scene', { read: ElementRef }) sceneRef!: ElementRef;
@ViewChild('screenRef', { static: true }) screenRef: any;
@ViewChild("canvas") public canvas!: ElementRef;

  public screenShot(){

    const video = Array.from((this.screenRef.nativeElement as HTMLDivElement).children)
    .find(child => child.tagName == 'VIDEO')! as HTMLVideoElement

    video.pause();
    const screen = this.sceneRef.nativeElement.components.screenshot.getCanvas('perspective') as HTMLCanvasElement;

    const videoWidth = (video.srcObject! as MediaStream).getVideoTracks()[0].getSettings().width ?? 640;
    const videoHeight = (video.srcObject! as MediaStream).getVideoTracks()[0].getSettings().height ?? 480;
    const context = this.canvas.nativeElement.getContext("2d")
    const canvas = this.canvas.nativeElement as HTMLCanvasElement;

    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;

    let outputRatio = canvas.width/canvas.height

    if (outputRatio > 1 && window.innerWidth > videoWidth) {
      canvas.width = videoWidth;
      canvas.height = canvas.height*(videoWidth/window.innerWidth)
    } else if (outputRatio < 1 && window.innerHeight > videoHeight) {
      canvas.height = videoHeight;
      canvas.width = canvas.width*(videoHeight/window.innerHeight)
    }

    context.drawImage(video,
      (videoWidth-canvas.width)/2, (videoHeight-canvas.height)/2,
      canvas.width, canvas.height,
      0, 0,
      canvas.width, canvas.height
      );
      context.drawImage(screen,
        0, 0,
        canvas.width, canvas.height
        )

        let base64 = this.canvas.nativeElement.toDataURL("image/png")
        this.source = base64;
        video.play()

}

  public downloadImg(){
    let a = document.createElement("a");
    a.href = this.source!;
    a.download = "Mind_AR_screenshot.png";
    a.click();
    this.deleteSource()
  }

  public deleteSource(){
    this.source = undefined
  }

Thank you very much, it works 🙏

[mr339]

In HTML file:

<button (click)="screenshot()"> Screenshot </button>
<canvas #canvas id="canvas" width="640" height="480" style="display:none"></canvas>

<div class="fullscreen" #screenRef>
  <a-scene #scene
     ...
   </a-scene>
</div>

In the script (it's Angular, but you can translate in js easily):

public source?: string;

@ViewChild('scene', { read: ElementRef }) sceneRef!: ElementRef;
@ViewChild('screenRef', { static: true }) screenRef: any;
@ViewChild("canvas") public canvas!: ElementRef;

  public screenShot(){

    const video = Array.from((this.screenRef.nativeElement as HTMLDivElement).children)
    .find(child => child.tagName == 'VIDEO')! as HTMLVideoElement

    video.pause();
    const screen = this.sceneRef.nativeElement.components.screenshot.getCanvas('perspective') as HTMLCanvasElement;

    const videoWidth = (video.srcObject! as MediaStream).getVideoTracks()[0].getSettings().width ?? 640;
    const videoHeight = (video.srcObject! as MediaStream).getVideoTracks()[0].getSettings().height ?? 480;
    const context = this.canvas.nativeElement.getContext("2d")
    const canvas = this.canvas.nativeElement as HTMLCanvasElement;

    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;

    let outputRatio = canvas.width/canvas.height

    if (outputRatio > 1 && window.innerWidth > videoWidth) {
      canvas.width = videoWidth;
      canvas.height = canvas.height*(videoWidth/window.innerWidth)
    } else if (outputRatio < 1 && window.innerHeight > videoHeight) {
      canvas.height = videoHeight;
      canvas.width = canvas.width*(videoHeight/window.innerHeight)
    }

    context.drawImage(video,
      (videoWidth-canvas.width)/2, (videoHeight-canvas.height)/2,
      canvas.width, canvas.height,
      0, 0,
      canvas.width, canvas.height
      );
      context.drawImage(screen,
        0, 0,
        canvas.width, canvas.height
        )

        let base64 = this.canvas.nativeElement.toDataURL("image/png")
        this.source = base64;
        video.play()

}

  public downloadImg(){
    let a = document.createElement("a");
    a.href = this.source!;
    a.download = "Mind_AR_screenshot.png";
    a.click();
    this.deleteSource()
  }

  public deleteSource(){
    this.source = undefined
  }

yes, it works on plain js as well but can an image which is inside a div outside of the a-scene be also captured along with the background? `

    </div>`

[demisquare]

yes, it works on plain js as well but can an image which is inside a div outside of the a-scene be also captured along with the background? <div class="rear_view"> <img id="image" src="/image.png" /> </div>

I don't know honestly.

[mr339]

yes, it works on plain js as well but can an image which is inside a div outside of the a-scene be also captured along with the background? <div class="rear_view"> <img id="image" src="/image.png" /> </div>

I don't know honestly.

Ah, no probs, but is there a way to make the output image a bit clearer because it seems a bit blurry for me or does it depend on the mobile device? i tried changing the dimensions but it just resulted in missplaced objects.

[demisquare]

yes, it works on plain js as well but can an image which is inside a div outside of the a-scene be also captured along with the background? <div class="rear_view"> <img id="image" src="/image.png" /> </div>

I don't know honestly.

Ah, no probs, but is there a way to make the output image a bit clearer because it seems a bit blurry for me or does it depend on the mobile device? i tried changing the dimensions but it just resulted in missplaced objects.

Sadly I didn't managed to obtain a better image than that.

[mr339]

yes, it works on plain js as well but can an image which is inside a div outside of the a-scene be also captured along with the background? <div class="rear_view"> <img id="image" src="/image.png" /> </div>

I don't know honestly.

Ah, no probs, but is there a way to make the output image a bit clearer because it seems a bit blurry for me or does it depend on the mobile device? i tried changing the dimensions but it just resulted in missplaced objects.

Sadly I didn't managed to obtain a better image than that.

ah no worries :D