Frame Buffer size and Window size differences on macOS

[wesleymth]

Hello,

I am on macOS 13.3.1. And using these versions:

mujoco                        2.3.1.post1
mujoco-python-viewer          0.1.2

I am currently working on a research project where I need to retrieve the data from an offscreen camera.

My problem is the following: My window's framebuffer size is exactly twice the window's size. I have a predefined resolution that I need to use for downstream analysis which corresponds to the window size. I extract the pixels of my offscreen camera using read_pixels(...) but the dimensions are 2 times too big for the first two dimensions corresponding to (width , height)

Here is a very shortened version of the code:

import matplotlib.pyplot as plt
import mujoco_viewer
import numpy as np
import glfw

import my_project

resolution = np.array([480, 640]) # (width, height)

offscreen = mujoco_viewer.MujocoViewer(
        my_project.interface.model,
        my_project.interface.data,
        "offscreen",
        width=resolution[0],
        height=resolution[1],
    )

camera_feedback = np.zeros((resolution[1], resolution[0], 3)) # (640,480,3)

print("window_size:", glfw.get_window_size(offscreen.window)) # (480, 640)
print("framebuffer_size:", glfw.get_framebuffer_size(offscreen.window)) # (960, 1280)
print("window_content_scale:" , glfw.get_window_content_scale(offscreen.window)) # (2.0, 2.0)

print("read_pixels(...).shape:", offscreen.read_pixels(camid=0).shape) # (1280,960,3)

camera_feedback[:] = offscreen.read_pixels(camid=0)

# plot it
plt.imshow(camera_feedback / 255)
# Followed by downstream analysis...

Outputs:

window_size: (480, 640)
framebuffer_size: (960, 1280)
window_content_scale: (2.0, 2.0)
read_pixels(...).shape: (1280,960,3)
Traceback (most recent call last):
        camera_feedback[:] = offscreen.read_pixels(camid=0)
ValueError: could not broadcast input array from shape (1280,960,3) into shape (640,480,3)

I could fix my problem by down sampling the output of offscreen.read_pixels(camid=0) by the window content scale: offscreen.read_pixels(camid=0)[::2, ::2] or more elaborately:

window_content_scale = np.array(glfw.get_window_content_scale(offscreen.window), dtype='int')
camera_feedback[:] = offscreen.read_pixels(camid=0)[::window_content_scale[0], ::window_content_scale[1]]

But this is still not the ideal situation as potentially other machines have different DPI and scaling settings.

I was wondering wether there might be an option to either set the frame buffer size to be exactly the same as the window size when creating the MujocoViewer object, or alternatively to set the window content scaling to be 1:1, or to downsample automatically when calling read_pixels(...), or any other options?

I already tried adding various macOS specific glfw.window_hints(...) in mujoco_viewer/mujoco_viewer.py line 37 with no luck:

glfw.window_hint(glfw.COCOA_RETINA_FRAMEBUFFER, glfw.FALSE)
glfw.window_hint(glfw.DOUBLEBUFFER, glfw.FALSE)

Thanks in advance!