Headless mode option

[wangce94]

I am trying to run fluidlab on a remote pc. As such, I am trying to save the frames generated using the save variable in fluidlab.optimizer.recorder.py. However, I am running into an issue regarding the display:

(fluidlab) cewang@blackcoffee:~/FluidLab_SLURM$ python fluidlab/run.py --cfg_file configs/exp_latteart.yaml --record --renderer_type GL
[Taichi] version 1.1.0, llvm 10.0.0, commit f5bb6464, linux, python 3.7.13
[Taichi] Starting on arch=cuda
===>  TaichiEnv created.
===>    60000 particles of     milk  nowhere added.
===>    55480 particles of   coffee cylinder added.
===>  LatteArtEnv built successfully.
qt.qpa.xcb: could not connect to display
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/cewang/miniconda3/envs/fluidlab/lib/python3.7/site-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.

Aborted (core dumped)

I am wondering if there is a headless mode where I can bypass the cv2.imshow, or will I have to implement it myself. Thank you!

[zhouxian]

I think you can follow this route: https://github.com/zhouxian/FluidLab/blob/56348da2ac22598fb46097ded0ee85b805602344/fluidlab/optimizer/recorder.py#L52C1-L52C1

[wangce94]

I'm not quite sure if I understand what you mean. Do you mean that I should edit the render_frame() function in gl_renderer.py to disable imshow?

[zhouxian]

When calling render_frame(), you can specify the render mode: either 'human' or 'rgb_array'. The latter will simply return a rendered image array without showing a window.

For details, see: https://github.com/zhouxian/FluidLab/blob/56348da2ac22598fb46097ded0ee85b805602344/fluidlab/fluidengine/renderers/gl_renderer.py#L215

[zhouxian]

BTW, I am not sure what your exact need is and your timeline, but in case your need for fluid simulation is not urgent, this repo(https://github.com/Genesis-Embodied-AI/Genesis) might be of interest to you. It's a superset of FluidLab with a lot more features, and way easier to use. We expect to release it in 2~3 months.

[wangce94]

I took a look at the code, and under 'rgb_array', there is still imshow under mode == 'rgb_array':

        if mode == 'human':
            cv2.imshow('FluidLab', img[..., ::-1])
            cv2.setWindowTitle('FluidLab', f'FluidLab [FPS: {self.fps:.2f}]')
            cv2.waitKey(1)
            return None

        elif mode == 'rgb_array':
            cv2.imshow('FluidLab', img[..., ::-1])
            cv2.setWindowTitle('FluidLab', f'FluidLab [FPS: {self.fps:.2f}]')
            cv2.waitKey(1)
            return img

[zhouxian]

I see. I don't remember exactly why it's there, but i think you can comment out those code. The image is generated here: https://github.com/zhouxian/FluidLab/blob/56348da2ac22598fb46097ded0ee85b805602344/fluidlab/fluidengine/renderers/gl_renderer.py#L210