An error occurred(Segmentation fault) while loading the model

[GuaGod]

[dendenxu]

Hi @GuaGod , looks like the model has already been successfully loaded since the log you highlighted should be printed after model loading. The segmentation fault is most likely OpenGL-related so I might need your assistance on checking out your environment:

1. If on Linux, could you try running glxgears and check whether the window is displayed correctly
2. Try running evc-gui, which should open a black window for you with some controls, and check whether that works
3. Check OpenGl version by glxinfo -B to see if any error is shown and check PyOpenGL installation, which should be of version 3.1.7 as of today. If we're on 3.1.0, update using pip install -U pyopengl

[GuaGod]

1. The window is displayed correctly after glxgears.
2. An error occured after evc-gui
3. 

[dendenxu]

That's strange 😶‍🌫️ . Your OpenGL setup looks fine. Could you try whether the off-screen rendering works?

evc -t test -c configs/projects/realtime4dv/rendering/4k4d_sport1.yaml,configs/specs/eval.yaml

Another thing to try is to disable the compatibiliy profile window hint by commenting these two lines here:

[YongmoonPark]

Hello, and thank you for your excellent work.

In my situation, off-screen rendering works fine, but I encounter an error when attempting rendering with a GUI. I followed your suggestion to comment out two lines, but I am still experiencing issues running in GUI mode.

Here are the details of my environment: OS: Ubuntu 22.04 LTS GPU: NVIDIA GeForce RTX 3090 Driver Version: 545.23.08 CUDA Version: 11.8"

Package                   Version            Editable project location
------------------------- ------------------ ------------------------------------------
absl-py                   2.0.0
addict                    2.4.0
ansi2html                 1.9.1
antlr4-python3-runtime    4.9.3
anyio                     4.2.0
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asttokens                 2.4.1
async-lru                 2.0.4
attrs                     23.2.0
autopep8                  2.0.4
Babel                     2.14.0
beautifulsoup4            4.12.2
black                     23.12.1
bleach                    6.1.0
blessed                   1.20.0
blinker                   1.7.0
cachetools                5.3.2
calmsize                  0.1.3
certifi                   2022.12.7
cffi                      1.16.0
charset-normalizer        2.1.1
chumpy                    0.70
clang-format              17.0.6
click                     8.1.7
cloudpickle               3.0.0
cmake                     3.25.0
comm                      0.2.1
commentjson               0.9.0
ConfigArgParse            1.7
contourpy                 1.2.0
cuda-python               12.3.0
cycler                    0.12.1
dash                      2.14.2
dash-core-components      2.0.0
dash-html-components      2.0.0
dash-table                5.0.0
debugpy                   1.8.0
decorator                 5.1.1
defusedxml                0.7.1
detectron2                0.6                /media/ym/linux-data/_src/_ref/detectron2
diff-gauss                1.0.5
docstring-parser          0.15
dotmap                    1.3.30
easyvolcap                0.0.0              /media/ym/linux-data/_src/_test/EasyVolcap
einops                    0.7.0
et-xmlfile                1.1.0
exceptiongroup            1.2.0
executing                 2.0.1
fast-autocomplete         0.9.0
fastjsonschema            2.19.1
filelock                  3.9.0
Flask                     3.0.0
flatbuffers               23.5.26
fonttools                 4.47.0
fqdn                      1.5.1
func-timeout              4.3.5
fvcore                    0.1.5.post20221221
glfw                      2.6.4
google-auth               2.26.1
google-auth-oauthlib      1.2.0
gpustat                   1.2.dev19+gf4cd639
grpcio                    1.60.0
h5py                      3.10.0
human-body-prior          2.2.2.0
hydra-core                1.3.2
idna                      3.4
imageio                   2.33.1
imgui-bundle              1.2.1
importlib-metadata        7.0.1
iopath                    0.1.9
ipdb                      0.13.13
ipykernel                 6.28.0
ipython                   8.20.0
ipywidgets                8.1.1
isoduration               20.11.0
itsdangerous              2.1.2
jedi                      0.19.1
Jinja2                    3.1.2
joblib                    1.3.2
json5                     0.9.14
jsonpointer               2.4
jsonschema                4.20.0
jsonschema-specifications 2023.12.1
jupyter                   1.0.0
jupyter_client            8.6.0
jupyter-console           6.6.3
jupyter_core              5.7.1
jupyter-events            0.9.0
jupyter-lsp               2.2.1
jupyter_server            2.12.3
jupyter_server_terminals  0.5.1
jupyterlab                4.0.10
jupyterlab_pygments       0.3.0
jupyterlab_server         2.25.2
jupyterlab-widgets        3.0.9
kiwisolver                1.4.5
kornia                    0.7.1
lark-parser               0.7.8
lazy_loader               0.3
lit                       15.0.7
lpips                     0.1.4
Markdown                  3.5.2
markdown-it-py            3.0.0
MarkupSafe                2.1.3
matplotlib                3.8.2
matplotlib-inline         0.1.6
mdurl                     0.1.2
mediapipe                 0.10.9
memory-tempfile           2.2.3
mistune                   3.0.2
mpmath                    1.3.0
msgpack                   1.0.7
munch                     4.0.0
mypy-extensions           1.0.0
nbclient                  0.9.0
nbconvert                 7.14.0
nbformat                  5.9.2
nest-asyncio              1.5.8
networkx                  3.0
ninja                     1.11.1.1
notebook                  7.0.6
notebook_shim             0.2.3
numpy                     1.24.1
nvdiffrast                0.3.1              /media/ym/linux-data/_src/_ref/nvdiffrast
nvidia-ml-py              12.535.133
oauthlib                  3.2.2
omegaconf                 2.3.0
open3d                    0.18.0
opencv-contrib-python     4.9.0.80
opencv-python             4.9.0.80
openpyxl                  3.1.2
overrides                 7.4.0
packaging                 23.2
pandas                    2.1.4
pandocfilters             1.5.0
parso                     0.8.3
pathspec                  0.12.1
pdbr                      0.8.8
pexpect                   4.9.0
Pillow                    9.5.0
pip                       23.3.1
platformdirs              4.1.0
plotly                    5.18.0
plyfile                   1.0.3
portalocker               2.8.2
prometheus-client         0.19.0
prompt-toolkit            3.0.43
protobuf                  3.20.3
psutil                    5.9.7
ptyprocess                0.7.0
pure-eval                 0.2.2
pyasn1                    0.5.1
pyasn1-modules            0.3.0
pycocotools               2.0.7
pycodestyle               2.11.1
pycparser                 2.21
pyglfw                    0.2.2
PyGLM                     2.7.1
Pygments                  2.17.2
PyMCubes                  0.1.4
pymeshlab                 2023.12
pyntcloud                 0.3.1
PyOpenGL                  3.1.7
pyparsing                 3.1.1
pyperclip                 1.8.2
pyquaternion              0.9.9
python-dateutil           2.8.2
python-json-logger        2.0.7
pytorch-memlab            0.3.0
pytorch-msssim            1.0.0
pytorch3d                 0.7.5              /media/ym/linux-data/_src/_ref/pytorch3d
PyTurboJPEG               1.7.3
pytz                      2023.3.post1
PyYAML                    6.0.1
pyzmq                     25.1.2
qtconsole                 5.5.1
QtPy                      2.4.1
referencing               0.32.1
requests                  2.31.0
requests-oauthlib         1.3.1
retrying                  1.3.4
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rich                      13.7.0
rpds-py                   0.16.2
rsa                       4.9
ruamel.yaml               0.18.5
ruamel.yaml.clib          0.2.8
scikit-image              0.22.0
scikit-learn              1.3.2
scipy                     1.11.4
Send2Trash                1.8.2
setuptools                68.2.2
shtab                     1.6.5
simple-knn                0.0.0
six                       1.16.0
smplx                     0.1.28
sniffio                   1.3.0
sounddevice               0.4.6
soupsieve                 2.5
stack-data                0.6.3
sympy                     1.12
tabulate                  0.9.0
tenacity                  8.2.3
tensorboard               2.15.1
tensorboard-data-server   0.7.2
tensorboardX              2.6.2.2
termcolor                 2.4.0
terminado                 0.18.0
threadpoolctl             3.2.0
tifffile                  2023.12.9
timg                      1.1.6
tinycss2                  1.2.1
tinycudann                1.6
tomli                     2.0.1
torch                     2.0.1+cu118
torch-scatter             2.1.2
torch-tb-profiler         0.4.3
torchaudio                2.0.2+cu118
torchdiffeq               0.2.3
torchmcubes               0.1.0
torchvision               0.15.2+cu118
tornado                   6.4
tqdm                      4.66.1
traitlets                 5.14.1
trimesh                   4.0.8
triton                    2.0.0
types-python-dateutil     2.8.19.20240106
typing_extensions         4.4.0
tyro                      0.6.3
tzdata                    2023.4
ujson                     5.9.0
uri-template              1.3.0
urllib3                   1.26.13
wcwidth                   0.2.13
webcolors                 1.13
webencodings              0.5.1
websocket-client          1.7.0
Werkzeug                  3.0.1
wheel                     0.41.2
widgetsnbextension        4.0.9
xatlas                    0.0.8
yacs                      0.1.8
yapf                      0.40.2
zipp                      3.17.0

[dendenxu]

Hi @YongmoonPark thanks for trying out our code! Does the command also throw a segmentation fault in your case?

[dendenxu]

Hey guys @GuaGod @YongmoonPark , could you try replacing the evc command with python -q -X faulthandler easyvolcap/scripts/main.py to let the program print more useful information for the segmentation fault?

For example, try running python -q -X faulthandler easyvolcap/scripts/main.py -t gui -c configs/specs/gui.yaml.

[YongmoonPark]

Thank you, @dendenxu, for your prompt and kind response.

In my case, off-screen rendering works fine, but when I attempt rendering with a GUI, I encounter the same error. I tried following your instructions to comment out two lines, but I'm still facing an error when running in GUI mode. Here is the error message I received:

When executing the command: python -q -X faulthandler easyvolcap/scripts/main.py -t gui -c configs/specs/gui.yaml I encountered the following error:

2024-01-15 09:32:27.317839 __main__ -> preflight: Starting experiment: base, command: gui main.py:80
2024-01-15 09:32:27.318702 easyvolcap.utils.console_utils -> inner: Runtime exception: 'batch_size' console_utils.py:341
AttributeError: 'batch_size'
*** 'batch_size'
> /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/base_utils.py(101)__getitem__()

whole error message

``` (easyvolcap) ym@ym-linux:.../_test/EasyVolcap$ python -q -X faulthandler easyvolcap/scripts/main.py -t gui -c configs/specs/gui.yaml. 2024-01-15 09:32:27.317839 __main__ -> preflight: Starting experiment: base, command: gui main.py:80 2024-01-15 09:32:27.318702 easyvolcap.utils.console_utils -> inner: Runtime exception: 'batch_size' console_utils.py:341 ╭─────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/base_utils.py:99 in __getitem__ │ │ │ │ ❱ 99 │ │ │ return dict.__getitem__(self, key) │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ KeyError: 'batch_size' During handling of the above exception, another exception occurred: ╭─────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/console_utils.py:338 in inner │ │ │ │ ❱ 338 │ │ │ return func(*args, **kwargs) │ │ │ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:56 in inner │ │ │ │ ❱ 56 │ │ return call_from_cfg(func, cfg) │ │ │ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:47 in call_from_cfg │ │ │ │ ❱ 47 │ return func(**call_args) │ │ │ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/scripts/main.py:158 in test │ │ │ │ ❱ 158 │ val_dataloader: "VolumetricVideoDataloader" = DATALOADERS.build(val_dataloader_cfg) # reuse the validataion │ │ │ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:300 in build │ │ │ │ ❱ 300 │ │ return self.build_func(*args, **kwargs, registry=self) │ │ │ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:131 in build_from_cfg │ │ │ │ ❱ 131 │ return call_from_cfg(obj_cls, args) │ │ │ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:47 in call_from_cfg │ │ │ │ ❱ 47 │ return func(**call_args) │ │ │ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/dataloaders/volumetric_video_dataloader.py:93 in __init__ │ │ │ │ ❱ 93 │ │ if batch_sampler_cfg.batch_size == -1: batch_sampler_cfg.batch_size = len(dataset) │ │ │ │ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/base_utils.py:101 in __getitem__ │ │ │ │ ❱ 101 │ │ │ raise AttributeError(e) │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ AttributeError: 'batch_size' *** 'batch_size' > /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/base_utils.py(101)__getitem__() 99 return dict.__getitem__(self, key) 100 except KeyError as e: --> 101 raise AttributeError(e) 102 # MARK: Might encounter exception in newer version of pytorch 103 # Traceback (most recent call last): (Pdbr) [1]+ Stopped python -q -X faulthandler easyvolcap/scripts/main.py -t gui -c configs/specs/gui.yaml. ```

The process stopped with the error: AttributeError: 'batch_size'. Any guidance you could provide would be greatly appreciated.

[dendenxu]

Thanks for the feedback @YongmoonPark ! I could not reproduce the batch_size error on my end. That error usually means we are in the wrong directory or could not find the config file the user specified (yes I need to make the errors more human readable immediately :] ). Are you on the latest commit?

[YongmoonPark]

Thank you for your prompt response, @dendenxu . I am indeed using the latest commit. I'll take your advice and recheck everything, including reinstalling my environment, to ensure

[dendenxu]

@YongmoonPark Ah, no need for reintalling the environment. But you could try reinstalling 4K4D itself by executing pip install -e . inside the directory.

[dendenxu]

Could you check whether this file configs/specs/gui.yaml exists in your working directory? The first line of log should be like this, where the experiment name is changed to gui instead of base:

2024-01-15 09:36:28.746102 __main__ -> preflight: Starting experiment: gui, command: gui

[YongmoonPark]

I have verified the location of configs/specs/gui.yaml. However, I still encounter the following error when executing:

2024-01-15 11:42:30.025936 __main__ -> preflight: Starting experiment: base, command: gui                              main.py:80
2024-01-15 11:42:30.026796 easyvolcap.utils.console_utils -> inner: Runtime exception: 'batch_size'  

It seems that the issue might be occurring because the configs/specs/gui.yaml file is not being read and executed properly.

Thank you. Your advice has been helpful in narrowing down the cause of the problem.

[dendenxu]

Glad to know that we're making progress. The next reasonable step seems to be putting a breakpoint here and examine the content of args.config and os.getcwd(). To add a breakpoint, simply add the line breakpoint()

[dendenxu]

@GuaGod @YongmoonPark Curious to ask, are you running on WSL2 instead of native Windows/Ubuntu? Related issue.

[dendenxu]

@GuaGod @YongmoonPark Hi folks! Sorry to bother again, but I pushed an update that made the GUI more compatible and make errors messages easier to understand (including segmentation faults). Could you try reinstalling easyvolcap (just run pip install -e . inside the repo) and rerun the erronous commands above and see whether the errors go away?

[YongmoonPark]

@dendenxu Thank you. I am using native Ubuntu (22.04 LTS). I have been trying to trace the cause of a Segmentation fault error. I am still in the process of tracing it, but it seems like imgui_bundle was incompletely installed when set up through requirements.txt.

I have also confirmed the same error occurs when running the simple sample code of imgui_bundle below:

from imgui_bundle import imgui, immapp
imgui.create_context()

def gui():
    imgui.text("Hello, world!")

immapp.run(
    gui_function=gui,  # The Gui function to run
    window_title="Hello!",  # the window title
    window_size_auto=True,  # Auto size the application window given its widgets
    # Uncomment the next line to restore window position and size from previous run
    # window_restore_previous_geometry=True
)

[dendenxu]

Glad to know we're narrowing down the search. Maybe you could also try starting up python with -q -X faulthandler to debug the exact cause of the segfault like this:

python -q -X faulthandler -c "from imgui_bundle import imgui, immapp; imgui.create_context(); immapp.run(gui_function=lambda: imgui.text('Hello, world.'));"