rasterizer_op.so: undefined symbol

solitaryangler commented 2 years ago

Hi,

sincere thanks for developing this TF module. I use TF quite a bit but this is my first time with TF_graphics. I am getting the following error when trying to run inverse_rendering.ipynb locally on my machine - in particular this occurs in the Run Optimization cell:

2021-11-02 18:21:54.266818: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-11-02 18:21:54.449364: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/site-packages/tensorflow_graphics/rendering/kernels/rasterization_backend.py in <module>
     32 try:
---> 33   from tensorflow_graphics.rendering.kernels import gen_rasterizer_op as render_ops
     34 except ImportError:

ImportError: cannot import name 'gen_rasterizer_op' from 'tensorflow_graphics.rendering.kernels' (/home/aghosh/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/site-packages/tensorflow_graphics/rendering/kernels/__init__.py)

During handling of the above exception, another exception occurred:

NotFoundError                             Traceback (most recent call last)
/tmp/ipykernel_8992/2253749353.py in <module>
      9 
     10 learning_rate = 0.02 #@param {type: "slider", min: 0.002, max: 0.05, step: 0.002}
---> 11 start = render_forward(variables, rasterization_func)
     12 optimizer = tf.keras.optimizers.Adam(learning_rate)
     13 animation_images = [start.numpy()]

/tmp/ipykernel_8992/808894707.py in render_forward(variables, rasterization_func)
     72                          tf.matmul(look_at_4x4, model_rotation_4x4))
     73 
---> 74   rendered = rasterization_func(projection, image_width, image_height,
     75                                 variables['light_direction'])
     76 

/tmp/ipykernel_8992/808894707.py in rasterize_then_splat(projection, image_width, image_height, light_direction)
     44 def rasterize_then_splat(projection, image_width, image_height,
     45                          light_direction):
---> 46   return splat.rasterize_then_splat(vertices, triangles, {
     47       'uv': uv_coords,
     48       'normals': normals

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/site-packages/tensorflow_graphics/rendering/splat.py in rasterize_then_splat(vertices, triangles, attributes, view_projection_matrix, image_size, shading_function, num_layers, return_extra_buffers, backend, name)
    265     # Back face culling is necessary when rendering multiple layers so that
    266     # back faces aren't counted as occluding layers.
--> 267     rasterized = rasterization_backend.rasterize(
    268         vertices,
    269         triangles,

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/site-packages/tensorflow_graphics/rendering/rasterization_backend.py in rasterize(vertices, triangles, view_projection_matrices, image_size, enable_cull_face, num_layers, backend)
     74   """
     75   if backend == RasterizationBackends.CPU:
---> 76     backend_module = importlib.import_module(
     77         "tensorflow_graphics.rendering.kernels.rasterization_backend")
     78   elif backend == RasterizationBackends.OPENGL:

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/importlib/__init__.py in import_module(name, package)
    125                 break
    126             level += 1
--> 127     return _bootstrap._gcd_import(name[level:], package, level)
    128 
    129 

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/importlib/_bootstrap.py in _gcd_import(name, package, level)

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/importlib/_bootstrap.py in _find_and_load(name, import_)

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/importlib/_bootstrap.py in _load_unlocked(spec)

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/importlib/_bootstrap_external.py in exec_module(self, module)

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/site-packages/tensorflow_graphics/rendering/kernels/rasterization_backend.py in <module>
     35   import os
     36   dir_path = os.path.dirname(os.path.abspath(__file__))
---> 37   render_ops = tf.load_op_library(os.path.join(dir_path, "rasterizer_op.so"))
     38 # pylint: enable=g-import-not-at-top
     39 

~/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py in load_op_library(library_filename)
     56     RuntimeError: when unable to load the library or get the python wrappers.
     57   """
---> 58   lib_handle = py_tf.TF_LoadLibrary(library_filename)
     59   try:
     60     wrappers = _pywrap_python_op_gen.GetPythonWrappers(

NotFoundError: /home/aghosh/Softwares/miniconda3/envs/tf2x_py3.8/lib/python3.8/site-packages/tensorflow_graphics/rendering/kernels/rasterizer_op.so: undefined symbol: _ZN10tensorflow6StatusC1ENS_5error4CodeEN4absl12lts_2021032411string_viewEOSt6vectorINS_10StackFrameESaIS7_EE

My environment:

Ubuntu 18.04
TF - I tried versions: 2.4.1, 2.5.0
Tensorflow_graphics: latest - 2021.10.21
CUDA: 11.0
cuDNN: 8.1.0

What am I missing out? Your help will be greatly appreciated! Thanks again for this superb effort.

solitaryangler commented 2 years ago

I just realised that a similar problem was stated here - but for Google colab (same notebook - inverse_rendering.ipynb):

https://github.com/tensorflow/graphics/issues/650

But I seem to be facing the same issue in a local install. How to resolve this?

solitaryangler commented 2 years ago

Hi I was able to resolve this issue more or less. I thought of including my experience here for others while closing the issue.

The ticket that helped resolve my issue was: #535

Note-1: I had to use the --force-reinstall flag while pip installing tensorflow_graphics to ensure that all dependencies were actually updated (including TF versions).
Note-2: tensorflow-graphics-gpu is apparently deprecated: #152

Basically I had two issues: (i) OpenExr doesn't always install correctly in all environments (like macos BigSur, or AWS EC2 Ubuntu), but I didn't even need it (ii) I had to ensure that all dependencies were correctly updated by using the --force-reinstall flag.

To summarise, this is what I could use:

To use OpenExr:

first I tried the standard approach: https://github.com/tensorflow/graphics/blob/master/tensorflow_graphics/g3doc/install.md#installing-optional-packages---linux

if that failed then I installed it the from source as per #535:

wget https://github.com/AcademySoftwareFoundation/openexr/archive/v2.5.5.tar.gz                                                                                                                                                                                         
tar -xvf v2.5.5.tar.gz                                                                                                                                                                                                                                                  
cd openexr-2.5.5/                                                                                                                                                                                                                                                       
mkdir build                                                                                                                                                                                                                                                             
cd build                                                                                                                                                                                                                                                                
cmake ../                                                                                                                                                                                                                                                               
make                                                                                                                                                                                                                                                                    
sudo make install

To ignore OpenExr, I installed tensorflow_graphics from source by commenting out the requirement for OpenExr, following again #535:

git clone https://github.com/tensorflow/graphics.git                                                                                                                                                                                                                    
cd graphics
# XXX MANUAL: Comment out OpenEXR in requirements.txt XXX                                                                                                                                                                                                               
python setup.py sdist bdist_wheel   # Build pip pkg.                                                                                                                                                                                                                    
pip install --upgrade dist/*.whl --force-reinstall

Note that you have to use setup.py instead of build_pip_pkg.sh as per the documentation.

Whether I installed tensorflow-graphics from source or pip I had to use the --force-reinstall flag.

Caveat: the --force-reinstall flag sometimes updates keras to version 2.7 which has other issues, so after everything I had to (sometimes) downgrade keras to 2.6.* by running

pip install keras==2.6.*

Thanks to the developers again for this fabulous work!

kimjansheden commented 11 months ago

Is this still supposed to work? I tried following your steps but I still get the error:

File "/Users/foo/miniforge3/envs/tensorflow-graphics/lib/python3.10/site-packages/tensorflow_graphics/rendering/opengl/rasterization_backend.py", line 30, in render_ops = tf.load_op_library(os.path.join(dir_path, "rasterizer_op.so"))

NotFoundError: dlopen(/Users/foo/miniforge3/envs/tensorflow-graphics/lib/python3.10/site-packages/tensorflow_graphics/rendering/opengl/rasterizer_op.so, 0x0006): tried: '/Users/foo/miniforge3/envs/tensorflow-graphics/lib/python3.10/site-packages/tensorflow_graphics/rendering/opengl/rasterizer_op.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/foo/miniforge3/envs/tensorflow-graphics/lib/python3.10/site-packages/tensorflow_graphics/rendering/opengl/rasterizer_op.so' (no such file), '/Users/foo/miniforge3/envs/tensorflow-graphics/lib/python3.10/site-packages/tensorflow_graphics/rendering/opengl/rasterizer_op.so' (no such file)

tensorflow / graphics

rasterizer_op.so: undefined symbol #656