Gazebo 11 does not run correctly with Ogre 1.12 from conda-forge

osrf-migration commented 4 years ago

Original report (archived issue) by Silvio Traversaro (Bitbucket: traversaro).

There have been two reports of problems in running Gazebo compiled against Ogre 1.12, one by Sean Yen (seanyen-msft) ( https://osrf-migration.github.io/gazebo-gh-pages/#!/osrf/gazebo/issues/2686/terrain-rendering-does-not-support-shadows (#2686)#comment-56374947 ) and the other by Wolf Vollprecht (Wolf) ( https://github.com/conda-forge/gazebo-feedstock/pull/11#issuecomment-596067386 ), and in both cases the error message is:

[Err] [..\gazebo\gui\main.cc:34] Ogre Error:Ogre::RuntimeAssertionException::RuntimeAssertionException: Ogre/ShadowExtrudePointLight not found. Verify that you referenced the 'ShadowVolume' folder in your resources.cfg in Ogre::ShadowVolumeExtrudeProgram::initialise at ..\OgreMain\src\OgreShadowVolumeExtrudeProgram.cpp (line 71)

In both cases, they were using the Ogre 1.12 installed by conda-forge. Based on this error, and the fact that apparently Stephen Just ( https://osrf-migration.github.io/gazebo-gh-pages/#!/osrf/gazebo/pull-requests/3130/change-gazebo-rendering-heightmap-class-to/diff#comment-125305344, https://github.com/microsoft/vcpkg/pull/8178#issuecomment-552752416 ) was able to run a patched Gazebo 10 linked with Ogre 1.12 using vcpkg, I suspect that the error is that Ogre is not able to find some resource, possibly because some Ogre resource directory is not correctly set, but this is just an hypothesis.

Based on his work on https://osrf-migration.github.io/gazebo-gh-pages/#!/osrf/gazebo/pull-requests/3130/change-gazebo-rendering-heightmap-class-to/diff#comment-125305344, perhaps Martin Pecka (peci1) may be able to provide some input on this.

osrf-migration commented 4 years ago

Original comment by Silvio Traversaro (Bitbucket: traversaro).

Edited issue description

osrf-migration commented 4 years ago

Original comment by Silvio Traversaro (Bitbucket: traversaro).

Edited issue description

osrf-migration commented 4 years ago

Original comment by Silvio Traversaro (Bitbucket: traversaro).

Edited issue description

osrf-migration commented 4 years ago

Original comment by Silvio Traversaro (Bitbucket: traversaro).

changed title from "Gazebo 11 does not run correctly with Ogre 1.12" to "Gazebo 11 does not run correctly with Ogre 1.12 from conda-forge"

osrf-migration commented 4 years ago

Original comment by Silvio Traversaro (Bitbucket: traversaro).

Edited issue description

osrf-migration commented 4 years ago

Original comment by Silvio Traversaro (Bitbucket: traversaro).

If you check the environment used by Stephen Just to run Gazebo with Ogre 1.12 linked in https://github.com/microsoft/vcpkg/pull/8178#issuecomment-552752416, you can notice that he added %gz_root_path%Media to GAZEBO_RESOURCE_PATH. It may be a coincidence, but apparently the ShadowVolume folder that is referenced in the error, is indeed installed in <prefix>/Media by Ogre 1.12 (see https://github.com/OGRECave/ogre/blob/v1.12.0/CMakeLists.txt#L501) and it was not installed by Ogre 1.11 .
Perhaps it may be worth a shot to try to add that directory to GAZEBO_RESOURCE_PATH and see if that solves the problem.

‌

osrf-migration commented 4 years ago

Original comment by Silvio Traversaro (Bitbucket: traversaro).

I managed to run Gazebo built against Ogre 1.12 provided by vcpkg (in particular vcpkg version 2020.01 ) by adding the <ogre_install_prefix>/Media to GAZEBO_RESOURCE_PATH in the setup.bat . I guess a similar solution (or workaround) probably should work also with Ogre 1.12 provided by conda-forge .

osrf-migration commented 4 years ago

Original comment by Silvio Traversaro (Bitbucket: traversaro).

Another (unrelated) problem that was present with Ogre 1.12 is the one mentioned (and fixed) in https://osrf-migration.github.io/gazebo-gh-pages/#!/osrf/gazebo/pull-requests/3204/fix-ogre-fontmanager-getbyname-use-in-ogre/diff .

osrf-migration commented 4 years ago

Original comment by Silvio Traversaro (Bitbucket: traversaro).

This change is explicitly mentioned in the Ogre 1.12 Changelog (https://github.com/OGRECave/ogre/blob/master/Docs/1.12-Notes.md#stable-media-files):

ACTION REQUIRED you will have to add the Media/ShadowVolume resource location to use the build-in algorithms.

‌

traversaro commented 4 years ago

I recently try again with the latest ogre installed from vcpkg and the latest commit in the gazebo11 branch, and I was unable to reproduce this issue, as gzclient starts correctly even without adding <ogre_install_prefix>/Media to GAZEBO_RESOURCE_PATH. I thought there was strange caching going on my machine, but I tried to export the binaries running on my machine on a new clean Windows laptop, and everything was working fine. Perhaps the fix in https://osrf-migration.github.io/gazebo-gh-pages/#!/osrf/gazebo/pull-requests/3205 had also an influence on this for some reason?

@seanyen @wolfv Just to understand, the gazebo package on conda-forge is still using ogre1.10 ? From https://github.com/conda-forge/gazebo-feedstock/pull/14 it would seem that is using the latest ogre, but I am not 100% sure.

seanyen commented 4 years ago

@traversaro There was an attempt to build Gazebo 11 with OGRE 1.12. Despite it is built, I ran into some issues at runtime which seems to be associated with:

ACTION REQUIRED you will have to add the Media/ShadowVolume resource location to use the build-in algorithms.

However, it seems to me that in your testing, it is actually working fine with the Vcpkg OGRE port. I think I might need to retry it again.

traversaro commented 3 years ago

Finally, I was able to reproduce the problem on an almost vanilla Ubuntu 20.04, using the Gazebo 11 with ogre 1.12 binary that we recently added in conda-forge (see https://github.com/conda-forge/gazebo-feedstock/pull/85):

(ros2) straversaro@iiticublap103:~/Downloads$ gazebo --verbose
Gazebo multi-robot simulator, version 11.7.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
Gazebo multi-robot simulator, version 11.7.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.59
[Msg] Loading world file [/home/straversaro/mambaforge/envs/ros2/share/gazebo-11/worlds/empty.world]
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.59
[Wrn] [Event.cc:61] Warning: Deleting a connection right after creation. Make sure to save the ConnectionPtr from a Connect call
[Wrn] [GuiIface.cc:120] Qt has caught an exception thrown from an event handler. Throwing
exceptions from an event handler is not supported in Qt.
You must not let any exception whatsoever propagate through Qt code.
If that is not possible, in Qt 5 you must at least reimplement
QCoreApplication::notify() and catch all exceptions there.

[Err] [main.cc:37] Ogre Error:RuntimeAssertionException: Ogre/ShadowExtrudePointLight not found. Verify that you referenced the 'ShadowVolume' folder in your resources.cfg in initialise at /home/conda/feedstock_root/build_artifacts/ogre_1624011825937/work/OgreMain/src/OgreShadowVolumeExtrudeProgram.cpp (line 70)
(ros2) straversaro@iiticublap103:~/Downloads$ conda list
# packages in environment at /home/straversaro/mambaforge/envs/ros2:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
alsa-lib                  1.2.3                h516909a_0    conda-forge
argcomplete               1.12.3             pyhd8ed1ab_2    conda-forge
assimp                    5.0.1                hedfc422_5    conda-forge
atk-1.0                   2.36.0               h3371d22_4    conda-forge
attr                      2.4.48               h516909a_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
boost                     1.74.0           py38hc10631b_3    conda-forge
boost-cpp                 1.74.0               h312852a_4    conda-forge
bullet                    3.17                 ha770c72_0    conda-forge
bullet-cpp                3.17                 h1abd341_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h7f98852_1    conda-forge
ca-certificates           2021.5.30            ha878542_0    conda-forge
cairo                     1.16.0            h6cf1ce9_1008    conda-forge
catkin_pkg                0.4.23             pyh9f0ad1d_0    conda-forge
certifi                   2021.5.30        py38h578d9bd_0    conda-forge
cffi                      1.14.6           py38ha65f79e_0    conda-forge
cfitsio                   3.470                hb418390_7    conda-forge
cmake                     3.21.0               h8897547_0    conda-forge
console_bridge            1.0.1                h4bd325d_0    conda-forge
cppcheck                  2.5              py38hbffb2f6_0    conda-forge
cppzmq                    4.7.1                hf7cf922_2    conda-forge
cryptography              3.4.7            py38ha5dfef3_0    conda-forge
curl                      7.77.0               hea6ffbf_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
dartsim                   6.10.1               he39ca3a_1    conda-forge
dbus                      1.13.6               h48d8840_2    conda-forge
distro                    1.5.0              pyh9f0ad1d_0    conda-forge
docutils                  0.17.1           py38h578d9bd_0    conda-forge
eigen                     3.3.9                h4bd325d_1    conda-forge
empy                      3.3.4              pyh9f0ad1d_1    conda-forge
expat                     2.4.1                h9c3ff4c_0    conda-forge
fcl                       0.6.1                h2cbc392_3    conda-forge
ffmpeg                    4.3.1                hca11adc_2    conda-forge
flake8                    3.9.2              pyhd8ed1ab_0    conda-forge
flann                     1.9.1             h2e58136_1008    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.13.1            hba837de_1005    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
foonathan-memory          0.6.2                he1b5a44_1    conda-forge
freeglut                  3.2.1                h9c3ff4c_2    conda-forge
freeimage                 3.18.0               h88c329d_7    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
freexl                    1.0.6                h7f98852_0    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
gazebo                    11.7.0               he292fef_1    conda-forge
gdbm                      1.18                 h0a1914f_2    conda-forge
gdk-pixbuf                2.42.6               h04a7f16_0    conda-forge
geos                      3.9.1                h9c3ff4c_2    conda-forge
geotiff                   1.6.0                h4f31c25_6    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
glew                      2.1.0                h9c3ff4c_2    conda-forge
glib                      2.68.3               h9c3ff4c_0    conda-forge
glib-tools                2.68.3               h9c3ff4c_0    conda-forge
gmock                     1.10.0               h4bd325d_7    conda-forge

traversaro commented 3 years ago

By manually adding $CONDA_PREFIX/share/OGRE/Media/ShadowVolume to GAZEBO_RESOURCE_PATH, the earlier problem is solved but now I get this error:

(ros2) straversaro@iiticublap103:~/mambaforge/envs/ros2/share/gazebo-11$ gazebo --verbose
Gazebo multi-robot simulator, version 11.7.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
Gazebo multi-robot simulator, version 11.7.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.59
[Msg] Loading world file [/home/straversaro/mambaforge/envs/ros2/share/gazebo-11/worlds/empty.world]
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.59
[Wrn] [Event.cc:61] Warning: Deleting a connection right after creation. Make sure to save the ConnectionPtr from a Connect call
[Wrn] [GuiIface.cc:120] Qt has caught an exception thrown from an event handler. Throwing
exceptions from an event handler is not supported in Qt.
You must not let any exception whatsoever propagate through Qt code.
If that is not possible, in Qt 5 you must at least reimplement
QCoreApplication::notify() and catch all exceptions there.

[Err] [main.cc:37] Ogre Error:RuntimeAssertionException: gpu program could not be created in createGpuPrograms at /home/conda/feedstock_root/build_artifacts/ogre_1624011825937/work/Components/RTShaderSystem/src/OgreShaderProgramManager.cpp (line 251)
(ros2) straversaro@iiticublap103:~/mambaforge/envs/ros2/share/gazebo-11$

Tobias-Fischer commented 3 years ago

I found this but it shouldn't matter as we already have 1.12.12 in conda-forge: https://forums.ogre3d.org/viewtopic.php?f=2&t=96240

Tobias-Fischer commented 3 years ago

Also see https://github.com/osrf/gazebo/issues/2986 where the same error occurs.

traversaro commented 3 years ago

Also see #2986 where the same error occurs.

Interesting, this may suggest that is a general problem with Ogre 1.12.12, and not a Conda-specific problem. Indeed, in the vcpkg-based build of Gazebo (that work fine, at least in Windows) we are using Ogre 1.12.9 (https://github.com/microsoft/vcpkg/blob/45fc55825db2a5bcaffccff1e6afadc519d164ea/ports/ogre/vcpkg.json), so this could be a change/regression related to Ogre 1.12.12 . Indeed, looking at https://repology.org/project/ogre/versions the only distros with 1.12.12 seem to be some OpenSUSE one and Rosa Linux, so it is possible that no one else found this problem.

However, I inspected a bit the Ogre code (in https://github.com/OGRECave/ogre/blob/v1.12.12/Components/RTShaderSystem/src/OgreShaderProgramManager.cpp#L251) and the error is related to a shader program that was not able to compile. This is compatible with the fact that the same Ogre version is used in RViz2, and there it seems to work fine, so the problem could be related to some Gazebo-specific shader. So to continue the debugging, we should at least collect some info on which shader compilation is failing, and hopefully on the specific compilation error. Probably this will require to build both Ogre and Gazebo from source.

kevinsmia1939 commented 3 years ago

Hi, I tried building with ogre 1.12.9 with gazebo 11.6.0 but still same issue. Not sure if it is expected from the comment above.

traversaro commented 3 years ago

Hi, I tried building with ogre 1.12.9 with gazebo 11.6.0 but still same issue. Not sure if it is expected from the comment above.

Interesting, thanks for reporting. I wonder why instead the 1.12.9 via vcpkg on Windows works fine.

kevinsmia1939 commented 3 years ago

Hi, I tried building with ogre 1.12.9 with gazebo 11.6.0 but still same issue. Not sure if it is expected from the comment above.

Interesting, thanks for reporting. I wonder why instead the 1.12.9 via vcpkg on Windows works fine.

Do you think I can use strace to debug this? Never use it before.

traversaro commented 3 years ago

If the shader compilation involve any kind of system call, using strace could indeed provide some useful information, but I do not know enough of shader compilation to know if this is the case before checking.

traversaro commented 3 years ago

FYI @mboisson depending on the ogre version you have in Easybuild, you may be affected by this problem as well.

mboisson commented 3 years ago

@traversaro thanks for the pointer. I'm not sure I follow everything, but which version of OGRE is safe to use ? I gather 1.12.12 is not. Is all of the 1.12 branch to avoid, and I should build with 1.11 ? or should 1.12.x (for x<12) be fine ?

traversaro commented 3 years ago

I'm not sure I follow everything, but which version of OGRE is safe to use ?

I don't know a-priori which version of ogre is safe, I can only tell you that:

apt on Ubuntu 20.04 uses ogre 1.9.0, and it seems not to be affected by this problem
The default build of Gazebo on conda-forge use ogre 1.10.12, and they are not affected by this problem
I have a vcpkg-based build of Gazebo on Windows using ogre 1.12.9, and it is not affected by this problem
On conda-forge we tried to use 1.12.12 on a variant build, but it is affected by this problem
According to @kevinsmia1939 , flatpack-based build of Gazebo that use both ogre 1.12.9 or 1.12.12 seems to be affected by this problem

However, it is not obvious to tell which version of Ogre may be affected without understanding the source of the issue.

mboisson commented 3 years ago

Ok. I'm gonna go back down to 1.10.12, as it seems issues happen starting in 1.11: https://osrf-migration.github.io/gazebo-gh-pages/#!/osrf/gazebo/issues/2686/page/1?slug=terrain-rendering-does-not-support-shadows#comment-56374947

mboisson commented 3 years ago

🤔 actually, ignition-rendering3_3.5.0 does not build with ogre 1.10.12 :( (https://github.com/ignitionrobotics/ign-rendering/issues/374)

traversaro commented 3 years ago

🤔 actually, ignition-rendering3_3.5.0 does not build with ogre 1.10.12 :( (https://github.com/ignitionrobotics/ign-rendering/issues/374)

Just FYI, ign-rendering is not a dependency of Classic Gazebo.

kevinsmia1939 commented 3 years ago

Hi,

I use Ogre 1.9.0 and Gazebo 11.7.0, Gazebo does not show Ogre Error:RuntimeAssertionException: gpu program could not be created in createGpuPrograms at /run/build/OGRE/Components/RTShaderSystem/src/OgreShaderProgramManager.cpp error anymore.

Now Gazebo is working and does not freeze now.

I can keep trying with newer Orge to see which version broke.

traversaro commented 3 years ago

I can keep trying with newer Orge to see which version broke.

Great, if you could to this it would be quite useful.

kevinsmia1939 commented 3 years ago

I will keep trying more, here is the test so far. Gazebo 11.7.0

      -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo
      -DCMAKE_INSTALL_PREFIX:PATH=/app

Ogre

       -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo
       -DOGRE_INSTALL_DOCS:BOOL=OFF
       -DOGRE_BUILD_TESTS:BOOL=OFF
       -DOGRE_BUILD_COMPONENT_CSHARP:BOOL=OFF
       -DOGRE_BUILD_DEPENDENCIES=FALSE

1.9.0 work 1.10.12 work 1.11.0 work 1.11.2 work 1.11.3 work 1.11.4 broke [Err] [main.cc:37] Ogre Error:InternalErrorException: Not all parameters could be constructed for the sub-render state. in IntegratedPSSM3::resolveParameters at /run/build/Gazebo/gazebo/rendering/CustomPSSMShadowCameraSetup.cc (line 236) 1.11.5 broke [Err] [main.cc:37] Ogre Error:InternalErrorException: Not all parameters could be constructed for the sub-render state. in IntegratedPSSM3::resolveParameters at /run/build/Gazebo/gazebo/rendering/CustomPSSMShadowCameraSetup.cc (line 236) 1.11.6 broke [Err] [main.cc:37] Ogre Error:InternalErrorException: Not all parameters could be constructed for the sub-render state. in IntegratedPSSM3::resolveParameters at /run/build/Gazebo/gazebo/rendering/CustomPSSMShadowCameraSetup.cc (line 236) 1.12.0 work 1.12.3 work 1.12.4 work 1.12.5 broke [Err] [main.cc:37] Ogre Error:RuntimeAssertionException: gpu program could not be created in createGpuPrograms at /run/build/OGRE/Components/RTShaderSystem/src/OgreShaderProgramManager.cpp (line 252) 1.12.9 broke Ogre Error:RuntimeAssertionException: gpu program could not be created in createGpuPrograms at /run/build/OGRE/Components/RTShaderSystem/src/OgreShaderProgramManager.cpp (line 251) 1.12.12 broke Ogre Error:RuntimeAssertionException: gpu program could not be created in createGpuPrograms at /run/build/OGRE/Components/RTShaderSystem/src/OgreShaderProgramManager.cpp (line 251)

kevinsmia1939 commented 3 years ago

So I found out that this bug occurred when Ogre change from 1.12.4 to 1.12.5.

Tobias-Fischer commented 3 years ago

Great investigative work! Here is the diff and changelog. Unfortunately quite a few commits related to the RT Shader System, it might be worth bisecting further ..

Tobias-Fischer commented 3 years ago

Maybe this PR (https://github.com/OGRECave/ogre/pull/1398) is the culprit?

traversaro commented 3 years ago

@WilliamLewww just in case and if you have time, do you have any idea on what could be causing this? Asking as you seem to have quite an experience with Ogre and shaders. If you don't have time to read all the thread, no problem!

WilliamLewww commented 3 years ago

I was able to reproduce similar results to the ones @kevinsmia1939 got. I'm hoping that OGRE just changes their syntax for their program scripts and not the actual behavior of the program parser. I'll look more into the error with v1.12.12 to find what is causing the issue.

traversaro commented 3 years ago

Thanks a lot @WilliamLewww, that's great!

traversaro commented 2 years ago

So, I tried to investigate this issue on Windows, with Gazebo 11.8.1 and Ogre 1.12.13 . First of all, on Windows (on Conda) the correct way to extend the GAZEBO_RESOURCE_PATH is:

set GAZEBO_RESOURCE_PATH=%GAZEBO_RESOURCE_PATH%;%CONDA_PREFIX%\Library\Media\ShadowVolume

because by default the OGRE's Media directory gets installed in a diffent location w.r.t. to Linux.

After that, the error is the usual one on the error in GPU program, but thanks to the experience in https://github.com/ignitionrobotics/ign-gazebo/issues/1116 I know where to look when you have a OGRE problem: in the ~/.gazebo/ogre.log file. By looking at the last line of that file, I see that the error is:

13:26:54: Added resource location 'C:\Users\STraversaro\AppData\Local\mambaforge\envs\gazebo-ogre-1-12\Library\share\gazebo-11\media\rtshaderlib\' of type 'FileSystem' to resource group 'General'
13:26:55: Texture 'SkyX_Starfield.png': Loading 1 faces(PF_A8R8G8B8,512x512x1) with 5 hardware generated mipmaps from Image. Internal format is PF_A8R8G8B8,512x512x1.
13:26:55: Texture 'SkyX_Moon.png': Loading 1 faces(PF_A8R8G8B8,512x512x1) with 5 hardware generated mipmaps from Image. Internal format is PF_A8R8G8B8,512x512x1.
13:26:55: Texture 'SkyX_MoonHalo.png': Loading 1 faces(PF_SHORT_RGBA,512x256x1) with 5 hardware generated mipmaps from Image. Internal format is PF_SHORT_RGBA,512x256x1.
13:26:55: Texture 'Noise.jpg': Loading 1 faces(PF_R8G8B8,1024x1024x1) with 5 hardware generated mipmaps from Image. Internal format is PF_R8G8B8,1024x1024x1.
13:26:55: RenderSystem::_createRenderWindow "OgreWindow(1)", 1672x786 windowed  miscParams: FSAA=4 contentScalingFactor=1.000000 externalWindowHandle=10095664 macAPI=cocoa macAPICocoaUseNSView=true stereoMode=Frame Sequential 
13:26:55: Warning: force-disabling 'lighting' and 'depth_check' of Material SelectionDebugMaterial for use with OverlayElement SelectionDebugPanel
13:26:55: [SkyX] VClouds warning: unregistered camera registered, manual unregistering is needed before camera destruction
13:26:55: 'ShadowExtrudePointLight.vert' WARNING: 1:1: '' :  #version directive missing

13:26:55: 'ShadowExtrudeDirLight.vert' WARNING: 1:1: '' :  #version directive missing

13:26:55: 'ShadowExtrudePointLightFinite.vert' WARNING: 1:1: '' :  #version directive missing

13:26:55: 'ShadowExtrudeDirLightFinite.vert' WARNING: 1:1: '' :  #version directive missing

13:26:55: 'ShadowBlend.frag' WARNING: 1:1: '' :  #version directive missing

13:26:56: Program '8029c501a900f9e4d7a9019d3dbcc613_VS' is not supported: '8029c501a900f9e4d7a9019d3dbcc613_VS' ERROR: 0:47: 'FFP_Transform' : no matching overloaded function found (using implicit conversion) 
ERROR: 0:47: 'FFP_Transform' : function is not known 

13:26:56: Error: RTSS - creating GpuPrograms for pass 0 of 'ground_plane::link::visual_MATERIAL_Gazebo/Grey' failed

The relevant error seems that 'FFP_Transform' : function is not known. At first, I tried to add more directories to GAZEBO_RESOURCE_PATH to see if that fixed the problem, with:

set GAZEBO_RESOURCE_PATH=%GAZEBO_RESOURCE_PATH%;%CONDA_PREFIX%\Library\Media\ShadowVolume;%CONDA_PREFIX%\Library\Media\RTShaderLib\materials;%CONDA_PREFIX%\Library\Media\RTShaderLib\GLSL;%CONDA_PREFIX%\Library\Media\RTShaderLib\HLSL_Cg;%CONDA_PREFIX%\Library\Media\Terrain

but that did not changed the problem. So, I searched for FFP_Transform online, and I noticed that it is defined in a Gazebo file, media/rtshaderlib/FFPLib_Transform.glsl that seems vendored from an old version of OGRE. Indeed, if I try to substitute the Gazebo version with the Ogre version (https://github.com/OGRECave/ogre/blob/v1.12.13/Media/RTShaderLib/GLSL/FFPLib_Transform.glsl), the error changes:

15:53:19: Program '0050cc6967c8b0e8511478f6b3696aa3_FS' is not supported: '0050cc6967c8b0e8511478f6b3696aa3_FS' ERROR: 0:59: 'SGX_Light_Directional_DiffuseSpecular' : no matching overloaded function found (using implicit conversion) 
ERROR: 0:59: 'SGX_Light_Directional_DiffuseSpecular' : function is not known 
ERROR: 0:61: 'SGX_ComputeShadowFactor_PSSM3' : no matching overloaded function found (using implicit conversion) 
ERROR: 0:61: 'SGX_ComputeShadowFactor_PSSM3' : function is not known

Using the same reasoning, I tried to blindly updating the following Gazebo files:

https://github.com/osrf/gazebo/blob/gazebo11_11.8.1/media/rtshaderlib/SGXLib_PerPixelLighting.glsl with https://github.com/OGRECave/ogre/blob/v1.12.13/Media/RTShaderLib/GLSL/SGXLib_PerPixelLighting.glsl for SGX_Light_Directional_DiffuseSpecular
https://github.com/osrf/gazebo/blob/gazebo11_11.8.1/media/rtshaderlib/SGXLib_IntegratedPSSM.glsl with https://github.com/OGRECave/ogre/blob/v1.12.13/Media/RTShaderLib/GLSL/SGXLib_IntegratedPSSM.glsl for SGX_ComputeShadowFactor_PSSM3

However, after this substitutions I still get the SGX_ComputeShadowFactor_PSSM3:

15:59:49: Program '0050cc6967c8b0e8511478f6b3696aa3_FS' is not supported: '0050cc6967c8b0e8511478f6b3696aa3_FS' ERROR: 0:61: 'SGX_ComputeShadowFactor_PSSM3' : no matching overloaded function found (using implicit conversion) 
ERROR: 0:61: 'SGX_ComputeShadowFactor_PSSM3' : function is not known

At this point, I guess I would need to understand something about shaders to actually progress on this issue.

traversaro commented 2 years ago

An additional point that I would like to undestand, but I was not able to understand, is why the Gazebo version (11.3.0) built with ogre (1.12.9) from vcpkg that one can get from https://github.com/robotology/robotology-superbuild-dependencies-vcpkg is able to work even without the need to change GAZEBO_RESOURCE_PATH.

I tought it was something related to the resources.cfg file installed in the <prefix>\lib library, but actually Gazebo it still working fine even after deleting all the OGRE's *.cfg files, and it is still working fine, so it must be something else. I wonder if it is simply something related to the Gazebo or OGRE version. I see that in https://github.com/Ace314159/ros2-foxy-windows-packages/packages/943427 has a Gazebo version 11.7.0 that I guess is built against OGRE 1.12.9 from vcpkg, @Ace314159 do you verified that that Gazebo binary is actually launching fine on Windows and is not printing any strange ShadowVolume errors?

Ace314159 commented 2 years ago

@traversaro Yes, I am able to launch Gazebo 11.7.0 without any errors, building with OGRE 1.12.9 from vcpkg as you guessed. I'm not directly using the binary, but I am able to launch and use the Navigation2 example:

However, I am setting GAZEBO_RESOURCE_PATH, GAZEBO_PLUGIN_PATH, GAZEBO_MODEL_PATH, and OGRE_RESOURCE_PATH since the build directory on CI doesn't match my install directory.

I am building using the master branch of my fork of vcpkg. I was planning to make a PR to upstream after cleaning it up a bit and once https://github.com/microsoft/vcpkg/pull/20565 was merged.

If you'd like to test, you can download the release, extract the x64 folder into C:/opt/ros/foxy, and set all the environment variables required as below

$env:GAZEBO_RESOURCE_PATH = "C:/opt/ros/foxy/x64/share/gazebo-11;" + $env:GAZEBO_RESOURCE_PATH
$env:GAZEBO_PLUGIN_PATH = "C:/opt/ros/foxy/x64/bin/gazebo-11/plugins;" + $env:GAZEBO_PLUGIN_PATH
$env:GAZEBO_MODEL_PATH = "C:/opt/ros/foxy/x64/share/gazebo-11/models;" + $env:GAZEBO_MODEL_PATH
$env:OGRE_RESOURCE_PATH = "C:/opt/ros/foxy/x64/tools/gazebo11"
$env:PATH += ";C:/opt/ros/foxy/x64/tools/gazebo11"

Then, you can launch the nav2 example with

$env:GAZEBO_MODEL_PATH+="C:/opt/ros/foxy/x64/share/turtlebot3_gazebo/models"
$env:TURTLEBOT3_MODEL="waffle"
ros2 launch nav2_bringup tb3_simulation_launch.py

traversaro commented 2 years ago

Thanks a lot for the detailed response @Ace314159 ! I can't believe that https://github.com/microsoft/vcpkg/issues/8014 is going to be finally solved.

scpeters commented 2 years ago

I haven't followed all the updates here, but are there any changes needed for gazebo11 to work with ogre 1.12?

j-rivero commented 2 years ago

After checking internally with the Gazebo team, we want to thanks everyone for the efforts regarding to the migration of the software to ogre-1.12. This migration is something that has not been tested by Open Robotics in any manner and could cause any kind of problems, from building errors up to any kind of rendering issues. We are specially concerned about problems with the terrain materials. Please be careful, we can not guarantee that the software keeps working as it does with ogre-1.9. Feel free to keep forwarding feedback and patches (issues are welcome although we can not assure to have the human resources to fix them in a short period of time).

If by any reason Open Robotics find the resources needed to accomplish the migration, we'll update this same issue. Thanks!

traversaro commented 2 years ago

I haven't followed all the updates here, but are there any changes needed for gazebo11 to work with ogre 1.12?

Beside everything that @j-rivero wrote, i.e. that Ogre 1.12 is in any case not feature parity with Ogre 1.9 for what regards integration with Classic Gazebo (but I guess I do not need to explain this to you : ) ), this is the current experienced outcome when using Ogre 1.12 (I just report distro and ogre version, but perhaps other factors play a role) :

Ogre Version	Distribution	Source	Working (at least starting)?
1.12.9	vcpkg	https://github.com/osrf/gazebo/issues/2700#issuecomment-949911867	✅
1.12.12	conda-forge	https://github.com/osrf/gazebo/issues/2700#issuecomment-892820396	❌
1.12.*	Flatpak	https://github.com/osrf/gazebo/issues/2700#issuecomment-893366972	✅ ogre <= 1.12.4, ❌ ogre >= 1.12.5
1.12.1 ? (not sure, just desumed from https://github.com/NixOS/nixpkgs/blob/50774ebcc17367071c977c79a99f0b0495e807e2/pkgs/development/libraries/ogre/default.nix#L34)	nixOS	https://github.com/lopsided98/nix-ros-overlay/issues/161#issuecomment-1019449771	❌

jspricke commented 2 years ago

I made some progress with Ogre 1.12.10 on Debian unstable. From reading the code I don't think setting GAZEBO_RESOURCE_PATH could work, so I copied /usr/share/OGRE/Media/ShadowVolume/* to /usr/share/gazebo-11/media/ and /usr/share/OGRE/Media/RTShaderLib/GLSL/* to /usr/share/gazebo-11/media/rtshaderlib/ (the proper way is probably to add /usr/share/OGRE/Media/ to the resources path in Gazebo and drop /usr/share/gazebo-11/media/rtshaderlib/. I had to adopt SGXLib_IntegratedPSSM.glsl a bit as well (we need to define PSSM_SAMPLE_CMP somewhere). After it gazebo worlds/pioneer2dx.world loads but some meshes are missing.

ljaniec commented 2 years ago

Are there any new updates? How can I help with debugging, maybe my logs will be useful?

My setup: Ubuntu 20.04, ROS 2 Galactic, Gazebo (11.10.1-1~focal). With the update to the gazebo11 (ver 11.10.1) in the last ROS 2 Galactic sync my launch files and Gazebo overall stopped working correctly (services are timed out). I described it with details in this issue: https://github.com/osrf/gazebo/issues/3173.

@jspricke Can these errors be connected? https://github.com/osrf/gazebo/issues/3173#issuecomment-1068011903

traversaro commented 1 year ago

Hello @ljaniec, sorry for the late response: I do not think your problem is connected, if you are using the official Gazebo binaries from apt you are probably using Ogre 1.9 from apt packages.

talregev commented 1 year ago

I was able to compile Gazebo on windows with my fix PR with ogre 1.12.9 and qwt 6.1.5 and graphviz 2.49.1. Please before your compile MAKE SURE your visual studio is the latest update. It have an atomic error bug in the middle versions. Also I did this for overcome the bug. Not sure if that help: For compile on windows I did this for atomic error I had: https://stackoverflow.com/questions/67732065/why-does-vs2019-pro-have-compile-errors-with-xutility-xmemory-and-atomic-when

For compiling:

git clone https://github.com/talregev/vcpkg -b TalR/fix_gazebo2
cd vcpkg
bootstrap-vcpkg.bat
vcpkg install  --x-manifest-root=ports/gazebo/ --x-install-root=installed --triplet=x64-windows-release --host-triplet=x64-windows-release --clean-after-build
vcpkg install gazebo[tools,plugins] --triplet=x64-windows-release --host-triplet=x64-windows-release

This compile gazebo but I didn't succeeded to run it. Can you help me to understand how to run it? I also can compile gazebo with ogre 1.11.x if needed. Write me on the comment, and I will make one.

Also vcpkg tool can help you to compile your 3rd libraries for windows and linux. Maybe for mac with extra work.

talregev commented 1 year ago

I made a PR that compile gazebo in windows: https://github.com/gazebosim/gazebo-classic/pull/3312

gazebosim / gazebo-classic

Gazebo 11 does not run correctly with Ogre 1.12 from conda-forge #2700