dlopen() hook not protected from reentrancy

[nathankidd]

VirtualGL currently hooks dlopen() blindly, without reentrancy checks ("does my stack chain contain myself?"), and calls getenv(). The new Cadence Virtuoso IC618 hooks getenv(), also blindly, and calls dlopen() from that hook. The result is a stack smash, starting off like below (excuse the poor OCR of a screenshot):

#84191 0x0000000003df13d5 in getenv
#84192 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/Iibopentextdlfaker.so.3
#84193 0x0000000003dfo960 in syswrap::init()
#84194 0x0000000003df13d5 in getenv
#84195 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84196 0x0000000003dfo960 in syswrap::init()
#84197 0x0000000003df13d5 in getenv
#84198 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84199 0x0000000003dfo960 in syswrap::init()
#84200 0x0000000003df13d5 in getenv
#84201 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84202 0x0000000003dfo960 in syswrap::init()
#84203 0x0000000003df13d5 in getenv
#84204 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84205 0x0000000003dfo960 in syswrap::init()
#84206 0x0000000003df13d5 in getenv
#84207 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84208 0x0000000003dfo960 in syswrap::init()
#84209 0x0000000003df13d5 in getenv
#84210 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84211 0x0000000003dfo960 in syswrap::init()
#84212 0x0000000003df13d5 in getenv
#84213 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84214 0x0000000003dfo960 in syswrap::init()
#84215 0x0000000003df13d5 in getenv
#84216 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84217 0x0000000003dfo960 in syswrap::init()
#84218 0x0000000003df13d5 in getenv
#84219 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84220 0x0000000003dfo960 in syswrap::init()
#84221 0x0000000003df13d5 in getenv
#84222 0x00007ffff7dfa732 in dlopen () from /opt/etxcn/etxcn-11.5/3rdparty/virtualgl/lib64/libopentextdlfaker.so.3
#84223 0x0000000003dfo960 in syswrap::init()
#84224 0x0000000003df13d5 in getenv
#84225 0x00007fffeo73c855 in _GLOBAL__sub_I_pstring.cc () from /app/cadence/IC618/IC618_000/tools/lib/64bit/libcdsenvutil.so
#84226 0x0000003eabeoe9cf in _d1_init_internal () from /lib64/ld-1inux-x86-64.s0.2
#84227 0x0000003eabeoob6a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#84228 0x0000000000000001 in
#84229 0x00007fffffffdf73 in
#84230 0x0000000000000000 in
#(gdb) 

(This from our custom build. I don't have ability to run HEAD VirtualGL in that environment, but I think the problem is understandable enough from a theoretical POV.)

XCB support had the same issue, resolved with TLS-based guards, IIRC. Seems we need the same kind of guard for dlopen()?

[dcommander]

Blerg. OK, I'll look into it.

[dcommander]

Should be fixed now. Test it and let me know.

[dcommander]

@nathankidd Have you had a chance to test? I would like to spin a release with this fix.

[nathankidd]

Just got confirmation the fix worked. Thanks!