search

hpc / Spindle

Scalable dynamic library and python loading in HPC environments
Other
96 stars 24 forks source link

$ORIGIN in $RPATH in nested dependency is not handled correctly #36

Open fukai-t opened 4 years ago

fukai-t commented 4 years ago

When I try SPINDLE, I found that $ORIGIN in $RPATH in nested dependency is not handled correctly and the process cannot load some libraries.

Example: When my python script on my environment imports matplotlib,

  1. matplotlib requires /path/to/lib/python2.7/site-packages/numpy/core/multiarray.so
  2. Spindle create cache of multiarray.so as /tmp/spindle.PIDNUM/b0-_path_to_lib_python2.7_site-packages_numpy_core_multiarray.so
  3. This multiarray.so requires $ORIGIN/../.libs/tls/x86_64/libopenblasp.so

In this case, $ORIGIN/../.libs/tls/x86_64/libopenblasp.so should be expands as /path/to/lib/python2.7/site-packages/numpy/core/../.libs/libopenblasp.so. However, SPINDLE expands as /tmp/spindle.PIDNUM/../.libs/tls/x86_64/libopenblasp.so.

i.e. SPINDLE expands $ORIGIN as /tmp/spindle.PIDNUM/ instead of /path/to/lib/python2.7/site-packages/numpy/core/

As a result, the process cannot load multiarray.so.

This issue may be similar to #17, but current SPINDLE runs with --debug=yes in default.

mplegendre commented 4 years ago

This was just reported from another source too, and a fix is underway. I'll post a pointer here when it's up.

fukai-t commented 4 years ago

Thank you for your answer. I'm looking forward to receiving good news.

mplegendre commented 4 years ago

The devel branch has a fix for this issue. It's an unfortunately large fix that changes a lot, so I won't be making a point release for it. It'll be included in the next major Spindle release.