search

PixarAnimationStudios / OpenUSD

Universal Scene Description
http://www.openusd.org
Other
6.01k stars 1.19k forks source link

Ship custom plugins via PIP as extensions to usd-core #2531

Open c-hofer opened 1 year ago

c-hofer commented 1 year ago

Disclaimer

I'm relatively new to USD so I have probably overlooked something and this post is nonsense/duplication. If so: Sorry :)

Description of Issue

Find a way to ship custom plugins via PIP as complementary add-on to the existing usd-core package.

To be clear, I do not think of modifying the existing pypi pipeline to integrate my custom cpp plugins and package my own extended version of usd-core.

What I have in mind is something like the following:

pip install usd-core 
pip install my-usd-core-extensions

where, e.g., the package my_usd_core_extension could register its shipped plugins (including necessary shared objects) against usd_core in its __init__.py.

Application example

Let

  1. my-uri-resolver contain a custom ArResolver mapped to myUri uri scheme
  2. my-file-format contain a custom SdfFileFormat mapped to myff file extension
from pxr import Usd 
import my_uri_resolver # plugin registration happens here
import my_file_format # plugin registration happens here
stage = Usd.Stage.Open("myUri://foo.myff")
...do stuff...

IMHO, the benefits would be to decouple the shipping of extensions from the usd-core building and thus allow for an easy-to-distribute menu of plugins, ready to be picked on demand in downstream python based applications.

I am currently digging into this, but found no clean solution so far, thus:

If there are thoughts in the community about this, ANY breadcrumbs are appreciated!

(If this is kind of an open problem I am happy to discuss my findings too ;) )

jesschimein commented 1 year ago

Filed as internal issue #USD-8490

spiffmon commented 1 year ago

I'll just say for now that I think this is very much an open problem, and not one that Pixar has on our roadmap, at present. Definitely interested in hearing others' thoughts on this, though!

c-hofer commented 1 year ago

After some troubles I found a solution which, a least for my use-case, works. Be aware that it involves stuff which may be considered by some people as black magic :) . Ideas for improving and making it cleaner are more than welcome.

The core idea is to generate binaries which are compatible with the ones shipped by the usd-core pip package and make them work in your python context without any user interference. I'll give a rough outline in the following.

Solution Proposal

Lets say you want to create a pip package my-usd-core-ext :

  1. Create a build docker-container to build your sources. I started from the relevant docker parts in pypi pipeline, but instead of using this container to build USD outside the container, build USD inside the container. Inside the container use the USD version you want to be compatible with.

  2. Use the build container to compile my-usd-core-ext's plugin sources against the compiled USD version inside your build container. This way you get binaries which are compatible (name mangling etc.) with the shipped usd-core binaries. Create a wheel from the compiled binaries.

  3. Repair the linking of your binaries. This is where it gets ugly. As the binaries of my-usd-core-ext have been compiled against USD before the application of auditwheels repair my-usd-core-ext's binaries link against the wrong name and location of the usd-core binaries. Thus, I choose to put some code in my-usd-core-ext which is executed at import time and fixes this by doing the following

    • Create symlinks for all relevant libraries in the usd-core.lib folder of your python interpreter's site-packages to mitigate the name changes auditwheels repair introduces, e.g., libusd_ar.so -> libusd_ar-<the unique ide introduced by auditwheels>.so

    • Use patchelf to set the rpath of my-usd-core-ext's binaries to point to ../site-packages/usd-core.lib to enable the linker to find the relevant dependencies. (make sure to force rpath, otherwise patchelf uses runpath which messes up your setup).

  4. The USD plugins shipped by my-usd-core-ext are registered at import time.

What makes this messy is

  1. that we have to modify the existing usd-core installation (the created symlinks, which, to the best of my knowledge, need to be in the usd-core.lib folder as the usd-core binaries also make use of rpath and things get messed up if you put your symlinks elsewhere)

  2. that the create my-usd-core-ext wheel does not only depend on the python version but also on the specific usd-version.