Port USD Python to pybind

[lpapp]

Description of Issue

boost::python can be challenging to work with as if a product using USD (and USD Python) already links against boost statically, this could cause issues for the product. It seems that double static boost could cause boost::python singleton issues. The workaround currently is to bundle USD Python with a dynamic boost::python.

But if the project switched to pybind like so many, this python bindings would become a lot easier to integrate into DCC products, I feel.

[spiffmon]

Hi @lpapp ! We agree with your assessment on boost::python, though we were recently made aware of nanobind by the same author, which seems extremely compelling for build-time and size. The big challenge for Pixar, which also makes accepting an isolated, external contribution difficult, is that we have a very large codebase (Presto) built on top of USD with several hundreds of boost-bound modules, and unless we can figure out how to make boost and pybind/nanobind work together, and in concert with the subtle lifetime management things we do in our TfPy* framework, we would need to port that gigantic codebase all at once, which is a project we are not currently staffed for. But we are keeping this undertaking in mind, and believe in its payoffs for the community and our own pipeline and processes.

[sunyab]

Filed as internal issue #USD-7465

[lpapp-foundry]

Thanks, yes, I was also going to mention nano bind for completeness. Sounds like a good idea to me.

I do not know the details. What is the issue about boost and pybind/nanobind working together?

[gitamohr]

Today, with boost.python bindings, a downstream module can boost.python wrap a function like void Foo(UsdPrim) and that works fine because UsdPrim is wrapped with boost.python. If we instead wrap UsdPrim using pybind11 or nanobind or anything else, then that downstream module's boost.python-bound Foo will cease to work. As Spiff says, at Pixar we have hundreds of libraries (comprising millions of lines of code) with boost.python bindings that depend on Usd, so for us it's not a simple matter to swtich.

[lpapp-foundry]

Would it be possible to add a compile-time switch? Deprecate the boost::python way? Give enough time for migration? Remove in the future?

This way, you would slowly phase out a suboptimal solution.

---

Are there tricky cases for the porting by the way? It seemed that the basic cases map simply from boost::python to pybind. Maybe, some of it can be even aided by script conversation to automate it for a relief.

[gitamohr]

Of course a build-time switch is possible, but then we have added another dimension of complexity to support and maintain both sets of bindings, and to carefully ensure that there are no semantic differences between the two. There are always plenty of tricky cases for big changes like these, especially with such a large volume of code to update and validate. And yes there's potential for some automation, but even still it will be a very large project.

And you must also understand that this project is high cost with little practical benefit from Pixar's point of view as a production studio. But like Spiff says, we keep talking about this project and we recognize the benefits it would have for the USD community.

[asluk]

Sunya's talk from ASWF USD WG on the pybind11 effort-- https://drive.google.com/file/d/1HtuyMDZ1TsjY9KmBLU7XiBCYtyavEJif/view