Dynamic Path Resolution: Case Sensitivity issue

[adalast]

Description of Issue

USD stages which are using dynamic pathing are failing when the pathing is remapped on a render farm.

The render farm uses Linux and pathing is case sensitive, whereas Windows is not. So scenes which attempt to link to layer files on disk resolve similar to @f:/path/to/geo/cache/payload.usdc@ and the path on the server is F:/path/to/geo/cache/payload.usdc and the software fails to load the geometry.

The issue seems to be pretty pernicious and is affecting more and more users as lots of people set up more complex scenes. Unfortunately the farm cannot change the naming scheme as it is pervasive in other systems that would be too onerous to change.

I have access to the Python API through Houdini for the renders, so any functions I can use which would either let me mutate the path inside the USD architecture itself to force a capital letter or the ability to capture the request leaving USD to go to the OS and mutate it so the proper path is followed would be required. As far as I have been able to find, the lower case seems to be baked in to the USD format and there seems to be no way to change it. Hoping that is not the case.

System Information (OS, Hardware)

Client : Windows Renderfarm : Linux (unsure what specific flavor) Source Software : Houdini Solaris 20

[spiffmon]

Can you please say more about the remapping process? In the original scenes, are there authored drive-letters/absolute-paths that are capitalized? And is the mapping process finding, resolving, and reauthoring all asset-paths using USD API's?

Thanks. --spiff

On Tue, Mar 26, 2024 at 3:11 PM adalast @.***> wrote:

Description of Issue

USD stages which are using dynamic pathing are failing when the pathing is remapped on a render farm.

The render farm uses Linux and pathing is case sensitive, whereas Windows is not. So scenes which attempt to link to layer files on disk resolve similar to @f https://github.com/f:/path/to/geo/cache/payload.usdc@ and the path on the server is F:/path/to/geo/cache/payload.usdc and the software fails to load the geometry.

The issue seems to be pretty pernicious and is affecting more and more users as lots of people set up more complex scenes. Unfortunately the farm cannot change the naming scheme as it is pervasive in other systems that would be too onerous to change.

I have access to the Python API through Houdini for the renders, so any functions I can use which would either let me mutate the path inside the USD architecture itself to force a capital letter or the ability to capture the request leaving USD to go to the OS and mutate it so the proper path is followed would be required. As far as I have been able to find, the lower case seems to be baked in to the USD format and there seems to be no way to change it. Hoping that is not the case. System Information (OS, Hardware)

Client : Windows Renderfarm : Linux (unsure what specific flavor) Source Software : Houdini Solaris 20

— Reply to this email directly, view it on GitHub https://github.com/PixarAnimationStudios/OpenUSD/issues/3020, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPOU2EEC3SWWKYIQ3IQ25LY2HXABAVCNFSM6AAAAABFJYURDCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDSNBTGE2DIMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[jesschimein]

Filed as internal issue #USD-9493

[adalast]

Can you please say more about the remapping process? In the original scenes, are there authored drive-letters/absolute-paths that are capitalized? And is the mapping process finding, resolving, and reauthoring all asset-paths using USD API's? Thanks. --spiff

Answering your last question first as the rest of it is long winded: The end goal is to remap them within USD at submission time. This seems like the most straightforward solution, but I need to understand why it is happening so I can make sure not to break user scenes if they get creative.

Unfortunately that is not my department with the farm and some of that falls under proprietary information, so I will do my best to answer. User data is housed in a directory structure that is associated with their accounts, then project files are stored in a directory in the mount that is named for the project. Inside that, full absolute windows path is stored housing all of the project files. From here I am fuzzy on the details, but I believe symlinks are used to map the paths within the software to believe that the path starts from the Windows drive letter instead of the Linux root. None of this process is currently handled in USD and is a general process for all software that is supported, hence why it cannot be changed.

As for the scenes, they are user authored, so we can only check the ones that have problems. Looking into their setups, none of them have explicitly set up any of the paths for the links, they seem to be authored either by Solaris or USD itself. All of the scenes that have had problems have had lowercase drive letters in the paths when they were resolved. I have checked them with Hython locally and on the servers to be sure.

It also seems to occur with relative paths that are baked in to the USD structures. When USD or Solaris, again, not sure who is doing it, resolves the relative path, they seem to be resolving on the symlink AND resolving in all lowercase. The full linux path does not resolve, just the symlinked Windows path with the lower case drive letter.

[spiffmon]

It seems likely that USD is the one that is responsible for converting your drive letters to lower-case, because if you are using the default ArDefaultResolver in your setup, it will do that. Look for the places where we "create identifiers" as part of resolving paths, and note that we pass the results through TfNormPath, which, in order to create normative paths, chooses lowercase for drive letters.

You can address this by creating a C++ ArResolver plugin that derives from ArDefaultResolver and changes those few behaviors in overridden methods. It's been suggested that you could also address this in the OpenAsset() method as an even more targetted fix, but if you are sure that capitalized drive letters are all you will ever see in your environment, then modifying the returned value of TfNormPath should work.

[adalast]

You can address this by creating a C++ ArResolver plugin that derives from ArDefaultResolver and changes those few behaviors in overridden methods.

I am aware of how to do this in principle, but in practice I have questions/concerns. Since we are packaging this and executing on user machines which we have no control over, I am worried that propagating the change would be impossible. Add onto this that the toolset necessarily relies on the Hython libraries for navigating the Houdini scenes along side the USD stage and I am not sure how to make sure that the new overload is used properly when accessed through the API.

I was really hoping for a function already present in the USD libraries which would allow me to intercept the paths and correct them. I am sure you are correct that this would be the only way though as USD is accessing the drives directly and bypassing Houdini and Hython while it does, then feeding the data it retrieves into Solaris for rendering/display. I just need to figure out the feasibility of doing so. I also need to be sure not to break the USD install in user machines by mucking around with their installs.

[spiffmon]

So, it turns out that lower-casing the drive letter only happens when a path is resolved on a Windows machine . Since your render farm is linux, that suggests to me that (some) fully resolved paths are getting saved into the USD files before they are shipped to the farm. Is that correct?

If that's the case, and you have the ability to modify the files, then you can use the Sdf API's to "fix up" the lowercase drive letters. And/or, maybe chat with the amazing SideFX folks... if the resolving/absolutizing is happening as part of layerStack flattening in Solaris, maybe there's a way to have a LOP fix up the paths prior to saving.

I don't know why we chose lowercase rather than uppercase for "normalizing" drive letters, though there is some concern about the effects of changing that decision now...

[marktucker]

I have been following along here, but I'm a little bit confused... I have definitely seen situations where the behavior of TfNormPath (and running a path through asset resolution) turns an "F:/" into "f:/". But Windows always treats drive letters in a case-insensitive manner (even if you set up your file systems to be case sensitive). So I've never seen an issue where a Windows machine was unable to open a file because it was accessed as "f:/" instead of "F:/". On Linux, on the other hand, I've never seen a situation where USD would resolve a path to "f:/" because Linux doesn't use drive letters. So as Spiff points out, this implies that the full "f:/" path is generated on a windows machine and saved into a USD file as a fully resolved path with a drive letter. On a Linux machine I don't see how any path starting with either "F:/" or "f:/" would ever be opened successfully, so I'm not sure why you think the capitalization of the drive letter is the problem? I'm no Linux expert so maybe I'm missing something here...

This all leads me to suspect that the real problem lies in the generation of the USD files being sent to the farm. They should not have references to other USD files that incorporate drive letters. Which indicates either a failing in your Houdini setup/pipeline, or a bug in Solaris. So you should probably send an email to support@sidefx.com so we can help you resolve this issue with authoring your USD files.

[spitzak]

I believe the issue here is that they actually made directories called "F:", etc, on the Linux machine, so that paths can be used completely unchanged, assuming the cwd is the directory containing all these "disk" directories.

An easy fix would be to add "f:" as a symbolic link on the Linux machine so both versions work.

However it does seem like path normalization on Windows should use upper case for the disks in order to match expectations and most other Windows software.

[adalast]

@spitzak you hit the nail on the head. I do think the symbolic link might be a workable fix from what you described, so I will suggest that to the engineering team, but yes, upper case drive letters would be more preferable as a "normalization" step since in windows upper case drive letters are the "normal" way to do things.

[spiffmon]

@adalast , just wanted to note here that, per AOUSD forum and internal discussions, we are also going to try modifying USD's NormPath and RealPath functions to match python.os's behavior, which should make your use of USD non-frictionful. But in the short term, @spitzak 's suggestion of symlinks hopefully will get you unstuck.

[adalast]

@spiffmon Thank you for the heads up, that thread looks like a great move forward and will definitely help. We are going to implement the symlink solution for the time being for users, but luckily future versions of Houdini (once the new library is included) will not have this issue.