Sporadic crash during recomposition on M1 Mac builds

[marktucker]

Description of Issue

There seems to be a recently (between 23.02 and 23.08) introduced race condition to USD composition on M1 builds. See https://forum.aousd.org/t/anyone-seeing-issues-with-usd-on-m1-macs/599 for some additional information.

Steps to Reproduce

1. Run the following python code (or something similar) many many times on an M1 Mac build of USD:

   s = Usd.Stage.CreateInMemory()
   prim = s.DefinePrim('/asset')
   prim.GetReferences().AddReference(asset_file_path)

2. Eventually get either a crash (in 23.08) or a runtime error (23.05)

System Information (OS, Hardware)

MacOS 13, clang 14, M1 Mac

Package Versions

23.05, 23.08

Build Flags

M1 arm64 builds

[jesschimein]

Filed as internal issue #USD-8668

[dgovil]

Just an FYI this is still an issue with 24.8 as well..

Sample snippet to run it in a loop

Running this arch -arm64 python3 test.py but not with arch -x86_64 python3. Substitute the path to the gramophone with https://developer.apple.com/augmented-reality/quick-look/models/gramophone/gramophone.usdz

I suspect there's some subtle memory ordering bug somewhere

import platform, sys
print(sys.version_info)
print(platform.processor())

from pxr import Usd
print(Usd.GetVersion())

for x in range(40_000):
    s = Usd.Stage.CreateInMemory()
    prim = s.DefinePrim('/asset')

    prim.GetReferences().AddReference("/Users/dhruvgovil/Downloads/gramophone.usdz")

print("Works")

[dgovil]

Actually, I notice that at 40k iterations, newer versions of USD deadlock. I went down the iteration count and I think its related to the lock here https://github.com/PixarAnimationStudios/OpenUSD/issues/3249

Under 32k iterations, you still see the issue Mark describes, but over 32k iterations I think the hashing structure enters the infinite loop I also describe in that other issue.