Running usdview results in "State file not found, a new one will be created" and then aborts on MacOs.

abrennec commented 2 years ago

Description of Issue

I have built and installed USD following the build instructions. Then I wanted to run the HelloWorld example in the /USD/extras/usd/tutorials folder. I ran the following commands:

python3 helloWorld.py
usdview helloWorld.usda

After running "usdview helloWorld.usda", the line "State file not found, a new one will be created." was printed to the command line; it then took some time of processor activity until the command line read "[1] 12841 terminated usdview HelloWorld.usda"

Steps to Reproduce

As described above.

System Information (OS, Hardware)

MacOS Monterey 12.4

Package Versions

Build Flags

First time I tried this, build flags were "--force-all"

Second time I tried this, build flags were "--ptex --openvdb --opencolorio --materialx --draco"

spiffmon commented 2 years ago

Hi @abrennec, the "state file not found" message is expected the first time you run usdview; unfortunately that second error does not give us much to go on. Can we assume you are pulling the 22.05 release version of USD?

abrennec commented 2 years ago

Hi @spiffmon, thank you very much for your quick reply! Yes, I just cloned the release repository yesterday. I'm using Python3.7, updated PATH and PYTHONPATH according to your documentation. Your message made me recheck my command line calls. I noticed that I always had to run the python calls as sudo. Now I have also run "sudo usdview HelloWorld.usda" and get this error message:

Traceback (most recent call last): File "/opt/local/USD/bin/usdview", line 28, in import pxr.Usdviewq as Usdviewq ModuleNotFoundError: No module named 'pxr'

I double checked the PYTHONPATH, it is correct (following https://graphics.pixar.com/usd/release/usdfaq.html), I also checked the /opt/local/USD/lib/Python/pxr folder. It is there, but maybe the installed Python version is not correct? Any hint's appreciated, thanks again! UPDATE: I just saw that I am using python 3.7.1, your doc says 3.7.7 ... sorry ... I will update and try again.

meshula commented 2 years ago

@abrennec If you were building into /usr/local on a recent MacOS, you'd definitely need sudo. But /opt/local isn't protected, so you shouldn't need sudo. Either way, building with sudo will lead to on-going difficulties whenever run time permissions aren't lined up with your build scenario.

I'd recommend a fresh build, with no sudo. If it's impossible to build without sudo, the next step should be to discover why that is.

When I am testing a usd build, I usually build into a throw-away place; I like to use /var/tmp because I know it's safe to delete anything in that directory.

So my test command line is often like this -

python /path/to/USD/build_scripts/build_usd.py /var/tmp/test-usd-Jun12 --no-usdview

Notice I put the date in there, the reason is just so that I don't confuse myself when I have a lot of test builds. I also do --no-usdview for testing because it speeds up the build, and I can use one of the tools like sdfdump or, start python and import pxr.usd as usd to quickly determine if things are working or not.

When the build finishes, I copy and paste the environment variable settings into a note for convenience, as well as setting the variables immediately for testing.

If I'm satisfied that things are working properly, then I build usd such that it's more generally available. For example you are installing into /opt/local. I only do something like that when I know I've got everything ready to go. Also, that's the time when I build with usdview enabled, since I know I have the basics covered.

Hope this helps!

abrennec commented 2 years ago

Hello @meshula, thank you very much! This helps me a lot! I will gladly follow your advice and try to fix the sudo thing first. Then I will rebuild and try your approach (no-usdview, python import).

( By the way, rebuilding with Python3.7.7 did not fix the problem. )

Thanks again!

sunyab commented 2 years ago

Filed as internal issue #USD-7425

creijon commented 2 years ago

@abrennec, great to see that you're getting the macOS release up.

From macOS 12.3 onwards we removed python2, so you'll have to run python3 which is 3.8.9 I believe. Are you using a separate install of Python rather than the system one? If so, make sure that the same version is used for both the build and launching usdview. It should be in /usr/bin/python3

Thanks for double-checking the PYTHONPATH, but the error you're getting (ModuleNotFoundError: No module named 'pxr') is a very symptomatic of this path being invalid. I'd triple-check it just to be sure. I've had errant spaces get in there that have caused problems.

As Nick says, avoid building under sudo since it causes problems later on. I build to a folder under my /Users/jonny-apple home.

Let me know how you get on.

abrennec commented 2 years ago

@jonny-apple Many thanks for your response! I am still on it, following @meshula's steps .. did a rebuilt of usd w/o usdview in a local folder under my /Users/.. dir. This worked fine without sudo. I have installed python 3.7.7 after checking the versions file. Maybe that was not necessary (?). If not, I will try to get back to the default python version which comes with Monterey 12.4, python 3.8.9.

Apart from that and with the current setup, pxr is still not found. I tried import pxr.usd as usd which leads to the ModuleNotFound error message. So you are probably right there is sth wrong with the PYTHONPATH and environment variable settings. I could not yet figure out what it is, but I am working on it :) I'll keep you posted. Many thanks for the help!

abrennec commented 2 years ago

Hi everyone, I have proceeded a bit but unfortunately my python setup is still not working correctly. It would be great if you could take another look at this:

I have cleaned my system from any user-installed python version. I am working with python3.8.9 on MacOS 12.4 (if I am not mistaken, the terminal commands yield the following)

$ Python3 --version
Python 3.8.9

$ which Python3 
/usr/bin/Python3

I have done a fresh rebuilt of USD in a local folder using --no-usdview (OpenImageIO/libtiff works btw, many thanks!). Then I updated the PATH settings. These were messed up before but are correct now afai understand it. I set them according to the installation as follows:

The following in your PYTHONPATH environment variable:
    /Users/abrennec/Documents/Develop/USD/tmp/220615_usd_build2/lib/python

    The following in your PATH environment variable:
    /Users/abrennec/Documents/Develop/USD/tmp/220615_usd_build2/bin

Now, when I call sdfdump like below, the PYTHONPATH somehow points to Python3.framework, the message also tells me that '/System/Library/Frameworks/Python3.framework/Versions/3.8/Python3' (no such file)' although I have not touched that folder, so I am a bit lost at this point. I appreciate any further hint.

➜  $ sdfdump HelloWorld.usda
dyld[87745]: Library not loaded: @rpath/Python3.framework/Versions/3.8/Python3
  Referenced from: /Users/abrennec/Documents/Develop/USD/tmp/220615_usd_build2/bin/sdfdump
  Reason: tried: '/Users/abrennec/Documents/Develop/USD/tmp/220615_usd_build2/lib/Python3.framework/Versions/3.8/Python3' (no such file), 
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/Python3.framework/Versions/3.8/Python3' (no such file), 
'/Users/abrennec/Documents/Develop/USD/tmp/220615_usd_build2/lib/Python3.framework/Versions/3.8/Python3' (no such file), 
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/Python3.framework/Versions/3.8/Python3' (no such file), '/Library/Frameworks/Python3.framework/Versions/3.8/Python3' (no such file), 
'/System/Library/Frameworks/Python3.framework/Versions/3.8/Python3' (no such file)
[1]    87745 abort      sdfdump HelloWorld.usda

... and finally otool yields the following:

$ otool -L sdfdump
sdfdump:
    @rpath/libusd_sdf.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libboost_program_options.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libusd_ar.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libusd_plug.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libusd_work.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libusd_vt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libusd_trace.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libusd_js.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libusd_gf.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libusd_tf.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libusd_arch.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)
    @rpath/Python3.framework/Versions/3.8/Python3 (compatibility version 3.8.0, current version 3.8.0)
    @rpath/libboost_python38.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libtbb.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.23.0)

meshula commented 2 years ago

Can you attach your /Users/abrennec/Documents/Develop/USD/tmp/220615_usd_build2/build/USD/CMakeCache.txt file? That should tell us where CMake thinks Python is. The build seemed to want to find it here @rpath/Python3.framework/Versions/3.8/Python3, hopefully the CMakeCache.txt will provide an essential clue.

abrennec commented 2 years ago

Sure, here you go: CMakeCache.txt

meshula commented 2 years ago

Okay, that confirms that the build is occurring versus the Python embedded within your copy of Xcode.... Not what one would want.

Doing a quick test locally by running /usr/bin/python3 -m site I get ~


sys.path = [
    '/private/var/tmp',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages',
]
USER_BASE: '/Users/nporcino/Library/Python/3.8' (doesn't exist)
USER_SITE: '/Users/nporcino/Library/Python/3.8/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

This must match your set up, as that's very similar to what I see in your CMakeCache.txt.

I'm not sure how to make usd work using Xcode's python... I'm personally using miniconda to manage a python environment for usd, and building within that environment. Other people are installing python via homebrew which also works, as does the official python installer. Any of those solutions would work. I wonder if anyone else has guidance about using the xcode-embedded python? Without that input, I'd suggest trying miniconda or homebrew, if that's applicable in your situation.

sunyab commented 2 years ago

My Mac test environments are using the Python 3 that ships with MacOS / Xcode and not homebrew but I haven't seen this issue. I wonder if there's a problem with the Python library discovery in build_usd.py, that recently changed in 43902541 in the dev branch.

meshula commented 2 years ago

Sunya, for completeness, could you check /your/path/to/python -m site? I'd like to know if yours resolves as mine inside Xcode across the board.

abrennec commented 2 years ago

Doing a quick test locally by running /usr/bin/python3 -m site I get ~


sys.path = [
    '/private/var/tmp',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages',
]
USER_BASE: '/Users/nporcino/Library/Python/3.8' (doesn't exist)
USER_SITE: '/Users/nporcino/Library/Python/3.8/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

This must match your set up, as that's very similar to what I see in your CMakeCache.txt.

Thanks, @meshula , yes, it is nearly the same. I get this

➜  ~ /usr/bin/python3 -m site        
sys.path = [
    '/Users/abrennec',
    '/Users/abrennec/Documents/Develop/USD/tmp/220615_usd_build2/lib/python',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
    '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages',
]
USER_BASE: '/Users/abrennec/Library/Python/3.8' (doesn't exist)
USER_SITE: '/Users/abrennec/Library/Python/3.8/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

So my next step would be setting up a separate python environment with miniconda or the official installer and try it again.

meshula commented 2 years ago

We've been able to reproduce this problem on a few machines, but only on latest OS/Xcode combos. A workaround that we tried successfully was to use a 3.9 release from here: https://www.python.org/downloads/ We haven't tested 3.10 using that method. I'm using miniconda and python 3.9, but of course conda has a slight learning curve. A workaround like that should get you up and running while we continue to investigate.

abrennec commented 2 years ago

Thanks a lot for your response and all of the effort you put into this. I have started to look into miniconda and will rebuilt everything as soon as my python setup works. Many thanks again!

abrennec commented 2 years ago

Following your recommendations, I have done a couple of further tests:

set up conda and rebuilt usd -> this resulted in a python error; researched the issue and found this patch (https://github.com/jwarrencode/USD/commit/096af14f71d6a4ccb80bf9155f99024290db982b)
rebuilt usd with the patch -> this resulted in an sdfdump error but usdview started and showed helloworld.usda'sphere !!
deactivated conda and installed the official python installer -> this resulted in sdfdump working properly but usdview did not start, it seemed python launcher got frozen

I have documented the steps in the following file; I hope this helps. At the moment, I can't seem to get it to work properly. If I can help with further tests, please let me know. 220617_usd_build_tests.md

meshula commented 2 years ago

You are getting very close. Continuing on with your attempt using the official python installer, the locking up sounds familiar. I think that behavior started with pyside2 5.15 or 5.14. Could you try uninstalling pyside2, and install pyside2 5.13.2? I think that was the last one that worked fully on MacOS. Someone please chime in if there is a specific version known to work properly other than 5.13.2...

The command line for a specific version is

python -m pip install PySide2==5.13.2

abrennec commented 2 years ago

Ok, changed PySide2 successfully :)

Now, unfortunately, the usd build fails and prints this message:

make: *** [all] Error 2

ERROR: Failed to run 'cmake --build . --config Release --target install -- -j4'
See /Users/abrennec/Documents/Develop/USD/tmp/220617_usd_build4_installer/build/usd_release/log.txt for more details.

Looking into PATH_TO/USD/tmp/220617_usd_build4_installer/build/usd_release/log.txt shows

An unexpected error occurred.
Check that you are using the latest version of PySide2 and report the error to
http://bugs.openbossa.org, including the ui file used to trigger the error.
[ 23%] Copying mesh.h ...
0
1
0
1
2
make[2]: *** [pxr/usdImaging/usdviewq/mainWindowUI.py] Error 1
make[2]: *** Deleting file `pxr/usdImaging/usdviewq/mainWindowUI.py'
make[1]: *** [pxr/usdImaging/usdviewq/CMakeFiles/_usdviewq_pysideuifiles.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Should I use another PySide version?

meshula commented 2 years ago

Did you uninstall pyside2 and re-run your build, or did you uninstall pyside2 and run a brand new fresh build? pyside-uic is matched to pyside version, so re-running would cause a mismatch.

abrennec commented 2 years ago

I did a fresh build of usd. First I uninstalled pyside2 5.15, then installed pyside2 5.13.2 following your command, then re-run the official build script ` python3 build_scripts/build_usd.py ../tmp/220618_usd_build_installer. When it comes to building usdview, the build fails if I understand it correctly.

building w/o usdview works (just for completion)
also checked the pyside-uic version: PySide2 User Interface Compiler version , running on PySide2 5.13.2
here is the full log.txt 220618_log.txt

meshula commented 2 years ago

By fresh build, I meant, into a new directory. This message ~

[ 23%] Generating Python for /Users/abrennec/Documents/Develop/USD/usd_release/pxr/usdImaging/usdviewq/mainWindowUI.ui ...
An unexpected error occurred.
Check that you are using the latest version of PySide2 and report the error to
http://bugs.openbossa.org, including the ui file used to trigger the error.

Usually means that the pyside generator and generated source are out of sync. So it seems like mainWindowUI.cpp was generated by pyside2 5.15, and is incompatible with 5.13.2. Unfortunately Pyside is a bit fragile in this way. The easiest way to avoid the incompatibility between pyside versions is to build into a new directory or scour out the generated files ~ but digging around to find them probably takes as long as just starting over completely by deleting everything or building to a different location.

Apologies in advance if if that's what you already did, but usually, that error message means that generated files from a different pyside version are still hanging around.

abrennec commented 2 years ago

Hi @meshula , thanks again for your support and effort, really appreciated. I understood that there must be files from different pyside versions hanging around that cause the error. That's why I re-ran the usd build scripts and did a fresh usd build into a new directory (that's what you meant, right?).

Today, I did everything again just to make sure I did not miss anything. The build fails with the same error message. To be really fresh, I removed everything, all usb builds, usd-repo, conda, python installer (and every symlink etc). Pulled the usd repo, installed the official python 3.9 installer, installed pyopengl and the pyside version you specified. Rebuild usd .. it fails with the same error message. So it still looks like there are mismatched versions used to create the UI, but I don't know exactly what to look for to fix it.

meshula commented 2 years ago

Ok, thanks for confirming that!

r-alias commented 2 years ago

I'm encountering the same problem.

I downloaded python3.7 from the official site (because I'm using M1Mac and the pyside2 package does not exist in the arm version of python3. ) and install from pip pip3 install pyopengl pyside2==5.13.2

but now i can display usdview in my environment by doing this.

I added environment variables QT_MAC_WANTS_LAYER=1 as described on this page. https://www.loekvandenouweland.com/content/pyside2-big-sur-does-not-show-window.html and then started usdview.

% export QT_MAC_WANTS_LAYER=1
% usdview USD/extras/usd/tutorials/convertingLayerFormats/Sphere.usda
State file not found, a new one will be created.
ERROR: Usdview encountered an error while rendering.('Unable to load OpenGL library', "dlopen(OpenGL, 0x000A): tried: 'OpenGL' (no such file), '/usr/lib/OpenGL' (no such file), '/Users/daiki/Documents/projects/OpenGL' (no such file)", 'OpenGL', None)

The window appears, but the preview screen is still not visible, so modify ctypesloader.py according to this comment. https://github.com/PixarAnimationStudios/USD/issues/1372#issuecomment-716925973

fullName = '/System/Library/Frameworks/OpenGL.framework/OpenGL'

ctypesloader.py exists in the following paths in my environment

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/OpenGL/platform/ctypesloader.py

creijon commented 2 years ago

Hi @r-alias we have build script and python changes to for native Apple Silicon support ready, but they aren't in the dev or release branches yet. So at the moment you'll have to build usdview under Rosetta on an M1 Mac. use:

arch -x86_64 /bin/zsh

Before running pip3 to make sure everything is installed correctly. And then run the python3 build_usd.py command with your required options.

r-alias commented 2 years ago

Thank you @jonny-apple ! I looking forward to the native Apple silicon support!

abrennec commented 2 years ago

I'm encountering the same problem.

I downloaded python3.7 from the official site (because I'm using M1Mac and the pyside2 package does not exist in the arm version of python3. ) and install from pip pip3 install pyopengl pyside2==5.13.2

but now i can display usdview in my environment by doing this.

I added environment variables QT_MAC_WANTS_LAYER=1 as described on this page. https://www.loekvandenouweland.com/content/pyside2-big-sur-does-not-show-window.html and then started usdview.
% export QT_MAC_WANTS_LAYER=1
% usdview USD/extras/usd/tutorials/convertingLayerFormats/Sphere.usda
State file not found, a new one will be created.
ERROR: Usdview encountered an error while rendering.('Unable to load OpenGL library', "dlopen(OpenGL, 0x000A): tried: 'OpenGL' (no such file), '/usr/lib/OpenGL' (no such file), '/Users/daiki/Documents/projects/OpenGL' (no such file)", 'OpenGL', None)
The window appears, but the preview screen is still not visible, so modify ctypesloader.py according to this comment. #1372 (comment)
fullName = '/System/Library/Frameworks/OpenGL.framework/OpenGL'
ctypesloader.py exists in the following paths in my environment
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/OpenGL/platform/ctypesloader.py

Thanks a lot @r-alias the QT setting works for me as well. Very nice : )

drwave commented 2 years ago

Sounds like your PYTHONPATH is not finding the USD python libraries.

Perhaps you can share how you built USD and what your PYTHONPATH is?

On Jun 12, 2022, at 7:39 PM, abrennec @.***> wrote:

Hi @spiffmon https://github.com/spiffmon, thank you very much for your quick reply! Yes, I just cloned the release repository yesterday. I'm using Python3.7, updated PATH and PYTHONPATH according to your documentation. Your message made me recheck my command line calls. I noticed that I always had to run the python calls as sudo. Now I have also run "sudo usdview HelloWorld.usda" and get this error message:

Traceback (most recent call last): File "/opt/local/USD/bin/usdview", line 28, in import pxr.Usdviewq as Usdviewq ModuleNotFoundError: No module named 'pxr'

I double checked the PYTHONPATH, it is correct (following https://graphics.pixar.com/usd/release/usdfaq.html https://graphics.pixar.com/usd/release/usdfaq.html), I also checked the /opt/local/USD/lib/Python/pxr folder. It is there, but maybe the installed Python version is not correct? Any hint's appreciated, thanks again!

— Reply to this email directly, view it on GitHub https://github.com/PixarAnimationStudios/USD/issues/1903#issuecomment-1153243139, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM2AOWJ7MAAGSUKK5GBRXDVOYOFTANCNFSM5YRVSPBQ. You are receiving this because you are subscribed to this thread.

PixarAnimationStudios / OpenUSD