search

visit-dav / visit

VisIt - Visualization and Data Analysis for Mesh-based Scientific Data
https://visit.llnl.gov
BSD 3-Clause "New" or "Revised" License
438 stars 116 forks source link

masonry should set PYTHONPATH (or equiv) when making manuals #17870

Open markcmiller86 opened 2 years ago

markcmiller86 commented 2 years ago

Describe the bug

I was trying to do a build with masonry from a partially built state. When I did, I encountered this error during the manuals_release phase (which is trying to run sphinx_build to make the manuals)...

Traceback (most recent call last):
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
ws.require(__requires__)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (Sphinx 5.0.2 (/Users/miller86/Library/Python/3.7/lib/python/site-packages), Requirement.parse('Sphinx==2.2.1'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/sphinx-build", line 6, in 
from pkg_resources import load_entry_point
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in 
@_call_aside
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
f(*args, **kwargs)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
return cls._build_from_requirements(__requires__)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Sphinx==2.2.1' distribution was not found and is required by the application
make[3]: *** [doc/CMakeFiles/manuals] Error 1
make[2]: *** [doc/CMakeFiles/manuals.dir/all] Error 2
make[1]: *** [doc/CMakeFiles/manuals.dir/rule] Error 2
make: *** [manuals] Error 2

Its a lengthy error message but the key point is that it winds up loading pkg_release from a wholly different python3 installation...the one that is part of Xcode. We've probably gotten lucky in the past that this didn't bite us somehow but to resolve, before we run sphinx_build we probably need to set PYTHONPATH to ensure python winds up getting all its modules from VisIt's python installation.

cyrush commented 2 years ago

I don't think that alone would fix it - if its still using the other python, even if PYTHONPATH is set, you could get very strange results.

Make manuals should be using the python passed to visit, do we know everything is ok there?

cyrush commented 2 years ago

Just to double check, --- Is masonry using VisIt's build system to make the manuals? (Calling our cmake targets) or is it doing them as a separate command?

markcmiller86 commented 2 years ago

Is masonry using VisIt's build system to make the manuals? (Calling our cmake targets) or is it doing them as a separate command?

haven't investigated that yet but I am not sure it matters. Either way would ultimately result in a sphinx-build command being invoked (either directly from masonry or indirectly from masonry calling make manuals and the makefiles running sphinx-build), right?

could the problem be that pkg_release winds up being an egg in our Python install? No matter what I do with the python installed with masonry for VisIt, I can't get it to find pkg_release there. It always resolves to the one in Xcode.

cyrush commented 2 years ago

ideally: make manual calls sphinx-build that exists inside of our python install. B/c of this it should be fully independent from the python used by masonry to bootstrap tpls and build visit.

cyrush commented 2 years ago

It is calling make manuals:

https://github.com/visit-dav/visit/blob/05a7248e1e11b16a43598f5b5257c1eecd9f9c62/src/tools/dev/masonry/bootstrap_visit.py#L215

Grumble, something seems off.

cyrush commented 2 years ago

do you have PYTHONPATH set in your env?

markcmiller86 commented 2 years ago

do you have PYTHONPATH set in your env?

Nope. env | grep -i py produces nothing.

Yes, it is calling make manuals which, after I threaded through the CMake nuggets, the actual command executed looks like...

cd /Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/build.release/doc && \ 
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/sphinx-build -b html \
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/visit/src/doc /Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/build.release/resources/help/en_US/manual -a

And this fails with...

Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (Sphinx 5.0.2 (/Users/miller86/Library/Python/3.7/lib/python/site-packages), Requirement.parse('Sphinx==2.2.1'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/sphinx-build", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Sphinx==2.2.1' distribution was not found and is required by the application

which is interesting because I do have a sphinx-build in ~/Library/Python/3.7/bin which is version 5.0.2. But that is NOT in my default path and, furthermore, I don't pass my default path with running masonry. I start masonry with env PATH=/usr/local/bin:/usr/bin:/bin:/sbin which is a vanilla path.

When I go examine the sphinx executables installed with VisIt via masonry, I find that they are all sh-banged with absolute path to Xcode.app python3...

find /Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7 -name '*.py' -exec grep 'Xcode.app' {} \; -print
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2xetex.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2latex.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2odt_prepstyles.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2html4.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2html5.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2xml.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2odt.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2pseudoxml.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2s5.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2html.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2man.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rstpep2html.py
            #   /Applications/Xcode.app/Contents/Developer/Platforms/
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/lib/python3.7/distutils/unixccompiler.py
    FRAMEWORKS_PATH = '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS8.1.sdk/System/Library/Frameworks/'
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/lib/python3.7/site-packages/Pygments-2.5.2-py3.7.egg/pygments/lexers/_cocoa_builtins.py
    #    can only be found inside Xcode.app if the "Command Line Tools"
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/lib/python3.7/_osx_support.py
markcmiller86 commented 2 years ago

Let me be a bit clearer...the search above examins only .py files that were installed to the Python bin directory VisIt created for itself. There are a number of other python scripts there that do not have .py extension and they are all sh-banged with abs path to Xcode.app python instead of this python.

[scratlantis:3.7.7/darwin-x86_64/bin] miller86% grep Xcode.app *
chardetect:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
cygdb:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
cython:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
cythonize:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
easy_install:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
easy_install-3.7:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
f2py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
f2py3:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
f2py3.7:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
pybabel:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
pygmentize:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2html.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2html4.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2html5.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2latex.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2man.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2odt.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2odt_prepstyles.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2pseudoxml.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2s5.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2xetex.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2xml.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rstpep2html.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
sphinx-apidoc:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
sphinx-autogen:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
sphinx-build:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
sphinx-quickstart:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
markcmiller86 commented 2 years ago

Changing the sh-bang in sphinx-build in the above from #!/Applications/Xcode.app/Contents/Developer/usr/bin/python3 to #!/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/python3 so that it winds up using the python3 installed as part of the VisIt build instead of the python3 in Xcode.app fixes the problem building the manuals.

So, somehow, we need to make sure that these tools all get installed with the proper shbang. I am guessing that somewhere in the build process we wound up using an unqualified python3 (resulting in whatever the path defaults too which for me results in

env PATH=/usr/local/bin:/usr/bin:/bin:/sbin which python3
/usr/bin/python3
env PATH=/usr/local/bin:/usr/bin:/bin:/sbin python3
Python 3.7.3 (default, Nov 15 2019, 04:04:52) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python37.zip', \
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7', \
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/lib-dynload', \
'/Users/miller86/Library/Python/3.7/lib/python/site-packages', \
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages', \
'/Library/Python/3.7/site-packages']
>>>
cyrush commented 2 years ago

I think when VisIt's Python was built, something had to be crossed with the system python.

If i look at my Sphinx build for 3.3, sh-bang has the VisIt built python:

  head build-mb-3.2.2-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/sphinx-build 

#!/Users/harrison37/Work/github/visit-dav/3.2.2_release/visit/build-mb-3.2.2-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'Sphinx==2.2.1','console_scripts','sphinx-build'
__requires__ = 'Sphinx==2.2.1'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])

Culprits are usually python in your path, or PYTHONPATH or even an anaconda install somewhere. But I know you checked all of those things. Mysteries like this are frustrating.

markcmiller86 commented 2 years ago

Ok, thanks for looking at the sh-bangs in your case. That is good to know it will get it right under the right conditions.

Well, I ensured $PATH is generic and there is no $PYTHONPATH in my env though I didn't think to explicitly override anything with env PYTHONPATH="" and I do have Anaconda installed but not "visible" to anything from my terminal.

By build completed but a) I accedentially vpn'd in to file TC in the middle of it...codesigning didn't fail but I haven't confirmed it yet and b) I am still finding some .so files in numpy and PIL with paths to /opt/local/lib...

    /opt/local/lib/libfreetype.6.dylib 
    /opt/local/lib/liblcms2.2.dylib 
    /opt/local/lib/libopenblas-r1.dylib 
    /opt/local/lib/libopenjp2.7.dylib 
    /opt/local/lib/libtiff.5.dylib 
    /opt/local/lib/libwebp.7.dylib 
    /opt/local/lib/libwebpdemux.2.dylib 
    /opt/local/lib/libwebpmux.3.dylib 
    /opt/local/lib/libxcb.1.dylib

I am investigating why this is happening with these two python modules.

markcmiller86 commented 2 years ago

@cyrush I got curious and examined the 3.2.2 release. I don't see anything there that depends on blas or openblas. So, I think the numpy build on my system is building some additional features we haven't necessarily supported in the past.

And, I think some of the python modules, like numpy, are kinda sorta designed to go looking for stuff in "special places" like in MacPorts (/opt/local/lib) or in Fink (/sw/include) because I see these paths in lists of paths used as part of the numpy install all over the numpy sources.

markcmiller86 commented 2 years ago

Sheesh...I've tried a ton of stuff to force numpy to NOT try to build with blas. Nothing worked. Then blew away the whole numpy-16.1 source tree and unzippped it afresh and am now able to build without blas. So, making progress on this and will have some tweeks to bv_python (I hope) shortly.

markcmiller86 commented 2 years ago

Ok, so I think I have issues with building numpy on a machine where libopenblas is available resolved.

Now looking at Pillow and comparing what I've recently built with my machine and what 3.2.2 has...

From my machine...

find /Volumes/VisIt-3.3.0/VisIt.app/Contents/Resources/3.3.0 -name '*.so' | grep PIL
/PIL/_imaging.cpython-37m-darwin.so
/PIL/_imagingcms.cpython-37m-darwin.so
/PIL/_imagingft.cpython-37m-darwin.so
/PIL/_imagingmath.cpython-37m-darwin.so
/PIL/_imagingmorph.cpython-37m-darwin.so
/PIL/_imagingtk.cpython-37m-darwin.so
/PIL/_webp.cpython-37m-darwin.so

From 3.3.2

miller86% find /Applications/VisIt.app/Contents/Resources/3.2.2/darwin-x86_64 -name '*.so' | grep PIL
/PIL/_imaging.cpython-37m-darwin.so
/PIL/_imagingmath.cpython-37m-darwin.so
/PIL/_imagingtk.cpython-37m-darwin.so
/PIL/_imagingmorph.cpython-37m-darwin.so

Because Pillow is actively looking in many places for libs it can use, it is finding stuff on my machine in /opt/local/lib and then building against that. So, it winds up building some more stuff than on @cyrush machine.

Our strategy is to avoid winding up with dependencies for libs in places other than standard ones like /usr/lib or /sys/lib but don't we then copy to the VisIt install point all the libs we need? If so, maybe we should be smarter about the operation that copies to the VisIt distro all the libs it depends.

@cyrus, which piece of the process is responsible for copying .so and .dylib files to VisIt install point?

markcmiller86 commented 2 years ago

Out of curiosity, why are we building PIL in python for a macOS binary user release in the first place?

markcmiller86 commented 2 years ago

I have MacPorts on my mac. I recall @cyrush warning against that a long while ago when I got my new Mac but I forgot to mention it to the team preparing my Mac and, in any event, I believe other work I’ve done has benefited from having MacPorts.

MacPorts puts a lot of useful stuff in /opt/local/{include,lib}

A lot of python tools installation processes are designed to interrogate commonly used locations (MacPorts, Fink, Homebrew, etc.) to help set their configuration when building. So, these find my MacPorts stuff and then depend on it. When the binary is installed on another user's system, if /opt/local/lib isn't there, things fail.

I think we have two options for our release build

I like the latter approach because it allows for more functionality, easily. For example, our numpy stuff build on Cyrus’ machine does not include any blas libs which hampers use of numpy for linear algebra stuff. Our Pillow stuff doesn’t handle jpeg or tiff files whereas it would if we took the later approach.