search

Eigenstate / vmd-python

Installable VMD as a python module
Other
129 stars 24 forks source link

Running into build problems trying to build for an IBM Power9 system #22

Open austinpagan opened 5 years ago

austinpagan commented 5 years ago

Before I dig too deep into this, I just wanted to know whether vmd-python was ever intended to run on Power systems...? If so, I will append the particular problem I'm seeing.

Eigenstate commented 5 years ago

At some point, yes, but timing is likely to be on the order of weeks. All that needs to be done is getting setuptools/distutils to get the right flags to the actual build, but setuptools is a mess and getting build/install to both work has been pretty time consuming and I need to do my actual work.

You're welcome to mess with setup.py to get it working and I'd be happy to merge any pull requests you send my way.

austinpagan commented 5 years ago

I am serene about this. I've never seen a "setup.py" file before, but I shall take this as my opportunity to bootstrap myself into a working level of knowledge about all things "python module distribution." I'm not confident that I'll be successful, but it's better than waiting.

I will say, I'm also unschooled in the concept of a "pull request," so I may just ship you my version of setup.py in an email, if I get it running... hope that's OK?

austinpagan commented 5 years ago

I've deleted my previous comment.
I'm still seeing the problem.
I had accidentally built it without the "--egl" flag. When I include that flag, I'm back to where I was before... sorry for the confusion.

Eigenstate commented 5 years ago

Alright I've fixed the build so the flags propagate correctly. I've merged the ppc64le changes, so now we're on a new branch called egl and you won't have to build and install separately.

git checkout egl python setup.py --egl install

If it doesn't work, rebuild to get debug output and paste what you get when you import vmd: python setup.py --egl --debug install

austinpagan commented 5 years ago

Sorry, but I'm not very experienced with github... here's what I did:

[root@f48872560257 vmd-python]# pwd /vmd-1.9.3/vmd-python [root@f48872560257 vmd-python]# ls -a . .binstar.yml .bumpversion.cfg .git LICENSE meta.yaml setup.py .travis.yml .. build docs .gitignore MANIFEST.in README.md test vmd [root@f48872560257 vmd-python]# git checkout egl error: pathspec 'egl' did not match any file(s) known to git. [root@f48872560257 vmd-python]#

austinpagan commented 5 years ago

instead, I tried this:

git clone --branch egl https://github.com/Eigenstate/vmd-python cd vmd-python python setup.py --egl install

This is building... will advise.

austinpagan commented 5 years ago

So, it eventually died, complaining that it couldn't find GL and Xext. I did the debug version, and here's that output:

[root@f48872560257 vmd-python]# python setup.py --egl --debug install running install running build Finding libraries... LIB: libtcl8.5.so -> /usr/lib64/ INC: tcl.h -> /usr/include/ LIB: libsqlite3.so -> /usr/lib64/ INC: sqlite3.h -> /usr/include/ LIB: libexpat.so -> /usr/lib64/ INC: expat.h -> /usr/include/ LIB: libnetcdf.so -> /usr/lib64/ INC: netcdf.h -> /usr/include/ Building with debug symbols Traceback (most recent call last): File "setup.py", line 451, in 'test': VMDTest, File "/usr/lib/python2.7/site-packages/setuptools/init.py", line 143, in setup return distutils.core.setup(**attrs) File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 417, in run self.run_command("build") File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 71, in run self.compile_vmd() File "setup.py", line 85, in compile_vmd self.set_environment_variables(pydir) File "setup.py", line 332, in set_environment_variables asandir = self._find_library_dir("libasan") File "setup.py", line 216, in _find_library_dir libfile); RuntimeError: Could not find library 'libasan.so' in standard library directories. Update $LD_LIBRARY_PATH to include the directory containing this file, or make sure it is present on your system [root@f48872560257 vmd-python]#

austinpagan commented 5 years ago

Notably, when I cleared out the directory, re-cloned it, and tried to build with the debug, it died instantly the same way, rather than building a bunch of stuff for ten minutes. I wonder if there's a bug in the usage of the '--debug' flag...

damora commented 5 years ago

@austinpagan have you verified that you have this library somewhere on the system? RuntimeError: Could not find library 'libasan.so' in standard library directories. Update $LD_LIBRARY_PATH to include the directory containing this file, or make sure it is present on your system

austinpagan commented 5 years ago

"ls -ltR | grep libasan" from the root directory returns nothing.

"yum install libasan" says there's no such package.

Jeff found an rpm: libasan-9.0.1-0.8.fc30.ppc64le.rpm

We tried "rpm -i libasan-9.0.1-0.8.fc30.ppc64le.rpm" and it said:

warning: libasan-9.0.1-0.8.fc30.ppc64le.rpm: Header V3 RSA/SHA256 Signature, key ID 3c3359c4: NOKEY error: Failed dependencies: libm.so.6(GLIBC_2.23)(64bit) is needed by libasan-9.0.1-0.8.fc30.ppc64le

I then tried "yum localinstall libasan-9.0.1-0.8.fc30.ppc64le.rpm", and it got further, but eventually also said:

Error: Package: libasan-9.0.1-0.8.fc30.ppc64le (/libasan-9.0.1-0.8.fc30.ppc64le) Requires: libm.so.6(GLIBC_2.23)(64bit)

I tried to search for anything that contained the string "libm.":

[root@f48872560257 vmd-1.9.3]# yum search 'libm.' Loaded plugins: fastestmirror, ovl Loading mirror speeds from cached hostfile

So I'm stumped...

austinpagan commented 5 years ago

I looked into building libasan (which apparently is a package called "Address Sanitizer") from source. The web page https://github.com/google/sanitizers/wiki/AddressSanitizerHowToBuild told me to do: svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm But "svn" is not found on my system. I did a "yum search" on "svn" and found a package called "git-svm". I did a "yum install" on "git-svm". Now, svn is on my system, and "svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm" checked out a bunch of code! The next step involved "cmake" but it failed saying I needed cmake version 3.4.3, but yum tells me that my version (2.8.12) is the latest available.

So, I'm stumped AGAIN.

Eigenstate commented 5 years ago

Don't worry about building ASAN - it's a library used to help me debug things but won't help with your specific problem. I know what's going on, just hold off on doing anything until I can fix it. If you could paste the output of what "complaining it can't find GL and Xext" that would be helpful.

austinpagan commented 5 years ago

I am so sorry that I did not notice you posted this reply. I trust my work email to notify me of these things, and it let me down... sigh

The last lines before the error look like this:

Compiling hash.c --> hash.o ... Compiling inthash.c --> inthash.o ... Compiling intstack.c --> intstack.o ... Compiling ptrstack.c --> ptrstack.o ... Compiling msmpot.c --> msmpot.o ... Compiling msmpot_compute.c --> msmpot_compute.o ... Compiling msmpot_cubic.c --> msmpot_cubic.o ... Compiling msmpot_setup.c --> msmpot_setup.o ... Compiling vmdsock.c --> vmdsock.o ... vmdsock.c: In function ‘vmdsock_accept’: vmdsock.c:133:30: warning: pointer targets in passing argument 3 of ‘accept’ differ in signedness [-Wpointer-sign] rc = accept(s->sd, (struct sockaddr ) &s->addr, &len); ^ In file included from /usr/include/netinet/in.h:24:0, from /usr/include/arpa/inet.h:22, from vmdsock.c:35: /usr/include/sys/socket.h:244:12: note: expected ‘socklen_t restrict’ but argument is of type ‘int *’ extern int accept (int fd, SOCKADDR_ARG __addr, ^ Compiling vmddlopen.c --> vmddlopen.o ... Compiling pcre.c --> pcre.o ... Compiling fitrms.c --> fitrms.o ... Linking libvmd.so ... /usr/bin/ld: cannot find -lGL /usr/bin/ld: cannot find -lXext collect2: error: ld returned 1 exit status

austinpagan commented 5 years ago

On this system, I find these libraries here:

[root@f48872560257 lib64]# pwd /usr/lib64 [root@f48872560257 lib64]# ls libGL. libGL.so.1 libGL.so.1.7.0 [root@f48872560257 lib64]# ls libXext. libXext.so.6 libXext.so.6.4.0 [root@f48872560257 lib64]#

damora commented 5 years ago

/usr/lib64/nvidia/

austinpagan commented 5 years ago

@damora I was hoping for a verb... (shy smile) So, in /usr/lib64, I did "mkdir nvidia". Then I copied the two libraries into that new directory. Then I re-created the links to those libraries to mirror what was happening in the original directory. Then I cleared out the vmd-python directory, and cloned a fresh one. When I rebuilt, it gave me the exact same error as I documented above. I shall now wait for wisdom from the package author, unless I misinterpreted your message, and you can clarify for me...

austinpagan commented 5 years ago

I woke with an inspiration. Maybe I also needed to create links that ended in .so (or perhaps .a)? So I created those links (both in /usr/lib64 and /usr/lib64/nvidia), and now the command python setup.py --egl install ends like this:

byte-compiling /usr/lib/python2.7/site-packages/vmd/init.py to init.pyc byte-compiling /usr/lib/python2.7/site-packages/vmd/Molecule.py to Molecule.pyc byte-compiling /usr/lib/python2.7/site-packages/vmd/Label.py to Label.pyc byte-compiling /usr/lib/python2.7/site-packages/vmd/Material.py to Material.pyc running install_egg_info running egg_info creating vmd_python.egg-info writing vmd_python.egg-info/PKG-INFO writing top-level names to vmd_python.egg-info/top_level.txt writing dependency_links to vmd_python.egg-info/dependency_links.txt writing manifest file 'vmd_python.egg-info/SOURCES.txt' reading manifest file 'vmd_python.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching 'bld.bat' warning: no files found matching 'build.sh' no previously-included directories found matching 'vmd/vmd_src/LINUXAMD64' no previously-included directories found matching 'vmd/plugins/build' writing manifest file 'vmd_python.egg-info/SOURCES.txt' Copying vmd_python.egg-info to /usr/lib/python2.7/site-packages/vmd_python-3.0.2-py2.7.egg-info running install_scripts [root@f48872560257 vmd-python]#

Does that look like a proper end to the process?

I will now go rooting around for test programs to run, to see if this is all properly built.

austinpagan commented 5 years ago

I started a "python session" and typed in the first couple of lines from your README: from vmd import molecule, vmdnumpy import numpy as np mask = vmdnumpy.atomselect(molid_avg, 0, "resname TYR")

The first command resulted in this:

[root@f48872560257 vmd-python]# python Python 2.7.5 (default, Oct 30 2018, 23:44:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

from vmd import molecule, vmdnumpy /usr/lib/python2.7/site-packages/vmd/init.py:17: UserWarning: Module vmd was already imported from /usr/lib/python2.7/site-packages/vmd/init.pyc, but /vmd-1.9.3/vmd-python is being added to sys.path from pkg_resources import resource_filename Info) EGL: node[0] bound to display[0], 4 displays total

Then followed a pile of warnings that looked like this:

Warning) Unable to dlopen '/usr/lib/python2.7/site-packages/vmd/plugins/LINUXPPC64/molfile/xsfplugin.so': Unable to dlopen plugin file /usr/lib/python2.7/site-packages/vmd/plugins/LINUXPPC64/molfile/xsfplugin.so Warning) Unable to dlopen '/usr/lib/python2.7/site-packages/vmd/plugins/LINUXPPC64/molfile/pbeqplugin.so': Unable to dlopen plugin file /usr/lib/python2.7/site-packages/vmd/plugins/LINUXPPC64/molfile/pbeqplugin.so

Then, the next two commands looked like this:

import numpy as np mask = vmdnumpy.atomselect(molid_avg, 0, "resname TYR") Traceback (most recent call last): File "", line 1, in NameError: name 'molid_avg' is not defined

Are those warnings OK? Does your distribution have any sample molecules I can use?

austinpagan commented 5 years ago

Also, I removed the "nvidia" directory from /usr/lib64, as the comment from @damora was intended for a different experiment I was trying, not documented in this stream, and I misunderstood him. My apologies.

austinpagan commented 5 years ago

Here's the list of links that got me past the problem:

[root@f48872560257 lib64]# ls -lt libGL. lrwxrwxrwx 1 root root 14 Mar 18 11:32 libGL.a -> libGL.so.1.7.0 lrwxrwxrwx 1 root root 14 Mar 18 11:31 libGL.so -> libGL.so.1.7.0 lrwxrwxrwx 1 root root 14 Jan 23 16:15 libGL.so.1 -> libGL.so.1.7.0 -rwxr-xr-x 1 root root 803280 Oct 30 21:26 libGL.so.1.7.0 [root@f48872560257 lib64]# ls -lt libXext. lrwxrwxrwx 1 root root 16 Mar 18 11:32 libXext.a -> libXext.so.6.4.0 lrwxrwxrwx 1 root root 16 Mar 18 11:31 libXext.so -> libXext.so.6.4.0 lrwxrwxrwx 1 root root 16 Jan 23 16:15 libXext.so.6 -> libXext.so.6.4.0 -rwxr-xr-x 1 root root 137232 Nov 22 2015 libXext.so.6.4.0 [root@f48872560257 lib64]#

austinpagan commented 5 years ago

On the same day that I created this problem, you commented as follows:

"Remove PBuffer support for now as getting a universal build is tricky I need libOpenGL.so to link against for headless systems, but it's not readily available on a lot of distros (including Travis' Ubuntu 14). Also, there's no conda package for it, so I'm tabling this for now"

Is this also true in our "egl" branch? If not, could you show me a version of the "render" test that I could use to try rendering something to a PBuffer?