Non gcc/linux64 portability issues

[ronawho]

Looks like the backend compiler is always given -static and --whole-archive options. The former is not supported on OSX, and the latter is a gcc flag. Other compilers might support it (I didn't check pgi, intel, or cray) but I'm pretty sure clang doesn't

[awallace-cray]

Confirmed, pychapel + clang on OS-X will not work. Gcc link commands (and other things) are largely hard-coded in pychapel/module/configs/pych.json, will only work with linux64, gcc, system-installed python 2.7.

Chapel itself includes a nice build system which supports compilation and linking of all things "Chapel", in many environments. Pychapel should take advantage of that. For example, instead of using the hard-coded gcc command line template in pychapel/module/configs/pych.json, perhaps Pychapel could "ask" chpl (or a new target in Chapel's Make system, or a new chplenv Python utility) for the appropriate link command for the current environment.

[lydia-duncan]

@ben-albrecht and I experimented with just sending the clang equivalent of --whole-archive instead and were able to get tests to run successfully. We have an action item for one of us to add the detection necessary to work in both situations

[ben-albrecht]

That flag is -all_load for clang. We also removed --no-whole-archive.

For completeness, here's the patch that will get OSX working for PyChapel:

https://gist.github.com/ben-albrecht/f02b33dbefad74a8629de49cab795628

[awallace-cray]

Thinking about creating/building a Pychapel Homebrew formula using a temporarily-patched Pychapel source tree. Even if the patch is not on master branch yet.

[awallace-cray]

Unable to get all pychapel unit tests to work on my Macbook.

• avoided some errors by rm'ing -fopenmp from gcc commandline template
• test cases using matplotlib are still broken

https://gist.github.com/awallace-cray/64e9d2d94349f44907556aabc3e27e10

[ben-albrecht]

FWIW, @lydia-duncan and I did not run the full test suite with these changes.

This SO post might be helpful for matplotlib setup (assuming it's not PyChapel-specific)

[awallace-cray]

That backend change gets the tests to run- good! I wonder if a user app could actually make a plot. I'm not going to stop and check.

[awallace-cray]

Created a set of rudimentary build scripts for Pychapel on Mac OS X. https://gist.github.com/awallace-cray/31da7c2f2333e56cb7949f94d5fcea5f

[awallace-cray]

~https://gist.github.com/awallace-cray/31da7c2f2333e56cb7949f94d5fcea5f no longer works. See issue #75~ 2018-04-03 : It was fixed and re-tested 01/23. https://gist.github.com/awallace-cray/31da7c2f2333e56cb7949f94d5fcea5f#gistcomment-2331061. Sorry, I missed noting that here.

[buddha314]

In an empty dir, ran this and got

+ :
+ : Build current pychapel on Mac OS X 10.11 as of 2017-08-16 11:22, updated 15:50
+ :
+ : pychpl-2-full-pych.sh : Pychapel, full install. One-time-only for new user/env.
+ :
+ : This script should be copied to an otherwise-empty directory, and run interactively.
+ : Pychapel will be downloaded there and built.
+ :
+++ dirname int.sh
++ cd .
++ pwd
+ WORK=/Users/buddha/tmp
+ cd /Users/buddha/tmp
+ ls -la
total 8
drwxr-xr-x    3 buddha  staff   102 Apr  3 10:01 .
drwxr-xr-x+ 175 buddha  staff  5950 Apr  3 09:59 ..
-rw-r--r--    1 buddha  staff  3399 Apr  3 09:59 int.sh
+ :
+ : 'Please "deactivate" any Python virtual env in the calling shell first.'
+ : 'If your interactive bash prompt starts with "(pychapel)", exit now; run "deactivate"; try again.'
+ :
+ : VIRTUAL_ENV= : it should be empty
+ test -z ''
+ :
+ : THIS WILL COMPLETELY REMOVE ANY EXISTING INSTALLATION.
+ :
+ : 'Ready?' Hit return to continue, or interrupt to start over.
+ read OK
OK
+ rm -rf /Users/buddha/tmp/.virtualenvs/pychapel
+ export WORKON_HOME=/Users/buddha/tmp/.virtualenvs
+ WORKON_HOME=/Users/buddha/tmp/.virtualenvs
+ export PATH=/usr/local/opt/python/libexec/bin:/usr/local/bin:/Users/buddha/.jenv/shims:/Users/buddha/.jenv/bin:/Users/buddha/github/chapel/bin/darwin:/Users/buddha/github/chapel/util:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/share/dotnet:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Library/TeX/texbin
+ PATH=/usr/local/opt/python/libexec/bin:/usr/local/bin:/Users/buddha/.jenv/shims:/Users/buddha/.jenv/bin:/Users/buddha/github/chapel/bin/darwin:/Users/buddha/github/chapel/util:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/share/dotnet:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Library/TeX/texbin
+ : source virtualenvwrapper.sh
+ set +ex
int.sh: line 38: virtualenvwrapper.sh: No such file or directory
+ :
+ : Initialize Python virtual env to be used w Pychapel
+ :
+ : mkvirtualenv -p /usr/local/bin/python2 pychapel
+ set +ex
int.sh: line 46: mkvirtualenv: command not found
+ :
+ : VIRTUAL_ENV= : it should not be empty
+ test -n ''

[awallace-cray]

It looks like https://gist.github.com/awallace-cray/31da7c2f2333e56cb7949f94d5fcea5f#file-pychpl-2-full-pych-sh was run without preparing the machine as described in https://gist.github.com/awallace-cray/31da7c2f2333e56cb7949f94d5fcea5f#file-pychpl-0-readme-txt. At least, the virtualenv tools do need to be installed.

[buddha314]

It was installed, but I re-installed the wrapper and setup tools:

+ :
+ : Build current pychapel on Mac OS X 10.11 as of 2017-08-16 11:22, updated 15:50
+ :
+ : pychpl-2-full-pych.sh : Pychapel, full install. One-time-only for new user/env.
+ :
+ : This script should be copied to an otherwise-empty directory, and run interactively.
+ : Pychapel will be downloaded there and built.
+ :
+++ dirname int.sh
++ cd .
++ pwd
+ WORK=/Users/buddha/tmp
+ cd /Users/buddha/tmp
+ ls -la
total 8
drwxr-xr-x    4 buddha  staff   136 Apr  3 10:16 .
drwxr-xr-x+ 175 buddha  staff  5950 Apr  3 09:59 ..
drwxr-xr-x    3 buddha  staff   102 Apr  3 10:17 .virtualenvs
-rw-r--r--    1 buddha  staff  3399 Apr  3 09:59 int.sh
+ :
+ : 'Please "deactivate" any Python virtual env in the calling shell first.'
+ : 'If your interactive bash prompt starts with "(pychapel)", exit now; run "deactivate"; try again.'
+ :
+ : VIRTUAL_ENV= : it should be empty
+ test -z ''
+ :
+ : THIS WILL COMPLETELY REMOVE ANY EXISTING INSTALLATION.
+ :
+ : 'Ready?' Hit return to continue, or interrupt to start over.
+ read OK
OK
+ rm -rf /Users/buddha/tmp/.virtualenvs/pychapel
+ export WORKON_HOME=/Users/buddha/tmp/.virtualenvs
+ WORKON_HOME=/Users/buddha/tmp/.virtualenvs
+ export PATH=/usr/local/opt/python/libexec/bin:/usr/local/bin:/Users/buddha/.jenv/shims:/Users/buddha/.jenv/bin:/Users/buddha/github/chapel/bin/darwin:/Users/buddha/github/chapel/util:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/share/dotnet:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Library/TeX/texbin
+ PATH=/usr/local/opt/python/libexec/bin:/usr/local/bin:/Users/buddha/.jenv/shims:/Users/buddha/.jenv/bin:/Users/buddha/github/chapel/bin/darwin:/Users/buddha/github/chapel/util:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/share/dotnet:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Library/TeX/texbin
+ : source virtualenvwrapper.sh
+ set +ex
/usr/local/opt/python/bin/python3.6: Error while finding module specification for 'virtualenvwrapper.hook_loader' (ModuleNotFoundError: No module named 'virtualenvwrapper')
virtualenvwrapper.sh: There was a problem running the initialization hooks.

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/local/opt/python/libexec/bin/python and that PATH is
set properly.
+ :
+ : Initialize Python virtual env to be used w Pychapel
+ :
+ : mkvirtualenv -p /usr/local/bin/python2 pychapel
+ set +ex
Running virtualenv with interpreter /usr/local/bin/python2
New python executable in /Users/buddha/tmp/.virtualenvs/pychapel/bin/python2.7
Also creating executable in /Users/buddha/tmp/.virtualenvs/pychapel/bin/python
Installing setuptools, pip, wheel...done.
/usr/local/opt/python/bin/python3.6: Error while finding module specification for 'virtualenvwrapper.hook_loader' (ModuleNotFoundError: No module named 'virtualenvwrapper')
+ :
+ : VIRTUAL_ENV= : it should not be empty
+ test -n ''