pip install pyChapel fails

buddha314 commented 6 years ago

Decided to try pyChapel again. Here is the tail of the error log.


    byte-compiling /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pych/extern.py to extern.pyc
    byte-compiling /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pych/object_store.py to object_store.pyc
    byte-compiling /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pych/runtime.py to runtime.pyc
    byte-compiling /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pych/specializer.py to specializer.pyc
    byte-compiling /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pych/types.py to types.pyc
    byte-compiling /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pych/utils.py to utils.pyc
    byte-compiling /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pych/version.py to version.pyc
    running install_data
    creating /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/share
    creating /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/share/pych
    creating /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/share/pych/config
    error: can't copy 'configs/pych.json': doesn't exist or not a regular file

    ----------------------------------------
Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/g5/4zwyfk513hd94tjs1fm5mdq00000gn/T/pip-build-fhKgtD/pyChapel/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/g5/4zwyfk513hd94tjs1fm5mdq00000gn/T/pip-1YpDnm-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/g5/4zwyfk513hd94tjs1fm5mdq00000gn/T/pip-build-fhKgtD/pyChapel/
Cache entry deserialization failed, entry ignored
macbuddha:~ buddha$

awallace-cray commented 6 years ago

I get the same. (I don't think anything has changed since issue #75)

buddha314 commented 6 years ago

Rightio, the error looked different so I created a new issue. Think it's the same thing?

awallace-cray commented 6 years ago

Sadly, yes.

awallace-cray commented 6 years ago

It looks like the last two versions released through pypi (pip) can not be successfully installed from pypi (ie, pip install pyChapel ), like it says in http://pychapel.readthedocs.io/introduction.html. Not on any kind of system. Because the last two versions on pypi do not include file configs/pych.json.

You could install pip install pychapel==0.1.15 though.

awallace-cray commented 6 years ago

Maybe we should make this issue about fixing the broken pypi package, only.

buddha314 commented 6 years ago

I will try install 0.1.15. As to the ticket, do what you think is best. You have the look of a confident professional.

awallace-cray commented 6 years ago

Yes, "Bob" had good hair. As for pip install pychapel==0.1.15, that definitely won't work on Mac OS. Even on plain-vanil Ubuntu 64, you would still have to build and copy in the Chapel libraries by hand. IMO, you would be better to stick with those "Rudimentary Pychapel build scripts" (either Ubuntu or Mac OS) for now.

buddha314 commented 6 years ago

You should have told me system that. How can I check?

pip install pychapel==0.1.15
Collecting pychapel==0.1.15
  Downloading pyChapel-0.1.15.tar.gz
Building wheels for collected packages: pychapel
  Running setup.py bdist_wheel for pychapel ... done
  Stored in directory: /Users/buddha/Library/Caches/pip/wheels/f9/e3/01/625cb58978db659fe7298bfdfd00d4bb2b6493ef2696926d13
Successfully built pychapel
Installing collected packages: pychapel
Successfully installed pychapel-0.1.15

awallace-cray commented 6 years ago

Yeah, I just saw that too. OK, try pych --version

buddha314 commented 6 years ago

Ya got me!

buddha$ pych --version
-bash: pych: command not found

awallace-cray commented 6 years ago

I got "pych" in PATH at least (used a virtualenv, but that should not matter). I'll fiddle with this a bit more, I need to figure it out sometime.

(pip-pychapel) # 2018/01/24-14:00:55 awallace@mbp-awallace:~/work2
type pych
pych is hashed (/Users/awallace/work0/.virtualenvs/pip-pychapel/bin/pych)
(pip-pychapel) # 2018/01/24-14:02:14 awallace@mbp-awallace:~/work2
file /Users/awallace/work0/.virtualenvs/pip-pychapel/bin/pych
/Users/awallace/work0/.virtualenvs/pip-pychapel/bin/pych: Python script text executable, ASCII text

awallace-cray commented 6 years ago

You dont need to wait for pip install pychapel to work, however. Also, anything I showed above (in the (pip-pychapel) venv) may be wrong!

awallace-cray commented 6 years ago

The only way pip install pychapel can ever work, is by ALSO building a local (or maybe a shared) copy of the Chapel release tarball or the Chapel Git source repo. Those shared lib files that the "Getting Started" instructions ask you to copy, those are just the beginning. I bet that lib-copying stuff was only put in there because it was too hard for pychapel to figure out that long subdirectory path underneath CHPL_HOME/lib. The other required chapel stuff- chpl compiler binary, various headers- would be easy to generate on-the-fly within pychapel, from the Chapel environment setup we see in the "Getting Started" instructions.

To properly support a "pip install pychapel", we would probably need a "pip install chapel" as a dependency. I guess that could be done. Especially since now chapel supports the concept of "make install".

buddha314 commented 6 years ago

So we need chapel installed independently of pychapel. That makes sense. I think that libs like psycogp2 require postgres to be installed, so maybe there is a pattern we could duplicate. Like checking for chpl and halting if it's not there.

awallace-cray commented 6 years ago

No enthusiasm for pip install chapel idea. I will stick to making the existing pip install pychapel work as-is.

buddha314 commented 6 years ago

Would it be more of a "brew" thing? I've used brew to install fortran compilers. I honestly don't care, but there has to be some kind of precedent, right?

awallace-cray commented 6 years ago

Yes, brew would be fine, and we already have a chapel formula. I don't think the Chapel options are the same as we use for Pychapel, but we could fix that. Then, the trick would be to get the Pychapel installer to know about Mac OS X and the Chapel homebrew installation. Could be some recent progress in issue #89. (Recent progress for me, at least- I'll get there, eventually)

awallace-cray commented 6 years ago

Only drawback w Homebrew is, you go to the trouble to create that package, and it only supports Mac OS X. Seems like we could support just about anything with one pip or conda package.

buddha314 commented 6 years ago

Yeah, that's a good point. But almost the same can be said for apt-get.

awallace-cray commented 6 years ago

Not to forget- the Rudimentary Pychapel build scripts for Mac OS X and Ubuntu 16.04 are working!

russel commented 6 years ago

@awallace-cray There is Linuxbrew which replaces the Homebrew brew scripts to be Linux specific rather than macOS specific, and then uses the same formula to install things on Linux. I am using this for node and a few other things. The only problem is when the formulae depend on macOS dependencies. For formulae that only depend on assumed installations or other formulae everything works fine.

awallace-cray commented 6 years ago

Linuxbrew - cool! Thanks @russel !

awallace-cray commented 6 years ago

I had hoped to fix up a few missing file(s) in the pypi archive, and pip install pychapel would at least work like it says in the installation instructions - you would still need to build Chapel, etc, but at least pip install would complete.

It's starting to look like that would be more trouble than its worth. So far, I don't see any way to just add a few files, fix up the instructions a little, and end up with a pip install pychapel that works well. Notes so far:

The pypi release (tar) would need to include a lot more of the pychapel.git than it does now.
User would need to pip-install a lot more of the requirements.txt, before using pychapel; or else pip-pychapel needs to require it. Either way, the Users would really want a Python virtual env for this, because without that,
- you need to be root to do the installation (and maybe even when you use pychapel- not sure),
- and you wind up with a bunch of python stuff in your /usr/local/lib/python2*/dist-packages that could easily conflict with something else you want to do on the same machine.

If pychapel's pypi release has to include most of the pychapel.git, and if we have to talk about using Python virtual env's in the pychapel installation instructions, and the User still has to build Chapel anyway, then I feel like we might as well:

take down the pypi release
pretty-up the "rudimentary build scripts" a little and use those as the official installation instructions, at least until...
we have Homebrew / linuxbrew packages

buddha314 commented 6 years ago

At this point, it seems the best thing to do is wait for the Homebrew package. What do you think?

awallace-cray commented 6 years ago

Which then begs the question, should we do a Homebrew package that (one way or other) uses the temporary "patch" file fix for the inherently non-portable pych.json (like the "rudimentary" scripts)? Or try to apply the Chapel "compileline" idea (issue #89 ) to make pychapel a lot more portable? At that point, linuxbrew starts making sense.

awallace-cray commented 6 years ago

Maybe the answer is, do both. Patch file idea -> Homebrew, first. Then the compileline idea -> linuxbrew, later.

buddha314 commented 6 years ago

I don't feel entirely qualified to chime in. What do you think is best? Homebrew is pretty great for installing packages, and I've used for things like gcc and gfortran, so it seems appropriate to use for both chapel and pychapel. I think the chapel might be out of date at the moment.

awallace-cray commented 6 years ago

Homebrew chapel out of date

Existing Homebrew chapel formula is the latest Chapel release, Oct 2017. It can seem out of date, when you are in dev/explore mode.

We could support brew install chapel --build-from-source --some-other-stuff (for example), or brew install pychapel --build-from-source.

awallace-cray commented 6 years ago

PR #90 tests successfully.

awallace-cray commented 6 years ago

2018-02-23 21:09 PDT : PR #90 merged. PyPI pychapel updated from master branch, as version 0.1.18 https://pypi.python.org/pypi?name=pyChapel&version=0.1.18&:action=display Tested "pip install" from default PyPI URL. Test OK. Version confirmed. However, the online documentation at http://pychapel.readthedocs.io did not update!

2018-02-26 21:17 PDT : PRs #92 and #93 merged, to make pychapel.readthedocs.io's auto-build work again, so that the online documentation http://pychapel.readthedocs.io would get updated. The newly-updated installations instructions in this PR are now visible.

awallace-cray commented 6 years ago

2018-02-26 21:40 PDT : This issue can be closed, if someone else besides me confirms that the online docs at http://pychapel.readthedocs.io really have been updated.
(A quick way to tell: http://pychapel.readthedocs.io/en/latest/pych/notes.html, see correct English spelling of "Miscellaneous") I have had considerable trouble with this documentation update, likely caused by forgetting to clear my browser cache at important test points.

awallace-cray commented 6 years ago

Note: in addition to PRs #92 and #93, the secret sauce included two configuration changes available only to admins of pychapel.readthedocs.io:

Go to Edit Advanced Project settings https://readthedocs.org/dashboard/pychapel/advanced/
DE-select (un-check the checkbox): Install Project: [ ] Install your project inside a virtualenv using setup.py install
In Requirements file text box: enter docs/source/requirements.txt
Scroll to bottom. Hit Submit button

awallace-cray commented 6 years ago

Lydia confirmed http://pychapel.readthedocs.io. Closing as Done.

awallace-cray commented 6 years ago

2018-02-27: This manual build process still works, but it is no longer needed.

https://gist.github.com/awallace-cray/20c2aaf7d5739b8302d45ca12c37a527

With PyChapel release 0.1.18 on GitHub and PyPI (February 2018), PyChapel's Pip installer works well on normal 64-bit GNU-Linux distros, and we recommend it for new users.

Please see the Installation Instructions, here

chapel-lang / pychapel

pip install pyChapel fails #88