rsync: command not found

[crbowman]

I get this error when trying to push my app to heroku.

-----> Fetching custom git buildpack... done
-----> Python app detected
-----> No runtime.txt provided; assuming python-2.7.4.
-----> Preparing Python runtime (python-2.7.4)
-----> Installing Distribute (0.6.36)
-----> Installing Pip (1.3.1)
-----> Noticed numpy/scipy. Bootstrapping prebuilt binaries.
-----> Creating/downloading binaries.
-----> Creating/downloading numpy bdist.
-----> Creating/downloading scipy bdist.
/tmp/buildpack_bgscjg2t0gd7/bin/steps/numpy-scipy: line 61: rsync: command not found

 !     Push rejected, failed to compile Python app

It looks like this is the offending piece of code:

# Move everything from venv directory to python directory
  if [ -d "venv" ]; then
    rsync -q -a venv/ python/
    rm -r venv
  fi

Can the rsync call be replaced with a mv call?

[dbrgn]

Hm, you're right, rsync is not available on Heroku... I wonder why I never got the error. Did you convert an existing app to this buildpack? Maybe it has to do with the transition from an older version of the Python buildpack.

In any case, can you replace it with cp -a, test it and post a pull request if it works? (And in case you're already using the hub Github client, you could even convert this issue to a pull request to avoid opening a separate one.)

[ghost]

dbrgn, I'm the gte647i poster on your website; I did not post the original issue here.

I forked your buildpack and replaced the 'rsync -q' with 'cp', and it eliminated the original error, but it still failed with the output below. I'm new to this so I'm not really sure what's going on. I did get numpy/scipy/pandas working with my app on Heroku awhile back. But I developed for a few months locally and then tried to push again and it failed. I searched again and found your buildpack, so I created a clean heroku app and a clean venv locally, reinstalled packages, copied over my app, and am now trying to push to heroku with your buildpack.

FYI, I'm on Windows, but as I said, I was able to get things working a few months ago.

--------------------------------------------------------------------------------------------------------------
       Downloading/unpacking scipy==0.12.0 (from -r requirements.txt (line 13))
         Running setup.py egg_info for package scipy
           Traceback (most recent call last):
             File "<string>", line 16, in <module>
             File "/tmp/pip-build-u52167/scipy/setup.py", line 165, in <module>
               setup_package()
             File "/tmp/pip-build-u52167/scipy/setup.py", line 145, in setup_pac
kage
               from numpy.distutils.core import setup
           ImportError: No module named numpy.distutils.core
           Complete output from command python setup.py egg_info:
           Traceback (most recent call last):

         File "<string>", line 16, in <module>

         File "/tmp/pip-build-u52167/scipy/setup.py", line 165, in <module>

           setup_package()

         File "/tmp/pip-build-u52167/scipy/setup.py", line 145, in setup_package

           from numpy.distutils.core import setup

       ImportError: No module named numpy.distutils.core

       ----------------------------------------
       Command python setup.py egg_info failed with error code 1 in /tmp/pip-bui
ld-u52167/scipy
       Storing complete log in /app/.pip/pip.log

 !     Push rejected, failed to compile Python app

[ghost]

Further confirmation scipy is the culprit. I removed scipy and statsmodels (has scipy dependency) from my requirements.txt file and was able to push to heroku. When trying to migrate the db, I first got a LIBLAPACK error, which I got past by setting the LD_LIBRARY_PATH config var per the scikit-learn SO post (even though I knew I didn't really have scipy installed correctly). Then the migrate fails with a "no module scipy" error.

To add to my confusion, the error in my previous post showed scipy 0.12.0 was being retrieved, which is what my requirements file contains. However, after adding scipy back into my requirements file, I got the error below which appears to be looking for 0.11.0 and can't even find that.

Is this a caching thing? Why is the buildpack looking for 0.11.0 instead of 0.12.0? Any help appreciated. Currently my app needs functions from scipy/statsmodels, so I've got to get them working somehow on heroku.

---

-----> Fetching custom git buildpack... done -----> Python app detected -----> No runtime.txt provided; assuming python-2.7.4. -----> Using Python runtime (python-2.7.4) -----> Noticed numpy/scipy. Bootstrapping prebuilt binaries. -----> Using cached binaries. -----> Using cached numpy. -----> Creating/downloading scipy bdist. tar: npscipy-binaries/scipy-0.11.0.tar.gz: Cannot open: No such file or directory tar: Error is not recoverable: exiting now

! Push rejected, failed to compile Python app

[dbrgn]

Oh, I think the problem has to do with the versions. The supported, precompiled versions are here: https://github.com/dbrgn/npscipy-binaries I forgot to mention that in the README. If you use other versions, then the buildpack tries to compile them on the fly, which doesn't work due to missing dependencies like Fortran.

Therefore you need to use Scipy 0.10.1 or 0.11.0.

Another issue was that simply replacing rsync with cp failed because it copied the first folder into the second one, instead of copying the contents. This is now fixed.

This works now:

$ mkdir testheroku
$ cd testheroku
$ git init
$ heroku create --buildpack https://github.com/dbrgn/heroku-buildpack-python-sklearn/
$ echo -e "numpy==1.7.0\nscipy==0.11.0\nscikit-learn==0.13.1" > requirements.txt
$ git add requirements.txt
$ git commit -m 'Added requirements'
$ git push heroku master
Counting objects: 3, done.
Writing objects: 100% (3/3), 260 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)

-----> Fetching custom git buildpack... done
-----> Python app detected
-----> No runtime.txt provided; assuming python-2.7.4.
-----> Preparing Python runtime (python-2.7.4)
-----> Installing Distribute (0.6.36)
-----> Installing Pip (1.3.1)
-----> Noticed numpy/scipy. Bootstrapping prebuilt binaries.
-----> Creating/downloading binaries.
-----> Creating/downloading numpy bdist.
-----> Creating/downloading scipy bdist.
-----> Moving everything from venv directory to python directory...
-----> Installing dependencies using Pip (1.3.1)
-----> Noticed scikit-learn. Installing...
-----> Setting environment vars...
-----> Installing scikit-learn==0.13.1 via pip
Downloading/unpacking scikit-learn==0.13.1
  Running setup.py egg_info for package scikit-learn

(...)

Successfully installed scikit-learn
Cleaning up...
       Cleaning up...

-----> Discovering process types
       Procfile declares types -> (none)

-----> Compiled slug size: 59.6MB
-----> Launching... done, v4
       http://ancient-eyrie-7305.herokuapp.com deployed to Heroku

To git@heroku.com:ancient-eyrie-7305.git
 * [new branch]      master -> master

[ghost]

This now works for me. Thanks so much.

[ghost]

Can this same approach be used for statsmodels? I can now get everything working if I totally remove statsmodels. I think it's choking on the same issue as it requires scipy and doesn't seem to notice that scipy is in fact installed, even when I have statsmodels as last item in my requirements.txt file.

[dbrgn]

You're welcome.

Hm, I have absolutely no experience with statsmodels, but after fixing issue #2 other libraries should be aware that SciPy is installed. Otherwise installing Scikit-Learn via PIP would probably not work.

Does the deploy step print any useful debug output?

If it doesn't work for you and if you think it has to do with this buildpack, feel free to open a separate issue.

[ghost]

Just tried it again and it worked! I seem to recall having similar success several months ago where I could install numpy etc. initially (although with much more difficulty than with your buildpack) and then add the additional packages that depend on those in followup commits/pushes.

Bottom line, I'm not quite sure why the sequential steps are required, but I think your buildpack is working. Thanks so much for it - sure makes it easier to get these packages installed reliably!