conda-buildpack and Django apps

[evdoks]

We had some trouble integrating conda-buildpack with our Django app running on Heroku. The problem is that our app requires pylibmc package which cannot be installed due to missing libmemcached library. Using conda-buildpack together with of heroku-buildpack-python (I believe, you are maintaining it), obviously did not help, since conda-buildpack repeats pip install -r requirements.txt and still fails when installing pylibmc. We solved the problem by creating a fork of conda-buildpack where we

• remove pip install -r requirements.txt step
• set PYTHONPATH to '/app/.heroku/miniconda/lib/python2.7/site-packages:$PYTHONPATH'

Wouldn't it make more sense to use conda-buildpack merely for installing Conda packages and leaving the Python part to heroku-buildpack-python?

[joshowen]

+1

[kennethreitz]

I like this suggestion — this would make the conda buildpack much more useful, in my opinion.

Thanks for it! I will investigate this soon and will keep this ticket updated as this idea evolves.

Additional comments welcome!

[t-io]

+1 Would love to see this fixed in this repo

[icoxfog417]

I'm using this buildpack on Python3. So I am a little worried about the solution that suggested above (fixed path to python2.7). When releasing this feature, please take care of the behavior on Python3.

[jvenezia]

Hi, I'm trying to achieve this using python3.4 with Django1.8

I forked (https://github.com/jvenezia/conda-buildpack) to be able to use runtime.txt to choose my python version (thanks to @metakermit).

I tried with python3.4 conda site-package by setting setting PYTHONPATH to '/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages:$PYTHONPATH' and get this error :

~ $ python manage.py migrate
Traceback (most recent call last):
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 20, in <module>
    import psycopg2 as Database
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/psycopg2/__init__.py", line 50, in <module>
    from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: /app/.heroku/miniconda/envs/condaenv/lib/libssl.so.1.0.0: undefined symbol: X509_get_signature_nid

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line
    utility.execute()
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/core/management/__init__.py", line 325, in execute
    django.setup()
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/apps/config.py", line 198, in import_models
    self.models_module = import_module(models_module_name)
  File "/app/.heroku/python/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/contrib/auth/models.py", line 41, in <module>
    class Permission(models.Model):
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/db/models/base.py", line 139, in __new__
    new_class.add_to_class('_meta', Options(meta, **kwargs))
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/db/models/base.py", line 324, in add_to_class
    value.contribute_to_class(cls, name)
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/db/models/options.py", line 250, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/db/__init__.py", line 36, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/db/utils.py", line 240, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/app/.heroku/python/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 24, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: /app/.heroku/miniconda/envs/condaenv/lib/libssl.so.1.0.0: undefined symbol: X509_get_signature_nid

I'm new to python environment and I'm getting stuck on this. Any Idea? Thanks

[jvenezia]

I finally go this working with PYTHONPATH '/app/.heroku/python/lib/python3.4/site-packages:/app/.heroku/miniconda/envs/condaenv/lib/python3.4/site-packages:$PYTHONPATH'

I'm not sure this is really a clean way to do that...

Now I have static assets errors when building with the official heroku-buildpack-python

[jvenezia]

I got it fully work with buildpack-multi using https://github.com/jvenezia/conda-buildpack and https://github.com/jvenezia/heroku-buildpack-python

But I find this solution I bit hacky.

I think we could find a way to do this without the need to change official's heroku-buildpack-python, and make it work without having to hardcode PYTHONPATH like this.

Please tell me I you have any suggestion to clean this up.

[ghost]

Thanks to all of you for your work. I was able to build on it to get Python 3.5 running. For any interested, I forked jvenezia's conda-buildpack at https://github.com/buildingspeak/conda-buildpack.git.

Couple of points. I set to Python 3.5 and this works in the condaenv but for some reason the main miniconda lib still seems to use 3.4. I'm guessing this is because "Miniconda3-latest-..." that I'm using is still on 3.4.

Because the slug size > 300 MB, I had to use a "rm -rf ~/.heroku/miniconda/pkgs" line in the conda_compile file to kill some of the miniconda stuff taking up lots of space, since they should generally no longer be needed, as I'm running from the condaenv. This reduced slug size considerably to about 160MB, so I will probably add some additional removals to further reduce the size. I'm using numpy, scipy, pandas, and statsmodels from conda and about 40 other packages (app is Django) in my regular requirements.txt.

Also, I reversed jvenezia's change, reinstating the use of pip and requirements.txt.

Right now it all seems to have worked, and my current issue I believe is related to my need to upgrade my code from Django 1.5 to 1.8, having already used 2to3 to get from Python 2.7 to 3.5. So I'm not totally successfully deployed yet, but I think I'm over the buildpack hurdle and wanted to share.

[dtran320]

@kennethreitz Thanks for all your great work across many packages we use :) I've been struggling with this issue since Friday and feel like I've tried everything to get scipy/scikit-learn deployed with Django. I've tried just the conda-buildpack, conda-buildpack + heroku-buildpack-python, some of the scipy-numpy buildpacks that distribute binaries floating out there, and even tried setting up a docker image for use with https://github.com/heroku/heroku-docker.

How would you recommend deploying numpy/scipy/scikit-learn with Django to Heroku without forking https://github.com/heroku/heroku-buildpack-python? Thanks!