.heroku/vendor disappears when clearing cache

[browniebroke]

This issue is probably related to #29, but the other way round. The first deploy with purged cache breaks my application (although it completes fine), and pushing an empty deploy immediately after fixes it. I'm using this buildpack with the default Heroku Python buildpack.

.buildpacks file:

https://github.com/cyberdelia/heroku-geo-buildpack.git#1.3
https://github.com/heroku/heroku-buildpack-python.git#v62

Commands to reproduce

> heroku repo:purge_cache
> git commit --allow-empty -m "Empty deploy"
> git push
Counting objects: 1, done.
Writing objects: 100% (1/1), 189 bytes | 0 bytes/s, done.
Total 1 (delta 0), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Deleting 0 files matching .slugignore patterns.
remote: -----> Fetching custom git buildpack... done
remote: -----> Multipack app detected
remote: =====> Downloading Buildpack: https://github.com/cyberdelia/heroku-geo-buildpack.git
remote: =====> Detected Framework: geos/gdal/proj
remote:        Using geos version: 3.4.2
remote:        Using gdal version: 1.11.1
remote:        Using proj version: 4.8.0_1
remote: -----> Fetching and vendoring geos
remote: -----> Fetching and vendoring gdal
remote: -----> Fetching and vendoring proj
remote: -----> Vendoring geo libraries done
remote: =====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-python.git
remote: =====> Detected Framework: Python
remote: -----> Running pre-compile hook
remote: -----> Removing old data
remote: -----> Installing runtime (python-2.7.10)
remote: -----> Installing dependencies with pip
remote:        Collecting boto==2.36.0 (from -r requirements.txt (line 1))
remote:          Downloading boto-2.36.0-py2.py3-none-any.whl (1.3MB)
remote:        Collecting celery>=3.1.17 (from -r requirements.txt (line 2))
remote:          Downloading celery-3.1.18-py2.py3-none-any.whl (515kB)
[...]
remote:          Running setup.py install for icalendar
remote:          Running setup.py install for django-ical
remote:          Running setup.py install for duedil
remote:          Running setup.py install for python-linkedin
remote:          Running setup.py install for Collectfast
remote:          Running setup.py install for bitly-api
remote:          Running setup.py install for opbeat
remote:        Successfully installed Collectfast-0.2.3 amqp-1.4.6 anyjson-0.3.3 billiard-3.3.0.20 bitly-api-0.3 boto-2.36.0 celery-3.1.18 dendrite-1.1.3 diff-match-patch-20121119 dj-database-url-0.3.0 django-1.6.11 django-appconf-1.0.1 django-autoslug-1.7.2 django-celery-3.1.16 django-celery-with-redis-3.0 django-compressor-1.3 django-crispy-forms-1.4.0 django-fsm-2.2.0 django-fsm-admin-1.2.1 django-fsm-log-1.2.1 django-ical-1.3 django-import-export-0.2.6 django-jsonfield-0.9.2 django-mixins-0.0.10 django-registration-1.0 django-robots-1.1 django-shortuuidfield-0.1.2 django-storages-1.1.8 djangorestframework-2.4.4 docutils-0.12 duedil-0.1.dev0 ecdsa-0.13 fabric-1.10.1 geopy-1.9.1 gunicorn-19.2.0 honcho-0.6.6 icalendar-3.9.0 kombu-3.0.26 markdown-2.6.2 mock-1.0.1 newrelic oauthlib-1.0.3 opbeat-3.0.4 paramiko-1.15.2 pillow-2.9.0 pipetools-0.2.7 pretend-1.0.8 psycopg2-2.6.1 pycrypto-2.6.1 pygal-2.0.1 pygeocoder-1.2.5 python-dateutil-2.4.2 python-linkedin-4.1 pytz-2015.4 redis-2.10.3 requests-2.5.1 requests-oauth2-0.2.0 requests-oauthlib-0.3.3 sh-1.11 shortuuid-0.4.2 six-1.9.0 south-1.0.2 tablib-0.10.0 yell-0.3.2
remote: 
remote: -----> Preparing static assets
remote:        Collectstatic configuration error. To debug, run:
remote:        $ heroku run python manage.py collectstatic --noinput
remote: 
remote: -----> Running post-compile hook
remote: Using release configuration from last framework (Python).
remote: -----> Discovering process types
remote:        Procfile declares types -> celery, duedil, web
remote: 
remote: -----> Compressing... done, 67.5MB
remote: -----> Launching... done, v1131
remote:        https://myapp.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy.... done.
To git@heroku.com:myapp.git
   530dd4d..f0c69ac  staging -> master

At this stage, my application is responding with Internal Server error and in the logs the exception raised is ImproperlyConfigured: Could not import user-defined GEOMETRY_BACKEND "geos" which means Django cannot find the geo libraries. I've noticed that the .heroku/vendor folder is missing on my application:

~ $ ls -la .heroku/
total 20
drwx------  3 u54328 54328 4096 Aug 18 09:23 .
drwx------ 12 u54328 54328 4096 Aug 18 09:28 ..
drwx------  7 u54328 54328 4096 Aug 18 09:25 python
-rw-------  1 u54328 54328    9 Aug 18 09:23 python-stack
-rw-------  1 u54328 54328   14 Aug 18 09:23 python-version

To get my application back up, I only need to run:

> git commit --allow-empty -m "Empty deploy"
> git push
Counting objects: 1, done.
Writing objects: 100% (1/1), 188 bytes | 0 bytes/s, done.
Total 1 (delta 0), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Deleting 0 files matching .slugignore patterns.
remote: -----> Fetching custom git buildpack... done
remote: -----> Multipack app detected
remote: =====> Downloading Buildpack: https://github.com/cyberdelia/heroku-geo-buildpack.git
remote: =====> Detected Framework: geos/gdal/proj
remote:        Using geos version: 3.4.2
remote:        Using gdal version: 1.11.1
remote:        Using proj version: 4.8.0_1
remote: -----> Vendoring geo libraries done
remote: =====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-python.git
remote: =====> Detected Framework: Python
remote: -----> Running pre-compile hook
remote: -----> Removing old data
remote: -----> Installing dependencies with pip
remote:
remote: -----> Preparing static assets
remote:        Running collectstatic...
remote:        Collected static files in 0:00:03.603913.
remote:        Skipped 603 already synced files.
remote:        0 static files copied.
remote:
remote: -----> Running post-compile hook
remote: Using release configuration from last framework (Python).
remote: -----> Discovering process types
remote:        Procfile declares types -> celery, duedil, web
remote:
remote: -----> Compressing... done, 141.7MB
remote: -----> Launching... done, v1132
remote:        https://myapp.herokuapp.com/ deployed to Heroku
remote:
remote: Verifying deploy.... done.
To git@heroku.com:myapp.git
   f0c69ac..c4b505c  staging -> master 

At this stage .heroku/vendor is back:

~ $ ls -la .heroku/
total 24
drwx------  4 u53523 53523 4096 Aug 18 09:29 .
drwx------ 12 u53523 53523 4096 Aug 18 09:36 ..
drwx------  7 u53523 53523 4096 Aug 18 09:29 python
-rw-------  1 u53523 53523    9 Aug 18 09:29 python-stack
-rw-------  1 u53523 53523   14 Aug 18 09:29 python-version
drwx------  6 u53523 53523 4096 Aug 18 09:29 vendor

I think it started happening when I upgraded to Cedar-14 from Cedar. This is a problem because yesterday I made a normal deploy and for some reason Heroku decided to clear the cache, breaking my application.

Not sure if due to the same root cause as #29 so opening a separate issue.

[browniebroke]

I "resolved" this issue by swapping the order of the buildpacks in the .buildpacks file:

https://github.com/heroku/heroku-buildpack-python.git#v62
https://github.com/cyberdelia/heroku-geo-buildpack.git#1.3

However, this causes other issues, at least collectstatic fails:

remote: -----> Preparing static assets
remote:        Collectstatic configuration error. To debug, run:
remote:        $ heroku run python manage.py collectstatic --noinput

Collectstatic is now run before geo-buildpack is installed.

Looking at the buildpack-multi source, it treats the last declared buildpack differently, so my workaround is probably invalid...

[shangxiao]

I am having trouble getting geodjango running too, the .heroku/vendor folder seems to be missing.

[shangxiao]

After a bit of investigation, for some reason when using this buildpack alone the vendor folder is created, but when the python buildpack is used afterwards it isn't.

For now as a temp workaround I've just forked the python buildpack and prepended this buildpack's compile script on the front and exported the appropriate env vars.

I'd investigate further but now the free postgres tier doesn't include support for postgis =/

[gareth-lloyd]

I am having the same problem. Will do some investigation of my own.

[gareth-lloyd]

My crude solution:

.buildpacks:

https://github.com/heroku/heroku-buildpack-python.git#v61
https://github.com/cyberdelia/heroku-geo-buildpack.git#1.3
https://github.com/heroku/heroku-buildpack-python.git#v61

Collectstatic fails the first time after a purge_cache, but succeeds the second time it tries.

[evenicoulddoit]

And if you have any bin/post_compile steps, then make sure to ignore the warnings generated in the first pass.

[longhotsummer]

I have been able to reproduce this on dokku. It looks like the first time a clean build is done, the contents of the $BUILD_DIR don't make its way into the actual image. Once a build has succeeded and the build dir has the data cached, the subsequent builds DO get the data into $APP_VENDOR, and so it succeeds.

So a workaround when deploying for the first time (or with a cleaned build cache), is to do a build but not fail any checks for the GEOS libraries. Then, deploy again and it'll all be okay.

[browniebroke]

Heroku seem to have done some work to improve compatibility with multi build packs, but upgrading doesn't help with this issue.

However, I've applied @gareth-lloyd workaround, and it works for me.

[browniebroke]

For what it's worth, I've tried today to migrate to the official multi buildpacks from Heroku and the problem persists:

$ heroku buildpacks
=== Buildpack URLs
1. heroku/nodejs
2. https://github.com/cyberdelia/heroku-geo-buildpack.git#1.3
3. heroku/python

I'm also using the nodejs buildpack to run bower, and I noticed that the .heroku/node doesn't vanish like .heroku/vendor does:

~ $ ls -la .heroku/
total 24
drwx------  4 u29867 dyno 4096 Jan  5 11:29 .
drwx------ 13 u29867 dyno 4096 Jan  5 11:31 ..
drwx------  7 u29867 dyno 4096 Jan  5 11:28 node
drwx------  8 u29867 dyno 4096 Jan  5 11:30 python
-rw-------  1 u29867 dyno    9 Jan  5 11:29 python-stack
-rw-------  1 u29867 dyno   14 Jan  5 11:29 python-version

[kennethreitz]

I believe this line is to blame.

https://github.com/heroku/heroku-buildpack-python/blob/master/bin/steps/python#L18

[browniebroke]

@kennethreitz: does it mean we can expect a change/fix in the heroku python buildpack? I appreciate changing the line you're pointing at might impact all third parties buildpack like this one, meaning clearing the .heroku/vendor is not the python's buildpack responsability? @gareth-lloyd workaround is not applicable with Heroku multi-buildpack.

[kennethreitz]

@browniebroke I am looking into the possibility of a fix this week.

The true problem here lies in the fact that the Python buildpack thinks the first push of an application is a stack change (I think). .heroku/vendor is actually a pattern that was created by the Python buildpack, and that it fully utilizes internally for all extra-supported C libraries.

[browniebroke]

Closing this issue as it's not happening anymore. Haven't seen it in a while and I just tried to cause it again on our testing environment, the problem is gone.

[kennethreitz]

Good news! I put some work into fixing it — glad to hear it is indeed fixed :)