pipenv links local package to wrong path

[dz0ny]

If we have a Pipenv and Pienv.lock file that define local package:

myapp  = {editable = true, path = "."}

When buildpack builds such project it will create "myapp.egg-link" with contents that links to build dir of that package, which is no longer valid at the moment when app runs.

After build myapp.egg-link will be:

/tmp/build/$randomhash$

the correct is

/app

I'am not sure how to fix such behavior in buildpack. One idea would be to replace relative path off path = "." with absolute path = "/app", but that creates more problems if you have multiple local packages.

[kennethreitz]

We have eggpath fixes in the buildpack — they may not be getting applied for pipenv. Will have to investigate.

[dz0ny]

@kennethreitz I think this is also needed https://github.com/heroku/heroku-buildpack-python/compare/master...dz0ny:patch-1

[domenkozar]

I've dug a bit into this, here's the larger picture:

• if you'd like to use zip or git dependency with pipenv, it has to be editable for pipenv to resolve dependencies
• buildpack uses --deploy and thus skips editable=True dependencies

[domenkozar]

Following line demonstrates what happens if dependency comes from an editable egg:

cat /app/.heroku/python/lib/python3.6/site-packages/pyramid-openapi3.egg-link
/tmp/build_d60eb12f2b2ba06869d3b6c43525a4fc/src/pyramid-openapi3

I'd recommend following (somewhat general) workaround to be applied after buildpack-python:

for f in src; do
    echo "/app/src" > /app/.heroku/python/lib/python3.6/site-packages/$f.egg-link
done

[billmccord]

@kennethreitz Is there any update on this or is the workaround by domenkozar still the only way to resolve this problem? I just deployed using pipenv and it appears this is still an issue.

[zupo]

@billmccord FWIW, yes, we're still using it to deploy the project @domenkozar worked on with pipenv.

[billmccord]

@zupo or @domenkozar I deployed to Heroku with pipenv and it seems that even without the workaround the editable packages are found and loaded properly. Can you help me understand the purpose of the workaround?

[zupo]

@billmccord: change some python code, leave Pipfile(.lock) intact, commit and push. The build should pass, but then your app won't start as the editable package won't be found in PYTHONPATH.

[billmccord]

@zupo I've tried this workflow:

1. Pushed my code and deployed it to Heroku.
2. Tested and verified that app works.
3. Used heroku ps:exec to SSH to machine and verified that the editable packages were installed into /app/src.
4. Modified some arbitrary Python file in my applications (did NOT modify Pipfile or Pipfile.lock file per your instruction.) To be specific, I changed a setting in a settings.py file.
5. Committed and pushed changes and deployed Heroku.
6. Tested and verified that app works.
7. Used heroku ps:exec to SSH to machine and verified that the editable packages were not removed and also the timestamp on the folders were not changed.

Is there some other scenario I'm missing where these editable packages get removed? The only thing that looks suspicious / dangerous is the file /app/src/pip-delete-this-directory.txt which contains the contents:

This file is placed here by pip to indicate the source was put
here by pip.

Once this package is successfully installed this source code will be
deleted (unless you remove this file).

However, it isn't clear to me under what circumstances the source code is deleted.

Thanks for your guidance.

[zupo]

I'll prepare a minimal example that exhibits the bug.

[billmccord]

@zupo Sorry to bother you, but were you able to reproduce the bug? I still haven't seen it manifest itself.

[zupo]

Sorry, not yet. Moving countries this week. Will definitely do it when things settle down a bit.

[billmccord]

@zupo Thanks for your effort, it is greatly appreciated!

[billmccord]

@zupo @dz0ny I went ahead and created a small Flask example repo. Basically, this uses pipenv and editable packages, but it doesn't specify the path. I don't seem to have any issues deploying or redeploying this project to Heroku. This is the way we use editable packages in our primary project, so I think this is only an issue if you specify a path, is that your understanding?

I just want to be confident that I'm not going to run into problems later if we are using editable packages like this without specifying a path. Thanks a lot!

[zupo]

@billmccord glancing through your example app the culprit might be that we are using setup.py in our apps. Preparing example app now.

[dz0ny]

@zupo and we are using a subdirectory for package source.

[zupo]

@billmccord: example with instructions on how to exercise the bug: https://github.com/zupo/heroku-pipenv-link-bug

And thank you for being so eager and pushing me to finally take time to do it.

[zupo]

If the pipenv version on Heroku was a more recent one, we possibly wouldn't have to do the pipenvbug -> e1839a8 replacement in Pipfile and that would possibly make the bug go away: https://github.com/heroku/heroku-buildpack-python/blob/master/bin/steps/pipenv#L55

[billmccord]

@zupo Thanks so much for creating the example. I think it is safe to say that our deployment process is using a very different approach by using Pipfile only and not installing from a setup.py. Also, we are not specifying a path for any of our editable packages. Perhaps the two combined have allowed us to avoid this.

I really appreciate you taking the time as it has given me the confidence to move forward with our deployment and I hope that our small sample projects can provide ways to reproduce for whomever will work on this issue as well as workarounds for others.

[zupo]

Any rough estimate on when could we possibly expect the heroku-buildpack-python pipenv version to be bumped to a more recent one?

[CaseyFaist]

Hello! Thanks for your patience on this one.

On when we can expect a pipenv bump, since the current version would require users to uninstall and reinstall their local pipenv environments before being able to deploy again successfully, we're not currently looking at doing one soon. This would require coordination, and in the best case would cause a bit of frustration if any users were unprepared.

I'll take a look at your reproduce cases though.

[zupo]

Hey @CaseyFaist, thanks for checking in! Actually, you can close this issue due to pipenv not seeing any new release in over 1.5 years and seems to be more or less a dead project.

We've use poetry these days and deploy it inside a docker image, configured by shell.nix file and I must say I am very happy with how this works. Here's a workable example, in case anyone is interested: https://github.com/niteoweb/pyramid-realworld-example-app/blob/master/Dockerfile

[CaseyFaist]

Nice! Poetry support is on the list, hopefully not too far off. Glad to hear you're happy with it! Closing issue now.