Clarify/revise lock file generation/updating in the "Pipfile in pip" road map

[ncoghlan]

(@dstufft recently mentioned wanting to spend time on native Pipfile support in pip after Warehouse launches and legacy PyPI is shut down, and that reminded me that I had some concerns regarding the proposed inclusion of lock file updating capabilities in https://github.com/pypa/pipfile#pip-integration-eventual)

When managing the relationship between Pipfile and Pipfile.lock, there are two quite distinct ways of handling it:

1. Make the changes to the current environment, then generate a lock file based on that environment (the pip freeze model)
2. Build a requirements set from Pipfile, then calculate Pipfile.lock based on that requirements set (the pip-compile model)

pipenv uses the latter approach, and the fact that it ignores any extra packages installed directly with pip is considered a feature, not a bug.

If we move the pipenv approach down to the pip layer, though, then things get a lot more complicated, as we have to ask whether or not we're going to make Pipfile mandatory in order to use pip at all, and I don't think it makes sense for us to do that, for the same reason that we ended up keeping the "application dependency management with pipenv" tutorial separate from the core "package installation with pip" tutorial: Pipfile doesn't really make sense as a model for managing user site-packages directories, and similar ad hoc personal environments.

So at the pip layer, I think the "lock the current environment" model makes more sense than the pipenv-style compilation model.

However, I also think that that would be a confusing model to use implicitly, so I'm thinking that Pipfile support in pip should look somewhat different from the way it looks at the pipenv layer:

1. Add pip install -p dirname and pip install --project dirname to handle directories containing Pipfile and Pipfile.lock, such that pip install -p . has the same semantics as pipenv install --deploy (no implicit update, but refuses to install if Pipfile.lock is out of date with respect to Pipfile). Note the suggested use of --project as the long form of the option name (again reflecting the ad hoc installation vs project dependency management split that we introduced in the tutorials).
2. Allow pip install -p fname to accept direct references to Pipfile formatted files as input (doesn't even check for Pipfile.lock in that case, so pip install -p Pipfile is akin to pipenv install --skip-lock)
3. Allow pip install -p fname to accept direct references to Pipfile.lock formatted files as input (doesn't even check for Pipfile in that case, pip install -p Pipfile.lock is akin to pipenv install --ignore-pipfile)
4. Allow pip install -r fname to accept Pipfile.lock formatted files as input in addition to the existing requirements.txt format support (functionally equivalent to pip install -p fname, so this could be instead of the previous option, rather than in addition to it). The purpose of this would be to ease integration with other tools that pass along requirements files to pip install -r commands, but otherwise don't care about their internal format.
5. Update pip freeze with a -p/--project option that emitted a Pipfile/Pipfile.lock pair for the currently active virtual environment in the current working directory (where all the Pipfile dependencies would be "*" deps, while Pipfile.lock would have the exact versions)

[januszm]

I can't wait for the support for Pipfile / Pipfile.lock to be implemented in pip itself. I may stick to pyenv, instead of pipenv in the near future, due to the simplicity and comfort the pyenv provides (mainly using the .python-version file). In addition, it seems that the issues of managing the versions of Python and managing packages dependencies are two separate topics, and Pipfile.lock lies more in the sphere of pip than Virtual Environments (I come from the Ruby world, where these issues are properly handled by RVM / rbenv and Bundler separately).

Do you have any roadmap, plan to add Pipfile.lock support to pip (via the -p flag)? When can we expect it more or less?

[dragon788]

You can also manage the python version in the Pipfile, though I agree it is a little weird if for example you are using tox for 2 and 3.x (eg 3.5/3.6/3.7/3.8) compatibility testing that you need to also use a specific package tox-pipenv to enable compatibility with the virtualenvs created by pipenv for Tox to test against multiple Python versions.