micropipenv doesn't install all the dependencies of my main dependencies

[yrro]

Describe the bug

I've found that micropipenv is not installing some of the transitive dependencies of my project. That is, I depend on hypercorn, but the transitive dependencies h2, hpack, hyperframe and wsproto are not installed.

To Reproduce Steps to reproduce the behavior:

1. git clone https://github.com/yrro/ngfw-edl-server/ && cd ngfw-edl-server
2. python3 -m venv /tmp/1234
3. PIP_PYTHON=/tmp/1234/bin/python micropipenv install --deploy (I can't include the full output because of GitHub's limit but I can see the missing dependencies show up in the develop section instead of the default section.
4. /tmp/1234/bin/python -m pip list shows installed packages, but h2, hpack, hyperframe, wsproto are missing:

   Package         Version
   --------------- -------
   aiofiles        23.2.1
   aioprometheus   23.12.0
   blinker         1.8.2
   click           8.1.7
   dnspython       2.6.1
   Flask           3.0.3
   gunicorn        22.0.0
   h11             0.14.0
   Hypercorn       0.16.0
   itsdangerous    2.2.0
   Jinja2          3.1.4
   MarkupSafe      2.1.5
   orjson          3.10.3
   packaging       24.0
   pip             23.3.2
   quantile-python 1.1
   Quart           0.19.5
   setproctitle    1.3.3
   uvicorn         0.29.0
   Werkzeug        3.0.3

Expected behavior All the following packages to be installed:

$ poetry list --only=main
aiofiles        23.2.1  File support for asyncio.
aioprometheus   23.12.0 A Prometheus Python client library for asyncio-based applications
blinker         1.8.2   Fast, simple object-to-object and broadcast signaling
click           8.1.7   Composable command line interface toolkit
dnspython       2.6.1   DNS toolkit
flask           3.0.3   A simple framework for building complex web applications.
gunicorn        22.0.0  WSGI HTTP Server for UNIX
h11             0.14.0  A pure-Python, bring-your-own-I/O implementation of HTTP/1.1
h2              4.1.0   HTTP/2 State-Machine based protocol implementation
hpack           4.0.0   Pure-Python HPACK header compression
hypercorn       0.16.0  A ASGI Server based on Hyper libraries and inspired by Gunicorn
hyperframe      6.0.1   HTTP/2 framing layer for Python
itsdangerous    2.2.0   Safely pass data to untrusted environments and back.
jinja2          3.1.4   A very fast and expressive template engine.
markupsafe      2.1.5   Safely add untrusted strings to HTML/XML markup.
orjson          3.10.3  Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy
packaging       24.0    Core utilities for Python packages
priority        2.0.0   A pure-Python implementation of the HTTP/2 priority tree
quantile-python 1.1     Python Implementation of Graham Cormode and S. Muthukrishnan's Effective Computation of B...
quart           0.19.5  A Python ASGI web microframework with the same API as Flask
setproctitle    1.3.3   A Python module to customize the process title
uvicorn         0.29.0  The lightning-fast ASGI server.
werkzeug        3.0.3   The comprehensive WSGI web application library.
wsproto         1.2.0   WebSockets state-machine based protocol implementation

Additional context Maybe it's something about how hypercorn declares its dependencies in its own package metadata... not sure.

[yrro]

In _poetry2pipfile_lock there's this comment:

# Poetry 1.5+ no longer provides category in poetry.lock so we have to
# guess it from the content of pyproject.toml.
# All deps in groups are considered dev dependencies.

Which seems relevant. I guess this is sort of a known issue but there weren't actually any issues open with it demonstrating the impact on a real project before...

[frenzymadness]

Thanks for the report. I'll take a look at it.

[frenzymadness]

So, the problem is in this part of the dependency graph:

quart==0.19.6
├── aiofiles [required: Any, installed: 23.2.1]
├── blinker [required: >=1.6, installed: 1.8.2]
├── click [required: >=8.0.0, installed: 8.1.7]
├── flask [required: >=3.0.0, installed: 3.0.3]
│   ├── blinker [required: >=1.6.2, installed: 1.8.2]
│   ├── click [required: >=8.1.3, installed: 8.1.7]
│   ├── itsdangerous [required: >=2.1.2, installed: 2.2.0]
│   ├── jinja2 [required: >=3.1.2, installed: 3.1.4]
│   │   └── MarkupSafe [required: >=2.0, installed: 2.1.5]
│   └── werkzeug [required: >=3.0.0, installed: 3.0.3]
│       └── MarkupSafe [required: >=2.1.1, installed: 2.1.5]
├── hypercorn [required: >=0.11.2, installed: 0.17.3]
│   ├── h11 [required: Any, installed: 0.14.0]
│   ├── h2 [required: >=3.1.0, installed: 4.1.0]
│   │   ├── hpack [required: >=4.0,<5, installed: 4.0.0]
│   │   └── hyperframe [required: >=6.0,<7, installed: 6.0.1]
│   ├── priority [required: Any, installed: 2.0.0]
│   └── wsproto [required: >=0.14.0, installed: 1.2.0]
│       └── h11 [required: >=0.9.0,<1, installed: 0.14.0]
├── itsdangerous [required: Any, installed: 2.2.0]
├── jinja2 [required: Any, installed: 3.1.4]
│   └── MarkupSafe [required: >=2.0, installed: 2.1.5]
├── MarkupSafe [required: Any, installed: 2.1.5]
└── werkzeug [required: >=3.0.0, installed: 3.0.3]
    └── MarkupSafe [required: >=2.1.1, installed: 2.1.5]

and in the fact that we are processing the dependencies one by one in the order as they are in the poetry.lock. That means that we process h2 and hpack for example sooner than quart and at that time we don't know the category for them because quart hasn't been processed yet and because they are not explicit dependencies we put them to dev category.

I think this is fixable but I don't know what will break.

[frenzymadness]

The fix is ready to be reviewed in #299 and I've prepared a new test with an exact copy of your configuration.

[yrro]

Thanks for this everyone. I've tested my project with the new release and it works!

One weird thing I've noticed is that running micropipenv install --deploy creates a Pipfile.lock file even though my project is a poetry project. Did micropipenv always do that (very possible since I mostly run it in a disposable container, so the stray file might have gone un-noticed until now...)

[fridex]

That is expected and can be controlled, see the README file:

Besides printing, the tool also writes the content of Pipfile.lock (if a locked software stack is used) to the directory where lock files are present (for Pipenv files, the Pipfile.lock is kept untouched). This behaviour can be suppressed by providing MICROPIPENV_NO_LOCKFILE_WRITE=1 environment variable.

[yrro]

Thanks :)