sync packages after multiple pip-compile calls

[msftcangoblowm]

Issue description

Each requirement file is created by a call to pip-compile. Long story short, the package versions are not sync'ed after multiple pip-compile calls.

grep .lock files in requirements/*.lock, for a specific package, will reveal packages with different required versions. One venv can only hold one package version resulting in as predictable as rain in autumn dependency resolution conflicts.

.in requirements file can have both constraints -c and included -r requirements:

• prod.shared.in

For an app, the packages dependencies are normally locked. prod.shared.lock would be set in pyproject.toml.

This is an unusual situation. And the dev might choose to keep the production venv unlocked.

For normal situation, packages are unlocked. prod.shared.unlock would be set in pyproject.toml.

• pins.shared.in

Shared means this constraints file will be included across multiple venv. Extreme care must be taken. Any nudge pins would be general in nature, e.g. tomli or typing_extensions

• issue specific pins, e.g. requirements/pins-cffi.in

Gives the opportunity to document why there is a package version preference (nudge pin). Upstream issues urls to track the issue and ongoing work. With this info everyone, especially the original dev, have a chance to understand both cause and source. Then can decide if the nudge pin is still warranted.

The normal status quo is nudge pins have no accompanying documentation. When issues come up concerning a nudge pin, the background has to to be tracked down. If it even exists. With nudge pin documentation, all background info would be in one place.

Occurs when

After updating requirements files via multiple pip-compile calls.

always occurs and is very time consuming to get right manually

Would Resolve

Remove the need to check the versions of every single package

Proposed fix/feature

Need a process which occurs after the multiple pip-compile calls, which syncs the packages in the requirements files

[msftcangoblowm]

These looks at the venv, not the requirements files:

• pip list --not-required

• pip list --outdated

• pip check

These are very useful for managing venv.

For the requirements files, would like similar tools, preferably automated. Should help decide which packages' version is needed and update the appropriate .in requirements files.

• .shared.in and pins-\*.in (issue specific nudge pins) files

On difficult issues, an issue specific nudge pins file is needed. But for mere package version disparity issues, a single nudge pins file is proposed.

• .lock files

This gets recompiled before .unlock files cuz these files are ideal to check whether package version conflicts exists.

• .unlock files

This is a resolved, not locked file. After nudge pin files are updated, to reflect the changes, .unlock file would need to be recompiled

[msftcangoblowm]

pins-cffi.in

# musllinux aarch64 support; 3.13 support; fixes distutils.msvc9compiler failing imports
cffi>=1.17.1

This issue specific nudge pin file says the reason is:

• added support for py313

• musllinux aarch64 support

• fixes distutils.msvc9compiler failing imports

The gh workflows for testing, the strategy.matrix include py313 and Windows OS.

So this package version is needed. There is little need to ever remove this nudge pin. Possibly manually adjust as a hypothetical future release adds support or fixes important issues.

This nudge pin is a good thing. Shows cffi team are doing a beyond everyones' expectations incredible job.

Where is this nudge pin file included

alias dgrep='function _dgrep(){ cd "$1" 2>/dev/null && grep -rInH "$2" $1; };_dgrep'

cd requirements
dgrep . "pins-cffi.in"

./manage.in:7:-c pins-cffi.in ./dev.in:5:-c pins-cffi.in

dgrep . "cffi==" ./manage.lock:11:cffi==1.17.1 ./dev.lock:17:cffi==1.17.1

Luckily cffi package is in sync

Without the == dgrep . "cffi"

./manage.unlock:19:cffi>=1.17.1 ./pins-cffi.in:2:cffi>=1.17.1 ./manage.in:7:-c pins-cffi.in ./manage.lock:11:cffi==1.17.1 ./manage.lock:95: # via cffi ./dev.lock:17:cffi==1.17.1 ./dev.lock:118: # via cffi ./dev.unlock:26:cffi>=1.17.1 ./dev.in:5:-c pins-cffi.in

Other venvs?

docs/ folder has it's own venv. cffi is not a dependency. This is the reason why pins-cffi.in is not a pins-cffi.shared.in. The cffi package is not shared requirement in the same way prod.shared.in or pins.shared.in are.