Feature request: pip-compile support

[bullfest]

Cool project!

It would be nice with support for the pip-tools pip-compile command (basically creating lock files with pip dependencies from specification files)

Proposed solution If it's possible a solution could be to simply allow defining arbitrary lock-file commands if containerization is good enough so that it doesn't pose a security issue (haven't looked at any of the code, so no idea of how the project currently works).

Otherwise something like

"pipCompile": {
  "enabled": true 
   "inFile": "requirements.in"
   "outFile": "requirements.txt"
}

would probably be a good configuration that runs the command pip-compile [--outputfile <outFile>] [<inFile>].

[MrNaif2018]

Hi!

deterministic ordering of packages (to match pip-freeze ones)

I mean the pip-compile output itself is sorted by how pip freeze would output it (case-insensitive sort on package names as far as I know) The ordering of packages of course doesn't need to be configured. What I mean is, let's say there is the following output file:

a==1.0.1
b==1.0.0

then if we do upgrade b to 2.0.0, without that fix it would then do:

b==2.0.0
a==1.0.1

With this change it will leave the order of dependencies in output files the same

[MrNaif2018]

Update on this: I plan to work on making it work better in the following weeks (sorry, no exact time range, if someone can pick it up and implement it I will help during review and testing). I want to implement it this way, let me know if I'm missing some edge cases:

I would add a new config option to the pip-compile manager: something like outputPattern. It could be a string in sed-like syntax, i.e. s/in/txt. Or it could be a dictionary with 2 keys: searchPattern and replacePattern doing the same thing. File pattern should probably be left unconfigured, or configured to find .in files by default (as recommended by pip-compile docs, but not always used this way). Maybe some logic could be used to hint the user that we support pip-compile, but I guess searching for pip-compile header across all files would make API limits run out faster, so just if we document it that's fine. So the manager finds file by fileMatch just as others, but it combines two managers in one. The input files which are found by fileMatch are registered with pip-requirements manager with a special hook to call pip-compile each time input files change. By using sed expression got from config, which can still default to .in/.txt replacement, we get output files. So then if we need to update a dependency we run: pip-compile input-file -o output-file --upgrade-package packagename==x.y.z If we do lockfile refresh, we call pip-compile input-file -o output-file --upgrade

That way we also don't need to change existing pip-requirements behaviour much: files without any version constraints can be left unregistered with pip-requirements manager (as it makes no sense to re-compile requirements unless we do lockfile refresh) But I also think some manager param like extraArgs could be added either as a string or a list of arguments (not sure about shlex/shell splitting in node.js) to support some arguments often passed to pip-compile Or at least we could search output files by regex to find --hash argument or others, then we enable hash generation (this one is one of the most useful options for security also)

Wasn't there a use case where pip-compile can take multiple inputs?

Yes there is, but I think it can be solved without complications. We can test the replacement regex with different values (all matched files), and group input files by output files. Because when having multiple input files we still have one output file, usually uniting all dependencies from all files into one. So we take all input files by fileMatch, apply their transformation patterns and get output files. In case of, let's say, a regex which just changes .in to .txt, for N input files we would have N output groups with 1 input file in each. If people have multiple input files, then their output file pattern is predictable and often constant (i.e. dev.in, test.in, prod.in files compiled into requirements.txt). We would then have one output group with all N input files matching. Or it can be multiple output groups if pattern is special.

[m1n9o]

Any progress on this?

[paveldedik]

Maybe someone finds this useful, but the workaround for now we did in our company:

renovate.json

"packageRules": [
        {
            "matchManagers": ["pip_requirements"],
            "postUpgradeTasks": {
                "commands": [
                    "cd $(dirname {{{packageFile}}}) && pip-compile --upgrade-package={{{depName}}}=={{{newVersion}}} $(basename {{{packageFile}}} .txt).in"
                ],
                "fileFilters": ["**/requirements*.txt"]
            }
        }
    ]

config.js

module.exports = {
  allowPostUpgradeCommandTemplating: true,
  allowedPostUpgradeCommands: ['^pip-compile', '^cd'],
}

[m1n9o]

Maybe someone finds this useful, but the workaround for now we did in our company:

renovate.json

"packageRules": [
        {
            "matchManagers": ["pip_requirements"],
            "postUpgradeTasks": {
                "commands": [
                    "cd $(dirname {{{packageFile}}}) && pip-compile --upgrade-package={{{depName}}}=={{{newVersion}}} $(basename {{{packageFile}}} .txt).in"
                ],
                "fileFilters": ["**/requirements*.txt"]
            }
        }
    ]

config.js

module.exports = {
  allowPostUpgradeCommandTemplating: true,
  allowedPostUpgradeCommands: ['^pip-compile', '^cd'],
}

Good one, I am using docker instead, since the version of Python on renovate is 3.10. The pain point is that I have to use root privilege to run renovate for docker-in-docker.

[viceice]

Good one, I am using docker instead, since the version of Python on renovate is 3.10. The pain point is that I have to use root privilege to run renovate for docker-in-docker.

You don't need to run renovate as root for DinD, our helm chart works fine without root https://github.com/renovatebot/helm-charts/tree/main/charts/renovate

[MaxWinterstein]

Maybe someone finds this useful, but the workaround for now we did in our company:

renovate.json

"packageRules": [
        {
            "matchManagers": ["pip_requirements"],
            "postUpgradeTasks": {
                "commands": [
                    "cd $(dirname {{{packageFile}}}) && pip-compile --upgrade-package={{{depName}}}=={{{newVersion}}} $(basename {{{packageFile}}} .txt).in"
                ],
                "fileFilters": ["**/requirements*.txt"]
            }
        }
    ]

config.js

module.exports = {
  allowPostUpgradeCommandTemplating: true,
  allowedPostUpgradeCommands: ['^pip-compile', '^cd'],
}

This - in fact - is a nice approach but sadly limited at some point, e.g. when you need to pin a requirement to a specific version.

We use pip-compile-multi and multiple .in files. At some point we pinned versions with known issues or when major refactoring would be needed.

Using the pip_requirements manager does not respect those pinned restrictions at all, as it just reads the requirements.txt files.

As we fight with a lot dependencies we tend to bulk upgrade packages, so instead of passing each update individually to pip-compile we can speedup things multiple times by only running it once time at the end:

                "commands": [
                    "pip install pip-compile-multi && cd backend && bash update_requirements_txt.sh {{#each upgrades}}--upgrade-package={{{depName}}}=={{{newVersion}}} {{/each}}"
                ],

(We use some wrapper script around pip-compile-multi but usage is more or less the same.)

Package upgrades passed to pip-compile-multi are processed, regardless if they are not allowed by some pinned version in the .in file. We just filter them out inside our wrapper script, by some grep.

This leads to pull requests that contain package upgrades that either are not allowed and therefore not in the real commit and maybe - did not verify yet - to incompatible package constellations as they might not be parsed against the pinned versions.

So for the moment, I am pretty stuck.