Python: function to remove requirements.txt/setup.py constraints

[FRidh]

Issue description

Python developers often pin their dependencies to exact versions. This is bad practice and most of the times absolutely not necessary either.

Typically we use substituteInPlace to remove the dependency requirement. It would be useful to have a function that would strip such exact requirements.

I've also opened an issue at setuptools with a request for a flag that ignores version pinning. https://github.com/pypa/setuptools/issues/894

[FRidh]

Example why we should have this: https://github.com/NixOS/nixpkgs/pull/21321

[johbo]

I can see the need in regards to the setup.py file. I've myself sometimes patched away a specific version pinning in a setup.py.

How do you see the use related to a requirements.txt file? Wouldn't this typically be used in a scenario to generate nix code e.g. via pip2nix or pypi2nix?

[FRidh]

Actually, many developers wrongly load the requirements.txt file and add the items to install_requires. Therefore, a method to just filter the install_requires and setup_requires values should be sufficient.

That's also why I opened an issue on the setuptools trackers, since I think it's best implemented there.

[mmahut]

Are there any updates to this issue, please?

[FRidh]

Reading https://github.com/pypa/setuptools/issues/894 again. The buildPhase will produce a wheel, and that will contain the constraints (in METADATA, the Requires-Dist rows). We can thus fix it at our side by patching the wheel. Would also need to fix then the hash for METADATA in RECORD. The are tools to manipulate wheels. This can be implemented in a hook.

[danbst]

continuing my line of though with sed patching, I came with following hook:

function pythonJailbreak {
    local file="$1"; shift

    for dep in $@; do
        if [[ "$file" =~ requirements*.txt ]]; then
            sed -r -i "s/$dep[><=].*?/'$dep'/g" $file
        else
            sed -r -i "s/['\"]$dep[><=].*?['\"]/'$dep'/g" $file
        fi
    done
}

function toJailbreak {
    local file="$1"
    if [[ "$file" =~ requirements* ]]; then
        sed -rn "s/([a-zA-Z0-9+-]+)[><=].*/\1/p" $file
    else
        sed -rn "s/.*['\"]([a-zA-Z0-9_-]+)[><=].*['\"].*/\1/p" $file
    fi
}

function jailbreakHook {
    if [[ "$doJailbreak" != "false" ]]; then
        if [ -f setup.py ]; then
            pythonJailbreak setup.py $(toJailbreak setup.py)
        fi
        for reqtxt in $(ls -d requirements*.txt); do
            pythonJailbreak "$reqtxt" $(toJailbreak $reqtxt)
        done
    fi
}

It jailbreaks ~95% of dependencies for apache-airflow, so is not "production-grade", but package is built and tested fine.

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[Artturin]

Added by https://github.com/NixOS/nixpkgs/pull/170813