Switch from `python_packages` to `python_requirements`

[bfirsh]

The design is outlined in this comment:

• support the requirements.txt format via python_requirements in cog.yaml. This exists today but is currently undocumented.
• continue to support python_packages in cog.yaml, but remove documentation for it.
• leave the door open for other dependency file types later (pipfile.lock, etc), but only support requirements.txt for now
• update python_requirements parsing behavior:
  • if an unqualified version of pytorch and tensorflow is specified (like 1.11.0), do magic to find the right version (torch==1.11.0+cpu, torch==1.11.0+cu113), and log it so the user knows this magic is happening.
  • if a fixed version of pytorch or tensorflow is specified by the user with extra qualifiers like torch==1.11.0+cu113 or torch==1.11.0+cpu, don't do any magic and install that exact version, and tell the user we're not doing any magic.
  • if a version range (package>=0.2,<0.3) is specified in requirements.txt (for any package), let pip handle the version resolution, but log a warning to the user recommending that versions in requirements.txt should be pinned for the sake of reproducibility.

Todo

• [x] #714
  • Define Python requirements with requirements.txt and python_requirements
  • Make python_packages generate a requirements.txt for backwards compatibility
  • Parse python_requirements to determine versions
• [ ] Tell user that magic is happening
• [ ] Display warnings if torch or tensorflow version can't resolved
• [ ] Documentation
• [ ] Update docs and examples

Design process

We should decide between python_requirements and python_packages and make it work properly.

There is some background here that is not written down anywhere that needs writing down. tl;dr python_requirements doesn't work properly, python_packages is not ideal in various ways.

Potential designs

1. Leave it how it is, remove python_requirements. The main downside of this is Python requirements are either in a non-standard location or duplicated.
2. Switch to python_requirements, and use a simple parser to determine torch and tensorflow versions. The main downside of this is that it is not clear to the user that Cog is rewriting versions.
3. Switch to python_requirements, but split out torch and tensorflow as separate top-level options to make it clear they do some special sauce behind the scenes. If torch/tensorflow is detected in python_requirements, then it ignores them, warns the user, or something else sensible.

User data

• Streamlit have learned that users normally already have some way of defining their dependencies (pip, Poetry, Conda, etc) and are unwilling to switch to a different method. So, they read dependencies from wherever the user has already defined them.
• Python dependencies should automatically install when you open a GitHub codespaces. They don't do that if they're inside python_packages, but they do if they're in a plain requirements.txt

(Consider this a wiki and please edit! Edited by @bfirsh, ...)

[zeke]

The main downside of this is that it is not clear to the user that Cog is rewriting versions.

Is Cog NOT rewriting versions when you use python_packages?

[bfirsh]

It is. And come to think of it this is a downside of both (1) and (2) -- it's not obvious Cog is rewriting versions in both cases.

Torch/Tensorflow packages need to be compiled for a particular version of CUDA, and in Torch's case, those packages are not on PyPi but on their custom index.

For example, if a user specifies torch==1.10.0 in python_packages, then Cog detects the best CUDA version to use and rewrites the install to be something like torch==1.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html.

[zeke]

users normally already have some way of defining their dependencies (pip, Poetry, Conda, etc) and are unwilling to switch to a different method. [...] read dependencies from wherever the user has already defined them.

☝🏼 I think this makes sense.

• Keep python_requirements
• Remove python_packages
• Use docs and logging to make it clear that torch and tensorflow versions are being modified.

[zeke]

Speaking of the myriad ways to define dependencies in Pythonland, I recently took the 2021 Python Developers survey and this was one of the questions:

I checked out the results from the 2020 survey thinking it might help us know which formats to prioritize, but it looks like they didn't ask that question last year.

[bfirsh]

A bit of interesting data from #325: we have some undocumented options python_extra_index_urls and python_find_links, but looks like nobody is using these for any public models:

https://github.com/search?q=python_find_links+filename%3Acog.yaml&type=code https://github.com/search?q=python_extra_index_urls+filename%3Acog.yaml&type=code

[zeke]

Here are some rough stats from GitHub on the popularity of different dependency formats:

Filename	Count
requirements.txt	2M
pipfile.lock	145K
pyproject.toml	136K
environment.yml	92K
poetry.lock	70K
constraints.txt	33K
environment.yaml	29K

[zeke]

Takeaways from an IRL conversation with @bfirsh today:

• support the requirements.txt format via python_requirements in cog.yaml. This exists today but is currently undocumented.
• continue to support python_packages in cog.yaml, but remove documentation for it.
• leave the door open for other dependency file types later (pipfile.lock, etc), but only support requirements.txt for now
• update python_requirements parsing behavior:
  • if an unqualified version of pytorch and tensorflow is specified (like 1.11.0), do magic to find the right version (torch==1.11.0+cpu, torch==1.11.0+cu113), and log it so the user knows this magic is happening.
  • if a fixed version of pytorch or tensorflow is specified by the user with extra qualifiers like torch==1.11.0+cu113 or torch==1.11.0+cpu, don't do any magic and install that exact version, and tell the user we're not doing any magic.
  • if a version range (package>=0.2,<0.3) is specified in requirements.txt (for any package), let pip handle the version resolution, but log a warning to the user recommending that versions in requirements.txt should be pinned for the sake of reproducibility.

[vccheng2001]

Hello. I was having a conversation with @andreasjansson just now about Conda support in Cog. This might be something to add future support for since many ML repos use conda/env.yml as the setup instead of pip, as Conda supports not only python packages but also C libraries, executables, etc.

Currently building C/C++ extensions in Cog (which many graphics/3D-related projects require) requires hacky workarounds (e.g. using subprocess calls, chaining commands under "run", etc), so it'd be cool to have a more robust solution for this.

[okaris]

@bfirsh @zeke Is there any new information about Conda support in Cog?

[mattt]

I agree with the plan of action @zeke articulated in https://github.com/replicate/cog/issues/157#issuecomment-1068606880, and I'm glad to see us making progress towards this with #714.

My top feedback from my first run of Cog has to do with a lack of requirements.txt file. Without it, PyLance and other editor functionality that I rely on didn't work. We should make python_requirements the default behavior in cog init going forward.

We can support other ecosystems like Conda by passing a different --package-manager flag to cog init (that flag would default to pip). For example, cog init --package-manager=conda would generate an environment.yml file.

[bfirsh]

Follow on from https://github.com/replicate/cog/pull/1016#issuecomment-1530003478...

I think the reason we didn't want to document it is because magic happens (waves hands), and it doesn't actually install the packages you expect in requirements.txt, which is unexpected. As outlined in the comments above, we want to tell the user clearly when magic happens so they know that package X is being replaced by package Y.

I don't think we need to remove the documentation for python_requirements though -- if anything this is a good motivation to get this fixed. 😄

[mattt]

I think the reason we didn't want to document it is because magic happens (waves hands), and it doesn't actually install the packages you expect in requirements.txt, which is unexpected. As outlined in the comments above, we want to tell the user clearly when magic happens so they know that package X is being replaced by package Y.

@bfirsh Ah, thanks for providing that context. I understand and appreciate why this was undocumented. Apologies if I unintentionally spoiled the trick. 😅

Just spitballing here, but what do you think about an approach like what pip-compile does? (I recently set this up for our Python client, actually: https://github.com/replicate/replicate-python/pull/84). You specify loose constraints are in pyproject.toml and then pip-compile uses those to generates a requirements.txt with exact requirements, acting as a lockfile. Rinse and repeat for dev requirements.

Maybe Cog could use packages defined in cog.yaml, resolve with magic, and then write out requirements.txt with the exact dependencies being installed? So still magic, but more "nothing up my sleeves" 🪄🎩

[bfirsh]

Looking back at this ticket, this is starting to feel like a clearer, more explicit design, particularly with the base image stuff that @andreasjansson and @tempusfrangit are working on...

3. Switch to python_requirements, but split out torch and tensorflow as separate top-level options to make it clear they do some special sauce behind the scenes. If torch/tensorflow is detected in python_requirements, then it ignores them, warns the user, or something else sensible.