Heroku deployment is incompatible with VCS-based versioning using `setuptools-scm` and friends

[dhdaines]

It's become common practice to use something like setuptools-scm to manage package versioning automatically using Git or other version control history (sometimes via another tool like hatch-vcs).

What this often means in practice (and I admit it is not too clear from the setuptools-scm documentation) is that the project version is defined in a Python module, e.g. mypackage/__version__.py which is NOT checked in to Git but is generated dynamically when the project is installed (possibly in "editable" mode with pip install -e .).

You may see where I'm going with this - this is fundamentally incompatible with the way Heroku deploys things, because, from what I can tell:

1. Heroku's so-called "compile" process does not actually build (or compile) your app in any way except if there is no requirements.txt file.
2. Thus, anything generated as part of the build process is not actually generated (no Cython for you!). Okay, well I guess you can use bin/post_compile to actually, um, compile your app.
3. But here's the really fun part: despite using Git for deployment, you don't actually have access to the Git history in the build process. So, no versions for you, either!

So, you can hack around this by using the SETUPTOOLS_SCM_PRETEND_VERSION environment variable, which I guess is fine as long as you don't actually care about having the same version number as you otherwise would have.

In the case where your Heroku app is also a library distributed on PyPI, this is kind of annoying.

[edmorley]

Hi!

You are correct that the .git/ directory is not present during the build itself (and thus not available to buildpacks). However, this design choice is not specific to the Python buildpack, but instead the whole Heroku build system. It has been implemented this way for some time - I would imagine due to some combination of:

1. Wanting to avoid the .git/ directory from ending up in the slug
2. Wanting to reduce the size of the payload sent from the git.heroku.com server to the build system
3. Wanting to avoid the need to clone the Git repo in the case of GitHub sync workflows and instead being able to use the direct source archive tarball instead
4. No one having asked for this before (deploying a library repo directly is very rare)

However, the originally commit information is available via the SOURCE_VERSION env var: https://devcenter.heroku.com/articles/buildpack-api#bin-compile

Since this issue is not specific to the Python buildpack, it would be best to file this as a "make Git repo available during the build" feature request here: https://github.com/heroku/roadmap

As a workaround for this specific scenario, it seems your options are either:

1. Set the SETUPTOOLS_SCM_PRETEND_VERSION env var on your app to a fixed fake value (which as you say, won't match the actual version)
2. See if there is a way to configure setuptools-scm to use SOURCE_VERSION if it's set.
3. Or, if setuptools-scm can't be made to use SOURCE_VERSION directly, then to create a buildpack that exports the SOURCE_VERSION env var to the env var SETUPTOOLS_SCM_PRETEND_VERSION using the export file: https://devcenter.heroku.com/articles/buildpack-api#composing-multiple-buildpacks

[dhdaines]

Thanks for the really detailed response - I definitely understand why copying the .git folder to the deployment isn't a great idea. I'm a little bit surprised nobody ran into this problem before though...

It looks like there is a workaround possible, so I'll post it here when I figure it out and close the issue at that point (since as you say this is by design).

[edmorley]

No problem!

3. Or, if setuptools-scm can't be made to use SOURCE_VERSION directly, then to create a buildpack that exports the SOURCE_VERSION env var to the env var SETUPTOOLS_SCM_PRETEND_VERSION using the export file: https://devcenter.heroku.com/articles/buildpack-api#composing-multiple-buildpacks

I haven't tested this (so there might be a typo in there), but a buildpack with a bin/compile script with something like the following contents should work:

#!/usr/bin/env bash

set -euo pipefail

BUILDPACK_DIR=$(cd $(dirname $0); cd ..; pwd)

# Make the `SETUPTOOLS_SCM_PRETEND_VERSION` env var available to later buildpacks.
# `SOURCE_VERSION` is set by the Heroku build system:
# https://devcenter.heroku.com/articles/buildpack-api#bin-compile
echo "export SETUPTOOLS_SCM_PRETEND_VERSION='${SOURCE_VERSION}'" > "${BUILDPACK_DIR}/export"

[edmorley]

Closing since this isn't an issue with the Python buildpack, and there are workarounds available.