listing git files failed on git repo

[jaraco]

While troubleshooting https://github.com/pypa/distutils/issues/224, I've stumbled onto another issue.

I'm running pyproject-build . on a number of projects, and in the output, I see the following warning for both sdist and wheel generation:

ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any

This is happening across multiple projects I develop.

``` draft @ git clone gh://jaraco/pip-run Cloning into 'pip-run'... remote: Enumerating objects: 3447, done. remote: Counting objects: 100% (819/819), done. remote: Compressing objects: 100% (318/318), done. remote: Total 3447 (delta 483), reused 788 (delta 467), pack-reused 2628 Receiving objects: 100% (3447/3447), 688.86 KiB | 6.75 MiB/s, done. Resolving deltas: 100% (2058/2058), done. draft @ cd pip-run pip-run main @ pyproject-build -V build 1.0.3 (/Users/jaraco/.local/pipx/venvs/build/lib/python3.12/site-packages/build) pip-run main @ pyproject-build . * Creating venv isolated environment... * Installing packages in isolated environment... (setuptools>=56, setuptools_scm[toml]>=3.4.1) * Getting build dependencies for sdist... running egg_info creating pip_run.egg-info writing pip_run.egg-info/PKG-INFO writing dependency_links to pip_run.egg-info/dependency_links.txt writing entry points to pip_run.egg-info/entry_points.txt writing requirements to pip_run.egg-info/requires.txt writing top-level names to pip_run.egg-info/top_level.txt writing manifest file 'pip_run.egg-info/SOURCES.txt' adding license file 'LICENSE' writing manifest file 'pip_run.egg-info/SOURCES.txt' * Building sdist... running sdist running egg_info writing pip_run.egg-info/PKG-INFO writing dependency_links to pip_run.egg-info/dependency_links.txt writing entry points to pip_run.egg-info/entry_points.txt writing requirements to pip_run.egg-info/requires.txt writing top-level names to pip_run.egg-info/top_level.txt adding license file 'LICENSE' writing manifest file 'pip_run.egg-info/SOURCES.txt' running check creating pip-run-12.4.1.dev26+gbf42021 creating pip-run-12.4.1.dev26+gbf42021/.github creating pip-run-12.4.1.dev26+gbf42021/.github/workflows creating pip-run-12.4.1.dev26+gbf42021/docs creating pip-run-12.4.1.dev26+gbf42021/examples creating pip-run-12.4.1.dev26+gbf42021/pip_run creating pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info creating pip-run-12.4.1.dev26+gbf42021/pip_run/retention creating pip-run-12.4.1.dev26+gbf42021/tests copying files to pip-run-12.4.1.dev26+gbf42021... copying .coveragerc -> pip-run-12.4.1.dev26+gbf42021 copying .editorconfig -> pip-run-12.4.1.dev26+gbf42021 copying .pre-commit-config.yaml -> pip-run-12.4.1.dev26+gbf42021 copying .readthedocs.yaml -> pip-run-12.4.1.dev26+gbf42021 copying LICENSE -> pip-run-12.4.1.dev26+gbf42021 copying NEWS.rst -> pip-run-12.4.1.dev26+gbf42021 copying README.rst -> pip-run-12.4.1.dev26+gbf42021 copying conftest.py -> pip-run-12.4.1.dev26+gbf42021 copying mypy.ini -> pip-run-12.4.1.dev26+gbf42021 copying pip-run.py -> pip-run-12.4.1.dev26+gbf42021 copying pyproject.toml -> pip-run-12.4.1.dev26+gbf42021 copying pytest.ini -> pip-run-12.4.1.dev26+gbf42021 copying ruff.toml -> pip-run-12.4.1.dev26+gbf42021 copying setup.cfg -> pip-run-12.4.1.dev26+gbf42021 copying towncrier.toml -> pip-run-12.4.1.dev26+gbf42021 copying tox.ini -> pip-run-12.4.1.dev26+gbf42021 copying .github/FUNDING.yml -> pip-run-12.4.1.dev26+gbf42021/.github copying .github/dependabot.yml -> pip-run-12.4.1.dev26+gbf42021/.github copying .github/workflows/main.yml -> pip-run-12.4.1.dev26+gbf42021/.github/workflows copying docs/conf.py -> pip-run-12.4.1.dev26+gbf42021/docs copying docs/cowsay.svg -> pip-run-12.4.1.dev26+gbf42021/docs copying docs/history.rst -> pip-run-12.4.1.dev26+gbf42021/docs copying docs/index.rst -> pip-run-12.4.1.dev26+gbf42021/docs copying examples/plotter.ipynb -> pip-run-12.4.1.dev26+gbf42021/examples copying examples/pydragon -> pip-run-12.4.1.dev26+gbf42021/examples copying examples/pydragon.py -> pip-run-12.4.1.dev26+gbf42021/examples copying examples/test-mongodb-covered-query.py -> pip-run-12.4.1.dev26+gbf42021/examples copying pip_run/__init__.py -> pip-run-12.4.1.dev26+gbf42021/pip_run copying pip_run/__main__.py -> pip-run-12.4.1.dev26+gbf42021/pip_run copying pip_run/_py38compat.py -> pip-run-12.4.1.dev26+gbf42021/pip_run copying pip_run/commands.py -> pip-run-12.4.1.dev26+gbf42021/pip_run copying pip_run/deps.py -> pip-run-12.4.1.dev26+gbf42021/pip_run copying pip_run/launch.py -> pip-run-12.4.1.dev26+gbf42021/pip_run copying pip_run/read-deps.py -> pip-run-12.4.1.dev26+gbf42021/pip_run copying pip_run/scripts.py -> pip-run-12.4.1.dev26+gbf42021/pip_run copying pip_run/usage.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run copying pip_run.egg-info/PKG-INFO -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info copying pip_run.egg-info/SOURCES.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info copying pip_run.egg-info/dependency_links.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info copying pip_run.egg-info/entry_points.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info copying pip_run.egg-info/requires.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info copying pip_run.egg-info/top_level.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info copying pip_run/retention/__init__.py -> pip-run-12.4.1.dev26+gbf42021/pip_run/retention copying pip_run/retention/destroy.py -> pip-run-12.4.1.dev26+gbf42021/pip_run/retention copying pip_run/retention/persist.py -> pip-run-12.4.1.dev26+gbf42021/pip_run/retention copying tests/test_commands.py -> pip-run-12.4.1.dev26+gbf42021/tests copying tests/test_deps.py -> pip-run-12.4.1.dev26+gbf42021/tests copying tests/test_launch.py -> pip-run-12.4.1.dev26+gbf42021/tests copying tests/test_scripts.py -> pip-run-12.4.1.dev26+gbf42021/tests copying pip_run.egg-info/SOURCES.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info Writing pip-run-12.4.1.dev26+gbf42021/setup.cfg Creating tar archive removing 'pip-run-12.4.1.dev26+gbf42021' (and everything under it) * Building wheel from sdist * Creating venv isolated environment... * Installing packages in isolated environment... (setuptools>=56, setuptools_scm[toml]>=3.4.1) * Getting build dependencies for wheel... running egg_info writing pip_run.egg-info/PKG-INFO writing dependency_links to pip_run.egg-info/dependency_links.txt writing entry points to pip_run.egg-info/entry_points.txt writing requirements to pip_run.egg-info/requires.txt writing top-level names to pip_run.egg-info/top_level.txt ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any reading manifest file 'pip_run.egg-info/SOURCES.txt' adding license file 'LICENSE' writing manifest file 'pip_run.egg-info/SOURCES.txt' * Installing packages in isolated environment... (wheel) * Building wheel... running bdist_wheel running build running build_py creating build creating build/lib copying pip-run.py -> build/lib creating build/lib/pip_run copying pip_run/deps.py -> build/lib/pip_run copying pip_run/read-deps.py -> build/lib/pip_run copying pip_run/__init__.py -> build/lib/pip_run copying pip_run/_py38compat.py -> build/lib/pip_run copying pip_run/launch.py -> build/lib/pip_run copying pip_run/scripts.py -> build/lib/pip_run copying pip_run/commands.py -> build/lib/pip_run copying pip_run/__main__.py -> build/lib/pip_run creating build/lib/pip_run/retention copying pip_run/retention/destroy.py -> build/lib/pip_run/retention copying pip_run/retention/persist.py -> build/lib/pip_run/retention copying pip_run/retention/__init__.py -> build/lib/pip_run/retention running egg_info writing pip_run.egg-info/PKG-INFO writing dependency_links to pip_run.egg-info/dependency_links.txt writing entry points to pip_run.egg-info/entry_points.txt writing requirements to pip_run.egg-info/requires.txt writing top-level names to pip_run.egg-info/top_level.txt ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any reading manifest file 'pip_run.egg-info/SOURCES.txt' adding license file 'LICENSE' writing manifest file 'pip_run.egg-info/SOURCES.txt' copying pip_run/usage.txt -> build/lib/pip_run installing to build/bdist.macosx-14-arm64/wheel running install running install_lib creating build/bdist.macosx-14-arm64 creating build/bdist.macosx-14-arm64/wheel creating build/bdist.macosx-14-arm64/wheel/pip_run copying build/lib/pip_run/deps.py -> build/bdist.macosx-14-arm64/wheel/pip_run copying build/lib/pip_run/read-deps.py -> build/bdist.macosx-14-arm64/wheel/pip_run creating build/bdist.macosx-14-arm64/wheel/pip_run/retention copying build/lib/pip_run/retention/destroy.py -> build/bdist.macosx-14-arm64/wheel/pip_run/retention copying build/lib/pip_run/retention/persist.py -> build/bdist.macosx-14-arm64/wheel/pip_run/retention copying build/lib/pip_run/retention/__init__.py -> build/bdist.macosx-14-arm64/wheel/pip_run/retention copying build/lib/pip_run/__init__.py -> build/bdist.macosx-14-arm64/wheel/pip_run copying build/lib/pip_run/_py38compat.py -> build/bdist.macosx-14-arm64/wheel/pip_run copying build/lib/pip_run/launch.py -> build/bdist.macosx-14-arm64/wheel/pip_run copying build/lib/pip_run/scripts.py -> build/bdist.macosx-14-arm64/wheel/pip_run copying build/lib/pip_run/commands.py -> build/bdist.macosx-14-arm64/wheel/pip_run copying build/lib/pip_run/__main__.py -> build/bdist.macosx-14-arm64/wheel/pip_run copying build/lib/pip_run/usage.txt -> build/bdist.macosx-14-arm64/wheel/pip_run copying build/lib/pip-run.py -> build/bdist.macosx-14-arm64/wheel running install_egg_info Copying pip_run.egg-info to build/bdist.macosx-14-arm64/wheel/pip_run-12.4.1.dev26+gbf42021-py3.12.egg-info running install_scripts creating build/bdist.macosx-14-arm64/wheel/pip_run-12.4.1.dev26+gbf42021.dist-info/WHEEL creating '/Users/jaraco/draft/pip-run/dist/.tmp-fg6udeqq/pip_run-12.4.1.dev26+gbf42021-py3-none-any.whl' and adding 'build/bdist.macosx-14-arm64/wheel' to it adding 'pip-run.py' adding 'pip_run/__init__.py' adding 'pip_run/__main__.py' adding 'pip_run/_py38compat.py' adding 'pip_run/commands.py' adding 'pip_run/deps.py' adding 'pip_run/launch.py' adding 'pip_run/read-deps.py' adding 'pip_run/scripts.py' adding 'pip_run/usage.txt' adding 'pip_run/retention/__init__.py' adding 'pip_run/retention/destroy.py' adding 'pip_run/retention/persist.py' adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/LICENSE' adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/METADATA' adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/WHEEL' adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/entry_points.txt' adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/top_level.txt' adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/RECORD' removing build/bdist.macosx-14-arm64/wheel Successfully built pip-run-12.4.1.dev26+gbf42021.tar.gz and pip_run-12.4.1.dev26+gbf42021-py3-none-any.whl ```

If I install the build requirements locally and run py -m build --no-isolation . and put a breakpoint in the setuptools_scm._file_finders.git:_git_toplevel, I see that when the error occurs, the git command is being run against a copy of the source that doesn't include the git directory. Something about the build ecosystem is causing the .git directory that I clearly cloned not to be included in the build. I tried running with some older versions of build going back to 0.9, but that didn't seem to make a difference.

So far, I've only replicated this on my own system. I'd test in an isolated environment, but I don't have ready access to Docker at the moment. I'll test again after restoring Docker.

[jaraco]

I've replicated the issue in a Linux Docker container as well, so the issue isn't peculiar to my environment.

Curiously, even though I'm seeing these errors when running builds, I'm finding that files are being included that would seem to depend on git metadata, such as Lorem ipsum.txt in jaraco.text.

Oh. I think I see what's happening now. build is building the sdist first, which does rely on the file finders, but then builds the wheel from the sdist, which relies on the manifest in the sdist. I'll have to revisit the original project (distutils) to see where things are going wrong there (why nothing is being discovered).

[jaraco]

Aha. So I've clarified my confusion. The errors are expected to be emitted. They just need not to appear during the sdist phase. The reason there were issues with the distutils build was not because of scm file finders, but because of an exclude directive. The error from setuptools_scm was a red herring.

[bulletmark]

@jaraco , sorry but your last two comments are a little too technical for me to understand. I get this same error every time I do a build even though the build seems successful. How can I get rid of that error message?

[jaraco]

Because of the way setuptools is engineered, it will necessarily involve setuptools_scm at both phases of the build, during the sdist and then again at the wheel. It's okay to see the error message during the wheel build because it no longer needs the SCM metadata because that's been embedded in the sdist during the sdist phase. In short, I don't think there is currently a way for Setuptools or setuptools_scm to know that it's doing a "wheel from sdist" and thus to disable features like "search for files using git metadata". Therefore, I don't think there's a way to get rid of the error message. Perhaps there's something that could be done, but it would require coordination between build and the backends (setuptools) and their plugins (setuptools_scm).

I'll move this issue to packaging-problems for consideration.

[bulletmark]

Thanks for your reply.

The message reports a big fat ERROR, not even a warning so I assumed I must have a mistake somewhere and have spent ages trying different fixes which did not change anything. It is disconcerting to see that ERROR reported so this clearly must be considered a bug.

[ssbarnea]

Is there anything we can to do avoid this message or the only option is to attempt to switch to another packaging alternative? Clearly the linked packaging issue seems to claim is a setuptools issue and not generic packaging one.

To my surprise i seen two that updated their scm to 8 to get rid of if, but that does not seem to be the case.

[RonnyPfannschmidt]

The error only ought to happen when git is detected but breaks horribly

We might be able to narrow it down better if the file finder hooks received more than a path

Unfortunately they don't so there's situations with false positives outweigh ing the rwst atm

[RonnyPfannschmidt]

I just took a look again, theres a bug in finding the git top level that's in need of a fix