pre-commit hook not picking all the pyproject.toml files in the repo

[Opalo]

I've the following repository layout:

- libs/
  - l1/
   - pyproject.toml
  - l2/
   - pyproject.toml

- batch/
  - b1/
   - pyproject.toml
  - b2/
   - pyproject.toml

- pipelines/
  - p1/
   - pyproject.toml
  - p2/
   - pyproject.toml

- pyproject.toml

and the following pre-commit hook defined:

---
files: .(yaml|yml|py|toml)$
repos:
  - repo: https://github.com/jazzband/pip-tools
    rev: 7.3.0
    hooks:
      - id: pip-compile
        name: Run pip-compile for 'prod' env.
        args:
          - -o
          - pyproject.lock
          - --generate-hashes
          - --strip-extras

But for some reason it does not pick all the pyproject.toml files - it only picks the one in the root folder. Why is that? I was trying to configure files: option for the hook to no avail. From my research I see that I'd need to create the hook conf for each file separately - is this a case?

Environment Versions

1. OS: macOS Darwin opalmac.local 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6000 arm64
2. Python version: Python 3.9.16
3. pip version: pip 23.2.1
4. pip-tools version: 7.3.0

Steps to replicate

You can play around with this repository. With the current settings no pyproject.toml files is being "locked". If I add files: pyproject.toml only the root file will be processed.

Expected result

All the pyproject.toml files in the repo have locked dependencies.

Actual result

None of the pyproject.toml files in the repo have locked dependencies.

[WhyNotHugo]

pass_filenames is set to false:

https://github.com/jazzband/pip-tools/blob/65b0d3672ec737db9d9a9dfef737f88b8344129d/.pre-commit-hooks.yaml#L7

So while the hook runs for when any matching file changes, the filename is not passed and pip-tools falls back to the default pyproject.toml.

[WhyNotHugo]

Changing pass_filenames: true would break some situations. For example, if requirements.txt changes, then it would be passed as input to pip-compile and produce erroneous output.

[Opalo]

Thanks, will try that out tomorrow. Can you elaborate on the possible breaking situations, please?

[RonnyPfannschmidt]

The hook is working as intended

It picks up a single requirements file and output's a single lock file

My understanding is that each lock you want needs a own call

[Opalo]

So, you say, that if I have n files I need to configure n hooks, correct? I'm not saying that this is incorrect but not something I expected. E.g. for other hooks it's enough to configure the hook once - flake8, black, etc.

[webknjaz]

So, you say, that if I have n files I need to configure n hooks, correct?

Yes.

[webknjaz]

for other hooks it's enough to configure the hook once - flake8

They work in an absolutely different context, though. Also, multiple pyproject.toml files implies multiple standalone projects.

[webknjaz]

With the current settings no pyproject.toml files is being "locked".

I think you misunderstand the concept of the lockfiles. They define pins for virtualenvs where you run your apps or tests. But there's no direct connection between a project and an env. There's no heuristic to magically guess how you intend to use the constraints either. Each case is unique. My projects have tens of constraint files, for example.

[Opalo]

Thanks for the reply.

I think you misunderstand the concept of the lockfiles.

Actually, I don't think so. I plan to use them as intended - to lock the dependencies and use them to install in an virtualenv/docker later on. I don't find any place for misunderstanding here, do I? The context here is I just have a monorepo with a set of python projects. I exepcted the hook to lock the dependencies for each project separately but this is not how it was designed - and this is what I'm asking here. Whether it was or not. To be honest I don't find any particular reason why it cannot lock the files for each project but maybe I'm missing something here.

[WhyNotHugo]

Thanks, will try that out tomorrow. Can you elaborate on the possible breaking situations, please?

The hook triggers when either requirements.txt or requirements.in are changed (to keep them in sync). When either changes, the hook runs and reads requirements.in as input and generates requirements.txt as output.

If you switch to pass_filenames: true and modify requirements.txt, then it will be used as input by pip-compile. This will not produce the same results as providing requirements.in as input.

So, you say, that if I have n files I need to configure n hooks, correct? I'm not saying that this is incorrect but not something I expected. E.g. for other hooks it's enough to configure the hook once - flake8, black, etc.

These hooks all operate on a file and alter that file in-place. None of these hooks modify a separate file as output.

[Opalo]

Thank you @WhyNotHugo!

[Opalo]

Ok, I guess nothing new will be posted here, thanks for your help!