Special variable inside requirements.txt which contains path to current requirements.txt

[Daiver]

First of all i'm sorry for my poor English and pip skills What's the problem this feature will solve? This feature request is based on my stackoverflow question https://stackoverflow.com/questions/57590215/path-to-current-file-inside-requirements-txt

Pip allow to specify path to other packages in requirements.txt. For example -e path/to/some/package. It allow pip users to reuse local packages between local projects

For example suppose i have following structure of folders:

my_collection_of_packages/
    package1/
        package1/
            some.py
        setup.py
        requirements.txt
    package2/
        same_structure_as_package1
        requirements.txt
project1/
    requirements.txt

package2 depends on package1, so package2/requirements.txt contains following string -e ../package1/. So if my projects depends on package2 i can perform something like pip install -e project2 and it will install package1 too. But unfortunately, it will not work if my project is not at the same folder level with my packages. For example, if my project is project1 and project1/requirements.txt contains -e ../my_collection_of_packages/package2/ it's impossible to install package2/requirements.txt because it contains relative path ../package1/ which is equal project1/../package1/ if we run package2/requirements.txt.

Describe the solution you'd like From my point of view a special variable which contains path to current requirements.txt can be solution for this problem. I have no idea which name is better, but suppose it is ${requirements_pwd}. So now package2/requirements.txt can reference package1 by -e ${requirements_pwd}/../package1/ And project1 can reference package2 as -e ${requirements_pwd}/../my_collection_of_packages/package2/

Alternative Solutions It's possible to achieve same result by using absolute paths, but this makes usage of packages on different machines almost impossible.

For linux another solution is to define environment variable ${package_collection} and use it inside requirements.txt. For example package2/requirements.txt should be ${package_collection}/my_collection_of_packages/package1/. Have no idea how to make it work on windows and linux simultaneously

Additional context Link to stackoverflow question: https://stackoverflow.com/questions/57590215/path-to-current-file-inside-requirements-txt

Thank in advance. Please let me know if something wrong with my problem statement.

[pfmoore]

This seems to be based on a misunderstanding. Pip doesn't read requirements.txt for dependencies, rather it reads the dependency metadata in the packages themselves. To control where dependencies are installed from, you should be using --extra-index-url or --find-links.

Requirements files are only ever read by pip when they are explicitly referenced on the command line using the -r option.

[Daiver]

@pfmoore thank you for answer. Yeah, it's my mistake, i should put two lines inside projects and packages requirements to make everything work:

So package2/requirements.txt should be:

-r ../package1/requirements.txt
-e ../package1

Same for project1/requirements.txt

I still need some way to specify paths to package1 and package1/requirements.txt without absolute paths

[pfmoore]

It still seems like you are misusing requirements files and should be using dependency metadata.

[Daiver]

@pfmoore do i understand correctly that i shouldn't use requirements.txt to manage my local packages?

[pfmoore]

This might help clarify.

[Daiver]

Thank you for article! -e . in requirements.txt is nice idea. But i still cannot realize how i should reference my local packages in their requirements.txt.

[chrahunt]

Typically you shouldn't reference local packages from a parent directory in requirements.txt.

Try this instead:

1. package2/setup.py should have install_requires=['package1']
2. project1/requirements.txt should have package2

Then do

1. pip install -e . in package1
2. pip install -e . in package2
3. pip install -r requirements.txt in project1

The order only matters here because more than likely package1 and package2 have not been uploaded to PyPI.

Later on, when package1 and package2 have had several versions uploaded, getting started working on project1 looks like:

1. git clone https://github.com/org/project1.git && cd project1 && pip install -r requirements.txt - this will pull the required versions of package2 and package1 from PyPI
2. If some change is needed to package2, then do git clone https://github.com/org/package2.git && cd package2 && pip install -e .
3. If some change is needed to package1, then do git clone https://github.com/org/package1.git && cd package1 && pip install -e .

If you work this way then you only need to have the packages that you want to change. If you hard-code relative paths in requirements.txt then you always need to manually download the packages. That is a pain - it would be better to just put the packages in the project at that point.

[Daiver]

Hi @chrahunt ! Thank you for answer and thank you for suggestions! Actually currently i'm doing something very similar. But i want to avoid pip install -e package1;pip install -e package2;pip install -r requirements.txt because i use a lot of local packages with complicated dependencies. I need something like pip install -r local-requirements.txt which will collect all my local dependencies automatically. So currently i have a script which installs all local packages to project's venv. But it's temporal solution. I'm totally agree that hard-coded relative paths is terrible, but i just cannot figure out other solution. Currently manual download is not an issue for me because i keep all packages in monorepository.