Local package dependency

[JPHutchins]

What is the problem or limitation you are having?

In a project layout:

briefcase_app/
my_library/

I'd like to be able to specify the local dependency, my_library, so that it is included in the built app_packages. It works!

requires = [
    "my_library @ file:///C:/Users/jp/repos/project_root/my_library",
]

But only if I hard code the path to the local dependency. If I try with an environment variable:

requires = [
    "my_library @ file:///${PROJECT_ROOT}/my_library",
]

It seems that the environment variable is not expanded:

[briefcase_app] Installing requirements...
Processing c:\${project_root}\my_library
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\\${PROJECT_ROOT}\\my_library'

Describe the solution you'd like

Expansion of environment variables in the requires list or any other guidance for requiring local packages.

Describe alternatives you've considered

Hard code package or publish to PyPI.

Additional context

No response

[JPHutchins]

I wondered about the 3 slashes - maybe it should be with 2 - but it still does not work quite right.

requires = [
    "my_library @ file://${PROJECT_ROOT}/my_library",
]

Processing \\${project_root}\my_library
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '\\\\${PROJECT_ROOT}\\my_library'

[JPHutchins]

I see the pip install call here: https://github.com/beeware/briefcase/blob/287bc03467a28c22e7ccab2b1bdb5867189fb870/src/briefcase/commands/create.py#L432-L478

I'm surprised that it doesn't expand because I can do

pip install "my_library @ file:///${PROJECT_ROOT}/my_library"

And it works.

[rmartin16]

I see the pip install call here:

https://github.com/beeware/briefcase/blob/287bc03467a28c22e7ccab2b1bdb5867189fb870/src/briefcase/commands/create.py#L432-L478

I'm surprised that it doesn't expand because I can do

pip install "my_library @ file:///${PROJECT_ROOT}/my_library"

And it works.

At the command line, the shell (e.g. bash) is responsible for replacing ${PROJECT_ROOT} with its set value. Briefcase's use of subprocess doesn't involve a shell, though; so, resolving environment variables, expanding file globs, interpreting shell commands, etc. are not supported....and are generally unsafe.

That said, you can get subprocess to use a shell:

❯ ASDF="hello world" python -c 'import subprocess; subprocess.run("echo $ASDF", shell=True)'
hello world

This isn't easily made portable, though, since shells vary across platforms.

I'll defer the larger questions in this issue to freakboy3742.

[JPHutchins]

Agreed, and I don't see any excuse for shell=True.

Looking into what pdm is doing to get this working: https://pdm-project.org/en/stable/usage/advanced/#use-pdm-to-manage-a-monorepo

[JPHutchins]

OK, looks like pdm is doing it manually and just leaning on traditional environment variable syntax to do so: https://github.com/pdm-project/pdm/blob/d925136c3f78293d13de88587025ce354c2ce3e8/src/pdm/models/backends.py#L53-L58

[freakboy3742]

As @rmartin16 has noted, the reason ${PROJECT_ROOT} isn't being expanded is that shell expansion is done by the shell, and Briefcase isn't doing full shell expansion on commands passed to pip et al.

The good news is that there is an easy workaround: use a relative path, rather than the file URL form for specifying paths:

requires = [
    "./my_library"
]

Briefcase will guarantee that any relative path is interpreted relative to the project root, regardless of where the final app is built. I'm not aware of any functional differences between using a relative path and a formal file:// URL for specifying dependencies.

Toga uses this trick extensively - all the example code uses relative path referencing to make the example code run against the current working version of Toga.

In the past, I have contemplated whether Briefcase should allow for shell/template expansion in various configuration values (it would likely need to be jinja-style template expansion to ensure cross-platform, cross-shell compatibility). For example, it might be useful to include the Python or Briefcase version in a template path. However, with the addition of the -C key=value command line option, it's now possible to control config variables from the command line; most of the potential uses for template expansion that I've found can be satisfied that way.

Closing on the basis that there is a known workaround.

[JPHutchins]

Looks like PIP solved this years ago 🤣

https://github.com/pypa/pipenv/pull/958

use a relative path, rather than the file URL form for specifying paths

Looking good! Minor correction for completeness with my package layout:

requires = [
    "../my_library"
]

Because briefcases' pyproject.toml is not at the root of the repo.