search

gerbenoostra / poetry-plugin-mono-repo-deps

Poetry plugin for mono repos to replace path dependencies with named dependencies
MIT License
45 stars 1 forks source link

Unable to find installed packages #8

Closed ydennisy closed 2 days ago

ydennisy commented 1 month ago

Hi @gerbenoostra !

Thanks for the plugin, but currently I am unable to get this to work and would appreciate your help.

I am working in a polyglot monorepo with the following simplified layout:

monorepo/
  -- libs/python/utils
  -- services/pythonapp

I have two pyproject.toml files in each of the lib and service, the service is referencing via paths the lib and works ok locally. The issue is during my docker build:

FROM python:3.11-slim AS build

WORKDIR /app

COPY services/test-py-app ./app
COPY lib/python/utils ./utils

RUN pip install poetry && poetry config virtualenvs.create false && pip install poetry-plugin-mono-repo-deps
RUN cd utils && poetry build && poetry install
RUN cd ../app && poetry install

The issue is on the last line where I get:

Path /usr/lib/python/utils for ag-utils does not exist
Skipping virtualenv creation, as specified in config file.
Replacing path dependency ag-utils @ file:///usr/lib/python/utils in group main with ag-utils (>=0.0.1,<0.1.0)
Installing dependencies from lock file

Package operations: 0 installs, 1 update, 0 removals

  - Updating ag-utils (0.0.1 /app/utils -> 0.0.1): Failed

  RuntimeError

  Unable to find installation candidates for ag-utils (0.0.1)

  at /usr/local/lib/python3.11/site-packages/poetry/installation/chooser.py:74 in choose_for
       70│
       71│             links.append(link)
       72│
       73│         if not links:
    →  74│             raise RuntimeError(f"Unable to find installation candidates for {package}")
       75│
       76│         # Get the best link
       77│         chosen = max(links, key=lambda link: self._sort_key(package, link))
       78│

Cannot install ag-utils.

Which makes sense as how would poetry know this package is already installed on the machine?

It seems to me I am not quite grasping the correct usage of this plugin :(

gerbenoostra commented 1 month ago

The goal of the plugin is to have poetry build create wheels and tar.gz files that are pip installable. Thus a poetry build doesn't have any effect to a subsequent poetry install.

You could have a 2-stage Dockerfile, where you do the following in the build step (or you build the tar.gz/whl files outside docker in your CI/CD script):

FROM python:3.11-slim AS build
WORKDIR /app
COPY services ./services
COPY lib ./lib

RUN pip install poetry && poetry config virtualenvs.create false && pip install poetry-plugin-mono-repo-deps
RUN cd services/test-py-app && poetry build
RUN cd lib/python/utils && poetry build

This uses regular unmodified PyProject files, so the folder structure must be the same. I'd, therefore, prefer having a shell script that runs the poetry build for all folders and then copy these into the docker image

And in the following step you then install those wheels:

FROM python:3.11-slim AS app
WORKDIR /app
COPY --from=build /app/services/test-py-app/dist/*.whl /tmp/
COPY --from=build /app/lib/python/utils/dist/*.whl /tmp/
RUN pip install --pre --no-cache-dir --find-links /tmp/ test-py-app
ydennisy commented 2 days ago

Hey @gerbenoostra thanks for your reply!

I just wanted to let you know that for now I decided to use uv, since it has native support for some of these monorepo features and they seem to be adding more.

Thanks again for the plugin and your reply! Best, D