Monorepo with serverless-python-requirements breaks plugin packaging

[david-a-jetter]

Are you certain it's a bug?

• [X] Yes, it looks like a bug

Are you using the latest version?

• [X] Yes, I'm using the latest version

Is there an existing issue for this?

• [X] I have searched existing issues, it hasn't been reported yet

Issue description

I'm not sure if this is better filed here or in https://github.com/serverless/serverless-python-requirements

Use case: A Python monorepo with all pypi requirements managed through Poetry in a single pyproject.toml file, containing multiple Serverless services, each using serverless-python-requirements plugin being migrated from serialized serverless deploy commands to Serverless Compose. After moving each service's yml file into its own subdirectory, adding the requisite serverless-compose.yml in the root, and running npx sls deploy, each package's zip file contains just the source code and no python packages. The serverless-python-requirements plugin did not run at all.

Expected Behavior: The source code matching the package.patterns is included, as well as all non-dev python dependencies produced by serverless-python-requirements are included.

Actual Behavior: Only the source code matching the package.patterns is included, but the serverless-python-requirements plugin does not run.

The repository structure is roughly outlined as:

pyproject.toml
poetry.lock
serverless-compose.yml
src/
package/
  - appone/
    - serverless.appone.yml
  - apptwo/
    - serverless.apptwo.yml
  - appthree/
    - serverless.appthree.yml

Workaround: Use copyfiles npm package (which ensures cross-platform compatibility) to copy the pyproject.toml file into each app's subdirectory, eg: npx copyfiles pyproject.toml poetry.lock package/appone before running npx sls deploy and the serverless-python-requirements plugin behaves as expected.

Service configuration (serverless-compose.yml) content

services:
  appone:
    path: package/appone
    config: serverless.appone.yml
  apptwo:
    path: package/apptwo
    config: serverless.apptwo.yml
  appthree:
    path: package/appthree
    config: serverless.appthree.yml

Command name and used flags

npx serverless deploy --stage dev

Command output

N/A

[mnapoli]

each package's zip file contains just the source code and no python packages

I'm understanding this is the problem?

Do you get the same issue if you run sls deploy in each service directory? (i.e. without compose) That might help to debug what's going on.

[david-a-jetter]

@mnapoli Thanks for the response. Yes, that bit you highlight is the problem. I edited my issue to clearly state expected + actual behavior to help clarify any future readers as well.

Running from within the individual service directory results in the same undesirable behavior (no python dependencies included).

Copying the pyproject.toml file into the individual directory and running the same sls command, bypassing compose, results in the plugin running and packages being included as desired.

[mnapoli]

Thanks! Sorry I'm not familiar with Python's tooling here, but it seems like it's not related to Compose directly. It's more that pyproject.toml seems to be needed by the plugin for packaging to happen correctly?

Maybe Compose created a false expectation here that things would work by putting it at the root?

[david-a-jetter]

@mnapoli Yeah, I think you're right - this is more of an issue with how the plugin behaves, expecting that the pyproject.toml file is in the same directory as the serverless.yml that's being deployed.

I've opened an issue in their github as well: https://github.com/serverless/serverless-python-requirements/issues/714

I figured having one here would help with visibility, and hopefully people stumble across the workaround that, while it seems simple enough, it took me several hours of headscratching to finally come up with 😅

[chapimenge3]

@david-a-jetter yes i just got this issue. When i deploy from the compose the serverless-python-requirements doesn't run but when i run it inside the serverless.yml file it works fine.

[sergoh]

@david-a-jetter We're running into same issue, appreciate you opening issues and talking about work-around 🙏🏽

[calebplum]

Not sure if this is the correct repo to ask this question in, but what's the suggested workflow for using a virtualenv with serverless-python-requirements inside a serverless-compose project?

Prior to using serverless-compose, I'd activate the venv inside my serverless project and that venv had all the correct python packages that my project needed (saved in requirements.txt), so when I ran sls deploy all the python dependencies were packaged correctly.

However, after putting the same serverless project into a serverless-compose structure, the python dependencies are no longer packaged when I deploy the serverless-compose project, even when I run it with the venv activated. I'm not sure if this is because the serverless-compose deployment process is running outside the venv, or because serverless-python-requirements isn't being invoked correctly in the serverless-compose process.