Poetry path dependencies and dockerizePip: true

[MarwanDebbiche]

Hi, I am having issue deploying a service with some poetry path dependencies and the option dockerizePip: true.

Here is my project structure:

repository-root
├── lambdas
│   ├── serverless.yaml
│   ├── handler.py
│   ├── pyproject.toml
│   └── ...
├── project2  # Some other project using common, not using serverless at all
│   └── ...
└── common  # Common libraries
    └── utils
        ├── pyproject.toml
        └── utils
            ├── __init__.py
            └── ...

lambdas/pyproject.toml:

[tool.poetry]
name = "lambdas"
version = "0.1.0"
description = ""

[tool.poetry.dependencies]
python = "^3.8"
pandas = "^1.1.2"
utils = {path = "../common/utils"}

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Note the utils path dependency.

As my project depends on non pure-python libraries (numpy, which is a pandas dependency), I need to use the dockerizePip: true option.

Here is my serverless.yaml:

service: project-lambdas
provider:
  name: aws
  runtime: python3.8

functions:
  hello:
    description: Hello function
    handler: handler.hello

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    usePoetry: true
    dockerizePip: true

But as the dependency installation runs inside of a docker container that does not know about the common directory, it fails with this error:

  Error: STDOUT: 

  STDERR: ERROR: Invalid requirement: '../common/utils' (from line 1 of /var/task/requirements.txt)
  Hint: It looks like a path. File '../common/utils' does not exist.

I tried mounting the common directory inside of the container with dockerRunCmdExtraArgs: ['-v', '${env:PROJECT_ROOT_DIR}/common:/var/common'], but then I get the following error:

  STDERR: ERROR: Exception:
  Traceback (most recent call last):
    File "/var/lang/lib/python3.8/shutil.py", line 788, in move
      os.rename(src, real_dst)
  OSError: [Errno 18] Invalid cross-device link: '/tmp/pip-target-ifpetc_3/lib/python/pytz' -> '/var/task/pytz'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/var/lang/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
      status = self.run(options, args)
    File "/var/lang/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
      return func(self, options, args)
    File "/var/lang/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 470, in run
      self._handle_target_dir(
    File "/var/lang/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 527, in _handle_target_dir
      shutil.move(
    File "/var/lang/lib/python3.8/shutil.py", line 798, in move
      copytree(src, real_dst, copy_function=copy_function,
    File "/var/lang/lib/python3.8/shutil.py", line 554, in copytree
      return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
    File "/var/lang/lib/python3.8/shutil.py", line 510, in _copytree
      raise Error(errors)
  shutil.Error: [('/tmp/pip-target-ifpetc_3/lib/python/pytz/zoneinfo/America/Argentina/Ushuaia', '/var/task/pytz/zoneinfo/America/Argentina/Ushuaia', '[Errno 5] Input/output error')]

When I set dockerizePip to false, the deployement doesn't fail, but of course the lambda invokation does with the error Importing the numpy C-extensions failed.

Is there a way to make poetry path dependencies work with dockerizePip: true?

Thank you for your help

[daampie]

Having the same issue with local libs although I don't think it's related to poetry. Issue seems to be that the common lib is not mounted properly to the container.

Project structure:

├── README.md
├── services
    ├── common
        ├── poetry.lock
        ├── pyproject.toml
        ├── src
            ├── common
                ├── __init__.py
                ├── common.py
    ├── service1
        ├── pyproject.toml
        ├── handler.py
        ├── serverless.yml
    └── ...

services/service1/pyproject.toml

[tool.poetry]
name = "service1"
version = "0.1.0"
description = ""
authors = ["None"]

[tool.poetry.dependencies]
python = "^3.7"
common = { path = "../common/" }

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

services/service1/serverless.yml

service: project

provider:
  name: aws
  runtime: python3.7

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: true
    usePoetry: true

functions:
  project:
    handler: handler.main
    description: handler

Environment information

Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              12.18.3
     Framework Version:         1.51.0
     Plugin Version:            1.3.11
     SDK Version:               2.3.2

services/service1/package.lock

{
  "name": "project",
  "description": "",
  "version": "0.1.0",
  "dependencies": {},
  "devDependencies": {
    "serverless-python-requirements": "^5.1.0"
  }
}

Then when running sls package or sls deploy the following error pops up:

Error --------------------------------------------------

  Error: STDOUT: 

  STDERR: ERROR: Invalid requirement: '../common' (from line 1 of /var/task/requirements.txt)
  Hint: It looks like a path. File '../common' does not exist.
  WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
  You should consider upgrading via the '/var/lang/bin/python3.7 -m pip install --upgrade pip' command.

When I set dockerizePip: false all works fine, but then you run into the issue as described by @MarwanDebbiche

[john-feusi-neuro]

I'm getting the OSError: [Errno 18] Invalid cross-device link error as well. I have usePoetry: false in my serverless file. The problem seems to be intermittent for me.

[john-feusi-neuro]

I turned off "Use gRPC FUSE for file sharing" and I think it fixed the problem for me.

[joelash]

Can confirm that @john-feusi-neuro's fix worked for me, and I'm not using poetry

[gcampionpae]

This trick was a lifesaver, thanks @john-feusi-neuro for finding this, it's been driving me bananas.

[HerrSchroedinger]

DiTo. Here the Tipp from @john-feusi-neuro helped!

[mohoromitch]

Thank you @john-feusi-neuro 💯! This fixed my issue which was only running a pip install within a Docker container.

[balusairam]

How does the problem get solved

[The-Podsiadly]

This error is now persistent on Windows 11 without the ability to turn off gRPC on Docker Desktop 4.8.2. Any suggestions?

[tonyroberts]

I'm having the same problem on Windows. Using dockerRunCmdExtraArgs to mount a folder doesn't work as the folder in the generated requirements.txt file is an absolute Windows path. Maybe running if there was an option to run poetry in docker as well as pip that would help since then mounting the folder should work?

In case this helps anyone else, what I ended up doing was excluding the local packages using the noDeploy option and including them as layers instead.