search

pex-tool / pex

A tool for generating .pex (Python EXecutable) files, lock files and venvs.
https://docs.pex-tool.org/
Apache License 2.0
2.53k stars 259 forks source link

The `__pex__` import hook does not handle registering top level PEP-420 packages #1910

Open jsirois opened 2 years ago

jsirois commented 2 years ago

The top-level package currently must have a dunder-init for the mechanism to work.

To see the issue in action 1st:

$ git clone https://github.com/nharada1/lambda-aws-pants-demo
$ cd lambda-aws-pants-demo
$ git reset --hard b7137c0b9c18184f28676d1ad04225542f16edbd
$ ./pants -V

Then:

$ ./pants package ::
14:09:35.21 [INFO] Completed: Building docker image myimage:latest
14:09:35.21 [INFO] Wrote dist/project.inner/lambda.zip
    Runtime: python3.9
    Handler: lambdex_handler.handler
14:09:35.21 [INFO] Built docker image: myimage:latest
Docker image ID: c79ec5f3a7c3
$ container_id=$(docker run --detach --env PEX_VERBOSE=9 -p 9000:8080 myimage:latest)
$ curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
{"errorMessage": "Unable to import module '__pex__.inner.lambda_example': No module named '__pex__.inner'", "errorType": "Runtime.ImportModuleError", "requestId": "6ce393c3-767c-4c4e-8ebd-cb9cfd9d3b17", "stackTrace": []}$
$ docker logs $container_id
17 Sep 2022 21:09:42,190 [INFO] (rapid) exec '/var/runtime/bootstrap' (cwd=/build, handler=)
17 Sep 2022 21:09:46,617 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory
17 Sep 2022 21:09:46,617 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
START RequestId: 6ce393c3-767c-4c4e-8ebd-cb9cfd9d3b17 Version: $LATEST
pex: Laying out PEX zipfile /var/task/lambda.zip
pex: Laying out PEX zipfile /var/task/lambda.zip :: Installing /var/task/lambda.zip to /root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30
pex: Laying out PEX zipfile /var/task/lambda.zip: 23.4ms
pex:   Installing /var/task/lambda.zip to /root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30: 23.0ms
pex: Installed VendorImporter(root='/var/task/lambda.zip/.bootstrap', importables=(_Importable(module='attr', is_pkg=True, path='pex/vendor/_vendored/attrs', prefix='pex.third_party'), _Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: pex.third_party.attr imported via _Loader(module_name='pex.third_party.attr', vendor_module_name='pex.vendor._vendored.attrs.attr')
pex: pex.third_party.packaging imported via _Loader(module_name='pex.third_party.packaging', vendor_module_name='pex.vendor._vendored.packaging.packaging')
pex: pex.third_party.packaging.utils imported via _Loader(module_name='pex.third_party.packaging.utils', vendor_module_name='pex.vendor._vendored.packaging.packaging.utils')
pex: pex.third_party.pyparsing imported via _Loader(module_name='pex.third_party.pyparsing', vendor_module_name='pex.vendor._vendored.packaging.pyparsing')
pex: pex.third_party.packaging.markers imported via _Loader(module_name='pex.third_party.packaging.markers', vendor_module_name='pex.vendor._vendored.packaging.packaging.markers')
pex: pex.third_party.packaging.requirements imported via _Loader(module_name='pex.third_party.packaging.requirements', vendor_module_name='pex.vendor._vendored.packaging.packaging.requirements')
pex: pex.third_party.packaging.specifiers imported via _Loader(module_name='pex.third_party.packaging.specifiers', vendor_module_name='pex.vendor._vendored.packaging.packaging.specifiers')
pex: pex.third_party.packaging.tags imported via _Loader(module_name='pex.third_party.packaging.tags', vendor_module_name='pex.vendor._vendored.packaging.packaging.tags')
pex: Installed VendorImporter(root='/root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30', importables=(_Importable(module='__main__', is_pkg=False, path='.', prefix='__pex__'), _Importable(module='lambdex_handler', is_pkg=False, path='.', prefix='__pex__'), _Importable(module='__pex__', is_pkg=True, path='.', prefix='__pex__')))
pex: Searching for pyenv root...
pex: No pyenv installation was found.
pex: Searching for pyenv root...: 0.1ms
pex: Found site-library: /var/lang/lib/python3.9/site-packages
pex: Found .pth file: /var/lang/lib/python3.9/site-packages/distutils-precedence.pth
pex: Not a tainted path element: /var/task/lambda.zip/.bootstrap
pex: Not a tainted path element: /var/task
pex: Not a tainted path element: /var/runtime
pex: Not a tainted path element: /var/task/lambda.zip
pex: Not a tainted path element: /var/lang/lib/python39.zip
pex: Not a tainted path element: /var/lang/lib/python3.9
pex: Not a tainted path element: /var/lang/lib/python3.9/lib-dynload
pex: Tainted path element: /var/lang/lib/python3.9/site-packages
pex: Scrubbing from user site: /root/.local/lib/python3.9/site-packages
pex: Scrubbing from site-packages: /var/lang/lib/python3.9/site-packages
pex: New sys.path: ['/var/task/lambda.zip/.bootstrap', '/var/task', '/var/runtime', '/var/task/lambda.zip', '/var/lang/lib/python39.zip', '/var/lang/lib/python3.9', '/var/lang/lib/python3.9/lib-dynload']
pex: Activating PEX virtual environment from /root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30
pex: Adding pex environment to the head of sys.path: /root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30
pex: Activating PEX virtual environment from /root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30 :: Searching dependency cache: /root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30/.deps
pex: Activating PEX virtual environment from /root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30: 0.2ms
pex:   Searching dependency cache: /root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30/.deps: 0.0ms
Traceback (most recent call last): Unable to import module '__pex__.inner.lambda_example': No module named '__pex__.inner'
END RequestId: 6ce393c3-767c-4c4e-8ebd-cb9cfd9d3b17
REPORT RequestId: 6ce393c3-767c-4c4e-8ebd-cb9cfd9d3b17  Init Duration: 0.15 ms  Duration: 300.76 ms     Billed Duration: 301 ms Memory Size: 3008 MB    Max Memory Used: 3008 MB
$ docker kill $container_id
7795f2572083b8789c500678f7f00113d558b99b0ec7d056030570f2bf1df0b8
$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

And the fix is just to add a dunder-init:

$ touch project/inner/__init__.py
$ ./pants package ::
14:11:24.50 [INFO] Canceled: Building docker image myimage:latest
14:11:24.54 [INFO] Completed: Building docker image myimage:latest
14:11:24.54 [INFO] Wrote dist/project.inner/lambda.zip
    Runtime: python3.9
    Handler: lambdex_handler.handler
14:11:24.54 [INFO] Built docker image: myimage:latest
Docker image ID: fd7014075b07
$ container_id=$(docker run --detach --env PEX_VERBOSE=9 -p 9000:8080 myimage:latest)
$ curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
null$
$ docker logs $container_id
17 Sep 2022 21:11:30,279 [INFO] (rapid) exec '/var/runtime/bootstrap' (cwd=/build, handler=)
17 Sep 2022 21:11:33,079 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory
17 Sep 2022 21:11:33,079 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
START RequestId: bfe841cb-c2b8-49c1-aefc-e1e2d98e931a Version: $LATEST
pex: Laying out PEX zipfile /var/task/lambda.zip
pex: Laying out PEX zipfile /var/task/lambda.zip :: Installing /var/task/lambda.zip to /root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96
pex: Laying out PEX zipfile /var/task/lambda.zip: 22.9ms
pex:   Installing /var/task/lambda.zip to /root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96: 22.6ms
pex: Installed VendorImporter(root='/var/task/lambda.zip/.bootstrap', importables=(_Importable(module='attr', is_pkg=True, path='pex/vendor/_vendored/attrs', prefix='pex.third_party'), _Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: pex.third_party.attr imported via _Loader(module_name='pex.third_party.attr', vendor_module_name='pex.vendor._vendored.attrs.attr')
pex: pex.third_party.packaging imported via _Loader(module_name='pex.third_party.packaging', vendor_module_name='pex.vendor._vendored.packaging.packaging')
pex: pex.third_party.packaging.utils imported via _Loader(module_name='pex.third_party.packaging.utils', vendor_module_name='pex.vendor._vendored.packaging.packaging.utils')
pex: pex.third_party.pyparsing imported via _Loader(module_name='pex.third_party.pyparsing', vendor_module_name='pex.vendor._vendored.packaging.pyparsing')
pex: pex.third_party.packaging.markers imported via _Loader(module_name='pex.third_party.packaging.markers', vendor_module_name='pex.vendor._vendored.packaging.packaging.markers')
pex: pex.third_party.packaging.requirements imported via _Loader(module_name='pex.third_party.packaging.requirements', vendor_module_name='pex.vendor._vendored.packaging.packaging.requirements')
pex: pex.third_party.packaging.specifiers imported via _Loader(module_name='pex.third_party.packaging.specifiers', vendor_module_name='pex.vendor._vendored.packaging.packaging.specifiers')
pex: pex.third_party.packaging.tags imported via _Loader(module_name='pex.third_party.packaging.tags', vendor_module_name='pex.vendor._vendored.packaging.packaging.tags')
pex: Installed VendorImporter(root='/root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96', importables=(_Importable(module='__main__', is_pkg=False, path='.', prefix='__pex__'), _Importable(module='lambdex_handler', is_pkg=False, path='.', prefix='__pex__'), _Importable(module='__pex__', is_pkg=True, path='.', prefix='__pex__'), _Importable(module='inner', is_pkg=True, path='.', prefix='__pex__')))
pex: Searching for pyenv root...
pex: No pyenv installation was found.
pex: Searching for pyenv root...: 0.1ms
pex: Found site-library: /var/lang/lib/python3.9/site-packages
pex: Found .pth file: /var/lang/lib/python3.9/site-packages/distutils-precedence.pth
pex: Not a tainted path element: /var/task/lambda.zip/.bootstrap
pex: Not a tainted path element: /var/task
pex: Not a tainted path element: /var/runtime
pex: Not a tainted path element: /var/task/lambda.zip
pex: Not a tainted path element: /var/lang/lib/python39.zip
pex: Not a tainted path element: /var/lang/lib/python3.9
pex: Not a tainted path element: /var/lang/lib/python3.9/lib-dynload
pex: Tainted path element: /var/lang/lib/python3.9/site-packages
pex: Scrubbing from user site: /root/.local/lib/python3.9/site-packages
pex: Scrubbing from site-packages: /var/lang/lib/python3.9/site-packages
pex: New sys.path: ['/var/task/lambda.zip/.bootstrap', '/var/task', '/var/runtime', '/var/task/lambda.zip', '/var/lang/lib/python39.zip', '/var/lang/lib/python3.9', '/var/lang/lib/python3.9/lib-dynload']
pex: Activating PEX virtual environment from /root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96
pex: Adding pex environment to the head of sys.path: /root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96
pex: Activating PEX virtual environment from /root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96 :: Searching dependency cache: /root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96/.deps
pex: Activating PEX virtual environment from /root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96: 0.2ms
pex:   Searching dependency cache: /root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96/.deps: 0.0ms
pex: __pex__.inner imported via _Loader(module_name='__pex__.inner', vendor_module_name='inner')
pex: __pex__.inner.lambda_example imported via _Loader(module_name='__pex__.inner.lambda_example', vendor_module_name='inner.lambda_example')
Hello AWS!
END RequestId: bfe841cb-c2b8-49c1-aefc-e1e2d98e931a
REPORT RequestId: bfe841cb-c2b8-49c1-aefc-e1e2d98e931a  Init Duration: 0.07 ms  Duration: 303.61 ms     Billed Duration: 304 ms Memory Size: 3008 MB    Max Memory Used: 3008 MB
$ docker kill $container_id
6663f1cf6241cf9b8eb52c9c23d9104b8b0bf4d20050eb94507481b9fe5daa24
$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
$

The key difference is before not "seeing" the inner package:

pex: Installed VendorImporter(root='/root/.pex/unzipped_pexes/c9800bff4cd62be0482f8df9305810e762d41d30', importables=(_Importable(module='__main__', is_pkg=False, path='.', prefix='__pex__'), _Importable(module='lambdex_handler', is_pkg=False, path='.', prefix='__pex__'), _Importable(module='__pex__', is_pkg=True, path='.', prefix='__pex__')))

And after, "seeing" it:

pex: Installed VendorImporter(root='/root/.pex/unzipped_pexes/23cc7e854f5364c1dfd7c85262fba384f7e7cd96', importables=(_Importable(module='__main__', is_pkg=False, path='.', prefix='__pex__'), _Importable(module='lambdex_handler', is_pkg=False, path='.', prefix='__pex__'), _Importable(module='__pex__', is_pkg=True, path='.', prefix='__pex__'), _Importable(module='inner', is_pkg=True, path='.', prefix='__pex__')))
jsirois commented 2 years ago

The vendor importer is installed here: https://github.com/pantsbuild/pex/blob/98c2640602ed3d2820f8e4f66dadc6c59df120dd/pex/pex_bootstrapper.py#L577-L587

And that leads to the issue via: https://github.com/pantsbuild/pex/blob/98c2640602ed3d2820f8e4f66dadc6c59df120dd/pex/third_party/__init__.py#L262-L279 https://github.com/pantsbuild/pex/blob/98c2640602ed3d2820f8e4f66dadc6c59df120dd/pex/third_party/__init__.py#L167-L175

The two module iterator implementations both assume packages have dunder-inits in them: https://github.com/pantsbuild/pex/blob/98c2640602ed3d2820f8e4f66dadc6c59df120dd/pex/third_party/__init__.py#L82-L93 https://github.com/pantsbuild/pex/blob/98c2640602ed3d2820f8e4f66dadc6c59df120dd/pex/third_party/__init__.py#L106-L129