Python is broken in jsii 1.2.0 when jsii modules are installed as submodules

[eladb]

:bug: Bug Report

Affected Languages

• [x] Python

What is the problem?

cdk8s has an import command which uses jsii to generate python code on-demand and puts it in the user project under imports/<module>.

For example:

cdk8s import k8s

Will create a jsii python module under $project_root/imports/k8s which users then import via:

from imports import k8s

When the cdk8s CLI is compiled with jsii 1.2.0, the generated python code will produce a module that cannot be imported this way and fails with the following error:

Traceback (most recent call last):
  File "__init__.py", line 14, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load("k8s", "0.0.0", "k8s", "k8s@0.0.0.jsii.tgz")
  File "/Users/benisrae/.local/share/virtualenvs/test-sodC6YKc/lib/python3.7/site-packages/jsii/_runtime.py", line 39, in load
    f"{assembly.module}._jsii", assembly.filename
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/resources.py", line 184, in path
    package = _get_package(package)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/resources.py", line 47, in _get_package
    module = import_module(package)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'k8s'

Analysis

The latest jsii submodules feature changed the calls to JSIIAssembly.load. Instead of passing __name__ it passes an jsii explicit module name instead (see PR comment).

In the example above, this means that instead of "imports.k8s" it will pass only "k8s", and then the python loader will not be able to find the tarball resource (because there is no module named k8s).

[eladb]

@RomainMuller I decided not to work around this in cdk8s because it creates a pretty convoluted hack, so we are pinning jsii to 1.1.0 in cdk8s until this is resolved.

[iferca]

I'm having the same issue even with jsii 1.1.0. cdk8s 0.17.0 Tried with python {3.6,3.7}

any hint @eladb ?

$ cdk8s synth
Traceback (most recent call last):
  File "./main.py", line 5, in <module>
    from imports import k8s
  File "/home/iferca/projects/pandas/gcp/gcp-k8s-deployment/imports/k8s/__init__.py", line 14, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load("k8s", "0.0.0", "k8s", "k8s@0.0.0.jsii.tgz")
  File "/home/iferca/.local/share/virtualenvs/gcp-k8s-deployment-po-oI4pI/lib/python3.7/site-packages/jsii/_runtime.py", line 38, in load
    f"{assembly.module}._jsii", assembly.filename
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/lib/python3.7/importlib/resources.py", line 184, in path
    package = _get_package(package)
  File "/usr/lib/python3.7/importlib/resources.py", line 47, in _get_package
    module = import_module(package)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'k8s'
Error: non-zero exit code 1
    at ChildProcess.<anonymous> (/home/iferca/.nvm/versions/node/v12.16.1/lib/node_modules/cdk8s-cli/lib/util.js:17:27)
    at Object.onceWrapper (events.js:418:26)
    at ChildProcess.emit (events.js:311:20)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)

[eladb]

@iferca the generated import in your snippet looks like it was generated by jsii 1.2.0. Are you certain you are using cdk8s 0.17.0?

[iferca]

@eladb from my Pipfie.lock:

"jsii": {
            "hashes": [
                "sha256:30ac1c704641363084bf752d8592a144f4a1cb35f7d03adcb0ae6c80ef6394f1",
                "sha256:77787b0ece3df3b01eec73ca870628b7a9fbc0068315a99ea8098979596e6861"
            ],
            "version": "==1.1.0"
        },
...
"cdk8s": {
            "hashes": [
                "sha256:12c23e0a797756fa318cf582fa67d3998b30afa53a021b29f3619f948a462ed0",
                "sha256:ab3f7ecd6f1b60242b2fa35c6385eb96245e1c8885e22af8e136ce45e69a90ec"
            ],
            "index": "pypi",
            "version": "==0.17.0"
        },

[eladb]

Ok, I think I know what the issue is. The jsii dependency you care about is actually the dependency taken by the CLI, not by your app, and in 0.17.0 we declared the jsii dependency as a caret dependency. This means that if you installed the CLI after jsii@1.2.0 was released, npm/yarn would take 1.2.0.

We need to release a new cdk8s version with the pinned dependency in order to fix this.

[eladb]

Okay, I've verified that indeed this is the case. Releasing cdk8s 0.18.0 which should include the pinned jsii version.

Also, I believe that brew install cdk8s will give you a working version.

[eladb]

Relesae: https://github.com/awslabs/cdk8s/runs/578002435

[eladb]

Okay, cdk8s@0.18.0 has been released. If you reinstall your CLI (npm install -g cdk8s-cli or yarn global add cdk8s-cli), you should be fine.

[cloudfiles-me]

cdk8s@0.18.0 fixed the issue. Thank a lot

[iferca]

@eladb as @jorgehrn mentioned, I can confirm that 0.18.0 can synth :+1:

Thanks