JsiiErrors / EAGAIN when using node>=13.2 with Python, Java & .NET

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code

https://aws.amazon.com/cdk

Apache License 2.0

11.62k stars 3.91k forks source link

JsiiErrors / EAGAIN when using node>=13.2 with Python, Java & .NET #5187

Closed DrLuke closed 4 years ago

DrLuke commented 4 years ago

When deploying, some file is temporarily unavailable.

Reproduction Steps

from aws_cdk import (
    core
)

class Stack(core.Stack):
    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

app = core.App()
stack = Stack(app, "Test")

app.synth()

Then deploy

$ cdk deploy --app "python stack.py"

Error Log

$ cdk deploy --app "python stack.py"
internal/fs/utils.js:220
    throw err;
    ^

Error: EAGAIN: resource temporarily unavailable, read
    at Object.readSync (fs.js:516:3)
    at SyncStdio.readLine (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13322:25)
    at InputOutput.read (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13272:34)
    at KernelHost.run (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7228:32)
    at Immediate.<anonymous> (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7236:37)
    at processImmediate (internal/timers.js:439:21) {
  errno: -11,
  syscall: 'read',
  code: 'EAGAIN'
}
Traceback (most recent call last):
  File "stack_bugreport.py", line 1, in <module>
    from aws_cdk import (
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/aws_apigateway/__init__.py", line 548, in <module>
    import aws_cdk.aws_certificatemanager
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/aws_certificatemanager/__init__.py", line 104, in <module>
    import aws_cdk.aws_cloudformation
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/aws_cloudformation/__init__.py", line 125, in <module>
    import aws_cdk.aws_iam
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/aws_iam/__init__.py", line 169, in <module>
    import aws_cdk.region_info
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/region_info/__init__.py", line 113, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load("@aws-cdk/region-info", "1.18.0", __name__, "region-info@1.18.0.jsii.tgz")
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_runtime.py", line 40, in load
    _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 209, in load
    self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 330, in load
    return self._process.send(request, LoadResponse)
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 310, in send
    self._next_message(), _ProcessResponse_R
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 261, in _next_message
    return json.loads(self._process.stdout.readline(), object_hook=ohook)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Subprocess exited with error 1

Workaround for the time being

Downgrade node to version 12.13.1

Environment

CLI Version : 1.18.0 (build bc924bc)
Framework Version: 1.18.0
OS : Linux
Language : Python

This is :bug: Bug Report

shivlaks commented 4 years ago

@DrLuke - Can you do a cdk ls from the directory with your cdk.json file to list all the stacks? You can provide those stacks in your cdk deploy [STACKS] to deploy the stack(s) named STACKS or cdk deploy "*" to deploy all stacks

DrLuke commented 4 years ago

I tried, but I get the same error message.

shivlaks commented 4 years ago

@DrLuke - can you share the output with a -v added to the cdk CLI commands you're running?

DrLuke commented 4 years ago

@shivlaks Like this?

$ cdk deploy "*" -v
CDK toolkit version: 1.18.0 (build bc924bc)
Command line arguments: {
  _: [ 'deploy' ],
  v: true,
  verbose: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: false,
  execute: true,
  '$0': 'cdk',
  STACKS: [ '*' ],
  stacks: [ '*' ]
}
Determining whether we're on an EC2 instance.
Does not look like EC2 instance.
cdk.json: {
  "app": "python3 stack.py"
}
cdk.context.json: {
  "@aws-cdk/core:enableStackNameDuplicates": "true"
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python3 stack.py',
  context: {},
  tags: [],
  assetMetadata: true,
  toolkitBucket: {},
  staging: true
}
Setting "CDK_DEFAULT_REGION" environment variable to eu-central-1
Resolving default credentials
Retrieved account ID xxx from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to xxx
context: {
  '@aws-cdk/core:enableStackNameDuplicates': 'true',
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'eu-central-1',
  CDK_DEFAULT_ACCOUNT: 'xxx',
  CDK_CONTEXT_JSON: '{"@aws-cdk/core:enableStackNameDuplicates":"true","aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '1.16.0',
  CDK_CLI_VERSION: '1.18.0'
}
internal/fs/utils.js:220
    throw err;
    ^

Error: EAGAIN: resource temporarily unavailable, read
    at Object.readSync (fs.js:516:3)
    at SyncStdio.readLine (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13322:25)
    at InputOutput.read (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13272:34)
    at KernelHost.run (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7228:32)
    at Immediate.<anonymous> (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7236:37)
    at processImmediate (internal/timers.js:439:21) {
  errno: -11,
  syscall: 'read',
  code: 'EAGAIN'
}
Traceback (most recent call last):
  File "stack.py", line 9, in <module>
    app = core.App()
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_runtime.py", line 66, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/core/__init__.py", line 3443, in __init__
    jsii.create(App, self, [props])
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 223, in create
    interfaces=[iface.__jsii_type__ for iface in getattr(klass, "__jsii_ifaces__", [])],
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 333, in create
    return self._process.send(request, CreateResponse)
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 310, in send
    self._next_message(), _ProcessResponse_R
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 261, in _next_message
    return json.loads(self._process.stdout.readline(), object_hook=ohook)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/usr/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:115:23)
    at ChildProcess.emit (events.js:210:5)
    at ChildProcess.EventEmitter.emit (domain.js:478:20)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:270:12)

$ cdk ls -v
CDK toolkit version: 1.18.0 (build bc924bc)
Command line arguments: {
  _: [ 'ls' ],
  v: true,
  verbose: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  long: false,
  l: false,
  '$0': 'cdk'
}
Determining whether we're on an EC2 instance.
cdk.json: {
  "app": "python3 stack.py"
}
Does not look like EC2 instance.
cdk.context.json: {
  "@aws-cdk/core:enableStackNameDuplicates": "true"
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python3 stack.py',
  context: {},
  tags: [],
  assetMetadata: true,
  toolkitBucket: {},
  staging: true
}
Setting "CDK_DEFAULT_REGION" environment variable to eu-central-1
Resolving default credentials
Retrieved account ID xxx from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to xxx
context: {
  '@aws-cdk/core:enableStackNameDuplicates': 'true',
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'eu-central-1',
  CDK_DEFAULT_ACCOUNT: 'xxx',
  CDK_CONTEXT_JSON: '{"@aws-cdk/core:enableStackNameDuplicates":"true","aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '1.16.0',
  CDK_CLI_VERSION: '1.18.0'
}
internal/fs/utils.js:220
    throw err;
    ^

Error: EAGAIN: resource temporarily unavailable, read
    at Object.readSync (fs.js:516:3)
    at SyncStdio.readLine (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13322:25)
    at InputOutput.read (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13272:34)
    at KernelHost.run (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7228:32)
    at Immediate.<anonymous> (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7236:37)
    at processImmediate (internal/timers.js:439:21) {
  errno: -11,
  syscall: 'read',
  code: 'EAGAIN'
}
Traceback (most recent call last):
  File "stack.py", line 9, in <module>
    app = core.App()
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_runtime.py", line 66, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/core/__init__.py", line 3443, in __init__
    jsii.create(App, self, [props])
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 223, in create
    interfaces=[iface.__jsii_type__ for iface in getattr(klass, "__jsii_ifaces__", [])],
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 333, in create
    return self._process.send(request, CreateResponse)
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 310, in send
    self._next_message(), _ProcessResponse_R
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 261, in _next_message
    return json.loads(self._process.stdout.readline(), object_hook=ohook)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/usr/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:115:23)
    at ChildProcess.emit (events.js:210:5)
    at ChildProcess.EventEmitter.emit (domain.js:478:20)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:270:12)

DrLuke commented 4 years ago

This happened to me with a project that has already been deployed successfully multiple times before upgrading to 1.18.0, any cdk command gives me the same error message.

shivlaks commented 4 years ago

@DrLuke - gotcha that's good to know, what version was the app created with? I'd like to get full reproduction steps so I can dig deeper

DrLuke commented 4 years ago

@shivlaks Probably around 1.14.0, but I didn't use cdk init to create it. I manually created the stack file and always deployed with cdk deploy --app "python stack.py".

shivlaks commented 4 years ago

@DrLuke - got it. Does your cdk.out folder contain the template? Looks like the error is at the point where it'd read the existing stack template

DrLuke commented 4 years ago

No, it's empty

shivlaks commented 4 years ago

@DrLuke - can you run cdk synth successfully?

DrLuke commented 4 years ago

Nope

$ cdk synth "*" -v
CDK toolkit version: 1.18.0 (build bc924bc)
Command line arguments: {
  _: [ 'synth' ],
  v: true,
  verbose: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  '$0': 'cdk',
  STACKS: [ '*' ],
  stacks: [ '*' ]
}
Determining whether we're on an EC2 instance.
cdk.json: {
  "app": "python3 stack.py"
}
Does not look like EC2 instance.
cdk.context.json: {
  "@aws-cdk/core:enableStackNameDuplicates": "true"
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python3 stack.py',
  context: {},
  tags: [],
  assetMetadata: true,
  toolkitBucket: {},
  staging: true
}
Setting "CDK_DEFAULT_REGION" environment variable to eu-central-1
Resolving default credentials
Retrieved account ID xxx from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to xxx
context: {
  '@aws-cdk/core:enableStackNameDuplicates': 'true',
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'eu-central-1',
  CDK_DEFAULT_ACCOUNT: 'xxx',
  CDK_CONTEXT_JSON: '{"@aws-cdk/core:enableStackNameDuplicates":"true","aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '1.16.0',
  CDK_CLI_VERSION: '1.18.0'
}
internal/fs/utils.js:220
    throw err;
    ^

Error: EAGAIN: resource temporarily unavailable, read
    at Object.readSync (fs.js:516:3)
    at SyncStdio.readLine (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13322:25)
    at InputOutput.read (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13272:34)
    at KernelHost.run (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7228:32)
    at Immediate.<anonymous> (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7236:37)
    at processImmediate (internal/timers.js:439:21) {
  errno: -11,
  syscall: 'read',
  code: 'EAGAIN'
}
Traceback (most recent call last):
  File "stack.py", line 9, in <module>
    app = core.App()
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_runtime.py", line 66, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/core/__init__.py", line 3443, in __init__
    jsii.create(App, self, [props])
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 223, in create
    interfaces=[iface.__jsii_type__ for iface in getattr(klass, "__jsii_ifaces__", [])],
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 333, in create
    return self._process.send(request, CreateResponse)
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 310, in send
    self._next_message(), _ProcessResponse_R
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 261, in _next_message
    return json.loads(self._process.stdout.readline(), object_hook=ohook)
  File "/usr/lib64/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/usr/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:115:23)
    at ChildProcess.emit (events.js:210:5)
    at ChildProcess.EventEmitter.emit (domain.js:478:20)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:270:12)

shivlaks commented 4 years ago

@DrLuke - seems like a bug. we'll investigate further. really appreciate the additional information. It should help us get to the root cause sooner

DrLuke commented 4 years ago

Thank you! Let me know if you need any further details.

eladb commented 4 years ago

Looks like a Jsii runtime issue. @RomainMuller can you take a look?

RomainMuller commented 4 years ago

Looks like the jsii Kernel (JS) gets EAGAIN trying time read from its own STDIN.

It might help to know the version of node that you re using, too (node --version)

RomainMuller commented 4 years ago

So I have tried running a reproduction and haven't managed to hit the issue to this point... In order to avoid us going back and forth asking for additional information (you already provided a lot, and that's great!), let me ask for all the stuff that comes to mind now and could play a role in this problem you're having:

What node runtime are you using? (node --version)
What Python runtime are you using? (python --version)
What (all) versions of Python modules are there? (pip list)
Possible behavior-affecting environment variables? (env | grep NODE - which may include personal information that is irrelevant to our issue & that you should feel free to redact as needed)
Are you running in a memory-constrained environment? (cat /proc/meminfo | grep Mem)
Run your application with the JSII_DEBUG=1 environment variable, and provide the resulting trace (in a gist or pastebin if needed). There as well you should feel free to redact any information present there that you do not wish to share -- simply make sure you preserve the structure of the data (don't remove entire lines, only the specific data that needs to remain private).

That's all I could think of at this point which could contribute to causing your issues... Hopefully this helps give us a better understanding of what is happening.

skatsuta commented 4 years ago

I encountered the same error. Here are my reproduction steps and information on my environment.

Reproduction steps

$ mkdir -p cdk_error && cd $_

$ cdk init app --language=python
Applying project template app for python
Initializing a new git repository...
Executing Creating virtualenv...
(...snip...)
Enjoy!

$ source .env/bin/activate
(.env)

$ pip install -r requirements.txt
(...snip...)
Successfully installed attrs-19.3.0 aws-cdk.core-1.18.0 aws-cdk.cx-api-1.18.0 cattrs-0.9.0 cdk-error jsii-0.20.8 publication-0.0.3 python-dateutil-2.8.1 six-1.13.0 typing-extensions-3.7.4.1
(.env)

$ JSII_DEBUG=1 cdk synth -v &> error.log
(.env)
$ cat error.log
CDK toolkit version: 1.18.0 (build bc924bc)
Command line arguments: {
  _: [ 'synth' ],
  v: true,
  verbose: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  '$0': '/usr/local/bin/cdk'
}
Determining whether we're on an EC2 instance.
Does not look like EC2 instance.
cdk.json: {
  "app": "python3 app.py"
}
cdk.context.json: {
  "@aws-cdk/core:enableStackNameDuplicates": "true"
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python3 app.py',
  context: {},
  tags: [],
  assetMetadata: true,
  toolkitBucket: {},
  staging: true
}
Setting "CDK_DEFAULT_REGION" environment variable to ap-northeast-1
Resolving default credentials
Retrieved account ID XXXXXXXXXXXX from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to XXXXXXXXXXXX
context: {
  '@aws-cdk/core:enableStackNameDuplicates': 'true',
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'ap-northeast-1',
  CDK_DEFAULT_ACCOUNT: 'XXXXXXXXXXXX',
  CDK_CONTEXT_JSON: '{"@aws-cdk/core:enableStackNameDuplicates":"true","aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '1.16.0',
  CDK_CLI_VERSION: '1.18.0'
}
> {"name":"@aws-cdk/cx-api","version":"1.18.0","tarball":"/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/cx_api/_jsii/cx-api@1.18.0.jsii.tgz","api":"load"}
[jsii-kernel] load {
  name: '@aws-cdk/cx-api',
  version: '1.18.0',
  tarball: '/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/cx_api/_jsii/cx-api@1.18.0.jsii.tgz',
  api: 'load'
}
[jsii-kernel] creating jsii-kernel modules workdir: /var/folders/3x/vrvl2bs171l55csv_l2zb1hh0000gn/T/jsii-kernel-NvSBHH
[jsii-kernel] removing staging directory: /var/folders/3x/vrvl2bs171l55csv_l2zb1hh0000gn/T/jsii-kernel-install-staging-c3OO11
< {"ok":{"assembly":"@aws-cdk/cx-api","types":29}}
> {"name":"@aws-cdk/core","version":"1.18.0","tarball":"/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/core/_jsii/core@1.18.0.jsii.tgz","api":"load"}
[jsii-kernel] load {
  name: '@aws-cdk/core',
  version: '1.18.0',
  tarball: '/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/core/_jsii/core@1.18.0.jsii.tgz',
  api: 'load'
}
[jsii-kernel] removing staging directory: /var/folders/3x/vrvl2bs171l55csv_l2zb1hh0000gn/T/jsii-kernel-install-staging-BZtDkO
< {"ok":{"assembly":"@aws-cdk/core","types":108}}
[jsii-kernel] removing install dir /var/folders/3x/vrvl2bs171l55csv_l2zb1hh0000gn/T/jsii-kernel-NvSBHH
internal/fs/utils.js:220
    throw err;
    ^

Error: EAGAIN: resource temporarily unavailable, read
    at Object.readSync (fs.js:516:3)
    at SyncStdio.readLine (/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13322:25)
    at InputOutput.read (/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13272:34)
    at KernelHost.run (/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7228:32)
    at Immediate.<anonymous> (/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7236:37)
    at processImmediate (internal/timers.js:439:21) {
  errno: -35,
  syscall: 'read',
  code: 'EAGAIN'
}
Traceback (most recent call last):
  File "app.py", line 8, in <module>
    app = core.App()
  File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_runtime.py", line 66, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/core/__init__.py", line 3443, in __init__
    jsii.create(App, self, [props])
  File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 223, in create
    interfaces=[iface.__jsii_type__ for iface in getattr(klass, "__jsii_ifaces__", [])],
  File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 333, in create
    return self._process.send(request, CreateResponse)
  File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 310, in send
    self._next_message(), _ProcessResponse_R
  File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 261, in _next_message
    return json.loads(self._process.stdout.readline(), object_hook=ohook)
  File "/Users/skatsuta/.pyenv/versions/anaconda3-2019.03/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/Users/skatsuta/.pyenv/versions/anaconda3-2019.03/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/skatsuta/.pyenv/versions/anaconda3-2019.03/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/usr/local/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:115:23)
    at ChildProcess.emit (events.js:210:5)
    at ChildProcess.EventEmitter.emit (domain.js:478:20)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:270:12)
(.env)

Our AWS account ID is masked here for security reason, but actually the correct ID is set. Attached error.log for your investigation: error.log

Environment

# OS
$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.13.6
BuildVersion:   17G9016

# AWS CDK
$ cdk --version
1.18.0 (build bc924bc)

# Node
$ node --version
v13.2.0

# Python
$ python --version
Python 3.7.3

# Python packages
$ pip list
Package           Version Location
----------------- ------- ---------------------------------------
attrs             19.3.0
aws-cdk.core      1.18.0
aws-cdk.cx-api    1.18.0
cattrs            0.9.0
cdk-error         0.0.1   /Users/skatsuta/tmp/cdk_error/cdk_error
jsii              0.20.8
pip               19.0.3
publication       0.0.3
python-dateutil   2.8.1
setuptools        40.8.0
six               1.13.0
typing-extensions 3.7.4.1
(.env)

# Env vars related to Node
$ env | grep NODE
(.env)

# Memory
# I'm using macOS, so I show the output of vmstat and top instead of `cat /proc/meminfo`
$ vm_stat
Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                               95560.
Pages active:                           1512991.
Pages inactive:                         1138637.
Pages speculative:                       142651.
Pages throttled:                              0.
Pages wired down:                       1067127.
Pages purgeable:                          50024.
"Translation faults":                1267941904.
Pages copy-on-write:                   45647892.
Pages zero filled:                    608317362.
Pages reactivated:                     68208285.
Pages purged:                          29002867.
File-backed pages:                       529691.
Anonymous pages:                        2264588.
Pages stored in compressor:             6440947.
Pages occupied by compressor:            237143.
Decompressions:                        98763361.
Compressions:                         132396336.
Pageins:                               58923959.
Pageouts:                                335282.
Swapins:                              136352534.
Swapouts:                             141541295.
$ top -l 1 -s 0 | grep PhysMem
PhysMem: 15G used (4258M wired), 711M unused.

Thank you for your sincere support. Hope it helps!

DrLuke commented 4 years ago

@RomainMuller

$ node --version
v13.2.0

$ python3 --version
Python 3.8.0

$ python3.7 --version
Python 3.7.5

$ pip list
Package           Version
----------------- -------
attrs             19.3.0 
aws-cdk.core      1.18.0 
aws-cdk.cx-api    1.18.0 
cattrs            0.9.0  
jsii              0.20.8 
pip               19.2.3 
publication       0.0.3  
python-dateutil   2.8.1  
setuptools        41.2.0 
six               1.13.0 
typing-extensions 3.7.4.1

$ env | grep NODE

$ cat /proc/meminfo | grep Mem
MemTotal:       16155612 kB
MemFree:         1661912 kB
MemAvailable:    4460092 kB

$ JSII_DEBUG=1 cdk deploy "*" -v
CDK toolkit version: 1.18.0 (build bc924bc)
Command line arguments: {
  _: [ 'deploy' ],
  v: true,
  verbose: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: false,
  execute: true,
  '$0': 'cdk',
  STACKS: [ '*' ],
  stacks: [ '*' ]
}
Determining whether we're on an EC2 instance.
cdk.json: {
  "app": "python3 stack.py"
}
Does not look like EC2 instance.
cdk.context.json: {
  "@aws-cdk/core:enableStackNameDuplicates": "true"
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python3 stack.py',
  context: {},
  tags: [],
  assetMetadata: true,
  toolkitBucket: {},
  staging: true
}
Setting "CDK_DEFAULT_REGION" environment variable to eu-central-1
Resolving default credentials
Retrieved account ID xxx from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to xxx
context: {
  '@aws-cdk/core:enableStackNameDuplicates': 'true',
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'eu-central-1',
  CDK_DEFAULT_ACCOUNT: 'xxx',
  CDK_CONTEXT_JSON: '{"@aws-cdk/core:enableStackNameDuplicates":"true","aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '1.16.0',
  CDK_CLI_VERSION: '1.18.0'
}
> {"name":"@aws-cdk/cx-api","version":"1.18.0","tarball":"/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/cx_api/_jsii/cx-api@1.18.0.jsii.tgz","api":"load"}
[jsii-kernel] load {
  name: '@aws-cdk/cx-api',
  version: '1.18.0',
  tarball: '/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/cx_api/_jsii/cx-api@1.18.0.jsii.tgz',
  api: 'load'
}
[jsii-kernel] creating jsii-kernel modules workdir: /tmp/jsii-kernel-EXT00j
[jsii-kernel] removing staging directory: /tmp/jsii-kernel-install-staging-dUQgRk
< {"ok":{"assembly":"@aws-cdk/cx-api","types":29}}
> {"name":"@aws-cdk/core","version":"1.18.0","tarball":"/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/core/_jsii/core@1.18.0.jsii.tgz","api":"load"}
[jsii-kernel] load {
  name: '@aws-cdk/core',
  version: '1.18.0',
  tarball: '/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/core/_jsii/core@1.18.0.jsii.tgz',
  api: 'load'
}
[jsii-kernel] removing staging directory: /tmp/jsii-kernel-install-staging-pYCVwk
< {"ok":{"assembly":"@aws-cdk/core","types":108}}
[jsii-kernel] removing install dir /tmp/jsii-kernel-EXT00j
internal/fs/utils.js:220
    throw err;
    ^

Error: EAGAIN: resource temporarily unavailable, read
    at Object.readSync (fs.js:516:3)
    at SyncStdio.readLine (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13322:25)
    at InputOutput.read (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13272:34)
    at KernelHost.run (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7228:32)
    at Immediate.<anonymous> (/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7236:37)
    at processImmediate (internal/timers.js:439:21) {
  errno: -11,
  syscall: 'read',
  code: 'EAGAIN'
}
Traceback (most recent call last):
  File "stack.py", line 9, in <module>
    app = core.App()
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_runtime.py", line 66, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/aws_cdk/core/__init__.py", line 3443, in __init__
    jsii.create(App, self, [props])
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 223, in create
    interfaces=[iface.__jsii_type__ for iface in getattr(klass, "__jsii_ifaces__", [])],
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 333, in create
    return self._process.send(request, CreateResponse)
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 310, in send
    self._next_message(), _ProcessResponse_R
  File "/home/drluke/work/di-client-api-message-dispatcher/venv/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 261, in _next_message
    return json.loads(self._process.stdout.readline(), object_hook=ohook)
  File "/usr/lib64/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/usr/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:115:23)
    at ChildProcess.emit (events.js:210:5)
    at ChildProcess.EventEmitter.emit (domain.js:478:20)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:270:12)

pepastach commented 4 years ago

I am having the same issue. Tried downgrading CDK dependencies to 1.16.3, but it still failed with 1.17.1 cli and latest node. Having downgraded node to 12.13.1 helped me. Hope it helps someone.

maskerade commented 4 years ago

Had exactly the same issue (on 2 different machines, local & CI/CD). Tried downgrading CDK & dependencies to previous version but still saw the same issue. As suggested by @pepastach downgrading node to v12.13.1 resolved the issue

DrLuke commented 4 years ago

I have prepared a Dockerfile which reproduces the issue.

cdk_EAGAIN_reproduction.zip

In the unzipped directory run:

docker image build -t cdk_bugreport:latest .
docker run -i -t cdk_bugreport:latest
source venv/bin/activate
cdk synth

(Warning, python3.7 has to be compiled when building the image, might take a while)

I hope this helps to narrow down the issue.

DrLuke commented 4 years ago

Using the Dockerimage I can confirm that downgrading node to version 12.13.1 is a workaround. I've added it to the first post in the issue.

pd5rm commented 4 years ago

FYI, nodejs v13.1.0 works as well, but 13.2.0 fails.

elvinx commented 4 years ago

having the same issue with CDK 1.18, Python 3.7 and nodejs v13.2.0 while doing cdk synth

RomainMuller commented 4 years ago

Okay - so we have a smoking gun - it appears that there is a behavior change in node 13.2 that causes the symptoms. We have to investigate deeper to understand whether node 13.2 reveals a latent bug in our code, or whether we should report a bug to them.

netroy commented 4 years ago

What worked for me on nodejs v13.2.0 was to catch EAGAIN in SyncStdio#readLine, and return this.readLine(). This is a hacky workaround, as it simply ignores the EAGAIN.

This also made me wonder if SyncStdio can be completely replaced by nodejs's readline module? SyncStdio seems to be implementing the same functionality that's available natively in nodejs.

eladb commented 4 years ago

Could this be related? https://github.com/nodejs/node/pull/30162

RomainMuller commented 4 years ago

Seems like the STDIN for node processes is opening with O_NONBLOCK, which means we may get EAGAIN if trying to perform synchronous operations on it. The behavior doesn't seem to have changed recently (looked at libuv's changelog as well as node's and found nothing super interesting there).

@netroy Using readline would result in an asynchronous API, and as the name implies, SyncStdio is meant to provide a synchronous API here... I'm currently investigating alternatives.

machielg commented 4 years ago

I can confirm downgrading nodejs from 13.2.0 to 13.1.0 fixes this problem.

jarikujansuu commented 4 years ago

I could fix one project with using by switching Node version. But newer project refuses to work. And noticed that even my requirements.txt currently is

-e .

aws_cdk.aws_ecr~=1.16.3
aws_cdk.aws_s3~=1.16.3
aws_cdk.aws_iam~=1.16.3

I get following installations, which not so surprisingly say they are incompatible. So mixture of 1.16.3 (which works in other project) and 1.18.0. And no it doesn't help if switch to 1.18.0.

Successfully installed attrs-19.3.0 aws-cdk.aws-ecr-1.16.3 aws-cdk.aws-events-1.18.0 aws-cdk.aws-iam-1.16.3 aws-cdk.aws-kms-1.18.0 aws-cdk.aws-s3-1.16.3 aws-cdk.core-1.18.0 aws-cdk.cx-api-1.18.0 aws-cdk.region-info-1.18.0 cattrs-0.9.0 ecom jsii-0.20.8 publication-0.0.3 python-dateutil-2.8.1 six-1.13.0 typing-extensions-3.7.4.1

And actual error I get with Node 12.13.0, other project works also with 13.1.0 but this doesn't work with either one.

internal/fs/utils.js:220
    throw err;
    ^

Error: EAGAIN: resource temporarily unavailable, read
    at Object.readSync (fs.js:516:3)
    at SyncStdio.readLine (/Users/jarik/.pyenv/versions/3.7.5/envs/ecom-3.7/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13322:25)
    at InputOutput.read (/Users/jarik/.pyenv/versions/3.7.5/envs/ecom-3.7/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13272:34)
    at KernelHost.run (/Users/jarik/.pyenv/versions/3.7.5/envs/ecom-3.7/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7228:32)
    at Immediate.<anonymous> (/Users/jarik/.pyenv/versions/3.7.5/envs/ecom-3.7/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7236:37)
    at processImmediate (internal/timers.js:439:21) {
  errno: -35,
  syscall: 'read',
  code: 'EAGAIN'
}
Traceback (most recent call last):
  File "app.py", line 6, in <module>
    from deployment_support import DeploymentSupport
  File "/Users/jarik/Projects/jouzen/ecom/ecom/deployment_support.py", line 3, in <module>
    from aws_cdk.aws_ecr import Repository
  File "/Users/jarik/.pyenv/versions/ecom-3.7/lib/python3.7/site-packages/aws_cdk/aws_ecr/__init__.py", line 49, in <module>
    import aws_cdk.aws_events
  File "/Users/jarik/.pyenv/versions/ecom-3.7/lib/python3.7/site-packages/aws_cdk/aws_events/__init__.py", line 143, in <module>
    import aws_cdk.aws_iam
  File "/Users/jarik/.pyenv/versions/ecom-3.7/lib/python3.7/site-packages/aws_cdk/aws_iam/__init__.py", line 169, in <module>
    import aws_cdk.region_info
  File "/Users/jarik/.pyenv/versions/ecom-3.7/lib/python3.7/site-packages/aws_cdk/region_info/__init__.py", line 113, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load("@aws-cdk/region-info", "1.18.0", __name__, "region-info@1.18.0.jsii.tgz")
  File "/Users/jarik/.pyenv/versions/ecom-3.7/lib/python3.7/site-packages/jsii/_runtime.py", line 40, in load
    _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
  File "/Users/jarik/.pyenv/versions/ecom-3.7/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 209, in load
    self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
  File "/Users/jarik/.pyenv/versions/ecom-3.7/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 330, in load
    return self._process.send(request, LoadResponse)
  File "/Users/jarik/.pyenv/versions/ecom-3.7/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 310, in send
    self._next_message(), _ProcessResponse_R
  File "/Users/jarik/.pyenv/versions/ecom-3.7/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 261, in _next_message
    return json.loads(self._process.stdout.readline(), object_hook=ohook)
  File "/Users/jarik/.pyenv/versions/3.7.5/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/Users/jarik/.pyenv/versions/3.7.5/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/jarik/.pyenv/versions/3.7.5/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Subprocess exited with error 1

sebgoa commented 4 years ago

same here. Downgraded to stable node version: 12.3.1 fixed the issue

I was on 13.3

liujun5885 commented 4 years ago

same here. Downgraded to stable node version: 12.3.1 fixed the issue

I was on 13.3

I had the same issue. After downgraded node version: 12.3.1. problems was solved.

chefren commented 4 years ago

Tested aws-cdk Python 1.20.0 now works with node v13.6.0

brainstorm commented 4 years ago

Tested aws-cdk Python 1.22.0 (build 309ac1b) and it does not work with:

$ node --version
v13.6.0

So I guess it's once again a regression from what @chefren says? It now yields this on my OSX machine:

$ cdk deploy
internal/fs/utils.js:230
    throw err;
    ^

Error: EAGAIN: resource temporarily unavailable, read
    at Object.readSync (fs.js:533:3)
    at SyncStdio.readLine (/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13186:25)
    at InputOutput.read (/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13136:34)
    at KernelHost.run (/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:6460:32)
    at Immediate.<anonymous> (/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:6467:37)
    at processImmediate (internal/timers.js:456:21) {
  errno: -35,
  syscall: 'read',
  code: 'EAGAIN'
}
Traceback (most recent call last):
  File "app.py", line 3, in <module>
    from stacks.cicd import CICDStack
  File "/Users/romanvg/dev/umccr/infrastructure/cdk/apps/umccrise/stacks/cicd.py", line 1, in <module>
    from aws_cdk import (
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/aws_cdk/aws_codebuild/__init__.py", line 13, in <module>
    import aws_cdk.aws_cloudwatch
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/aws_cdk/aws_cloudwatch/__init__.py", line 12, in <module>
    import aws_cdk.aws_iam
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/aws_cdk/aws_iam/__init__.py", line 13, in <module>
    import aws_cdk.region_info
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/aws_cdk/region_info/__init__.py", line 11, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load("@aws-cdk/region-info", "1.14.0", __name__, "region-info@1.14.0.jsii.tgz")
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/jsii/_runtime.py", line 40, in load
    _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 199, in load
    self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 328, in load
    return self._process.send(request, LoadResponse)
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 308, in send
    self._next_message(), _ProcessResponse_R
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 259, in _next_message
    return json.loads(self._process.stdout.readline(), object_hook=ohook)
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/romanvg/.miniconda3/envs/cdk/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Subprocess exited with error 1

MrArnoldPalmer commented 4 years ago

@chefren can you confirm that node node v13.6.0 is still working for you with python cdk v1.22.0? As far as I know we haven't made any changes that would lead to node v13.6.0 working.

chefren commented 4 years ago

Yes, confirmed - note: I'm only checking Python and using brew to handle node

node --version
v13.6.0

cdk --version
1.21.1 (build 842cc5f)

python --version
Python 3.7.4

pip freeze | grep cdk.core
aws-cdk.core==1.21.1

Brew: had to follow this to manage brew toggling correctly node versions.

brew search node

==> Formulae
libbitcoin-node             node ✔                      node-sass                   node@12 ✔                   nodebrew                    nodenv
llnode                      node-build                  node@10                     node_exporter               nodeenv

==> Casks
nodebox                                                 nodeclipse                                              soundnode

If you meant "node" specifically:
It was migrated from homebrew/cask to homebrew/core.

Also upgraded CDK again (which included node upgrade) and retested OK

node --version
v13.7.0

cdk --version
1.22.0 (build 309ac1b)

pip freeze | grep cdk.core
aws-cdk.core==1.22.0

brainstorm commented 4 years ago

@chefren Did you test cdk deploy and/or cdk synth instead of just --version?

chefren commented 4 years ago

yes, version is just a reference

cdk deploy -e 12312123-cdk
[Warning at /123123-cdk/ASG] desiredCapacity has been configured. Be aware this will reset the size of your AutoScalingGroup on every deployment. See https://github.com/aws/aws-cdk/issues/5215
123123-cdk: deploying...
1123123-cdk: creating CloudFormation changeset...
 0/2 | 09:57:40 | UPDATE_IN_PROGRESS   | AWS::CDK::Metadata                        | CDKMetadata 
 1/2 | 09:57:41 | UPDATE_COMPLETE      | AWS::CDK::Metadata                        | CDKMetadata 

 ✅  123123-cdk

Outputs:
123123-cdk.ExportsOutputRefBLA1231268A3C = fs-12312312
123123-cdk.URL = https://jbla.com

Stack ARN:
arn:aws:cloudformation:ap-southeast-2:123123123:stack/123123-cdk/123123-123123-123123-123123-123123

chefren commented 4 years ago

@chefren Did you test cdk deploy and/or cdk synth instead of just --version?

@brainstorm I noted you are not using pip and venv , but miniconda, this is suspicious in your stack trace, maybe not pointing at the correct libraries?

jsii.JSIIAssembly.load("@aws-cdk/region-info", "1.14.0",

Check the python libraries versions you are using as well as node and cdk ;)

brainstorm commented 4 years ago

Indeed @chefren , that was the culprit, I fixed it like this:

$ pip freeze | grep aws | sed 's/==.*//' | xargs pip install -U

Since I have quite a few of aws-cdk modules on that virtualenv (miniconda, but same thing)... IMHO, there should be a toplevel aws-cdk metapackage that updates all submodules in lockstep by just running:

pip install -U aws-cdk

As I raised on https://github.com/aws/aws-cdk/issues/3406#issuecomment-525688693 and also mentioned on issue https://github.com/aws/aws-cdk/issues/972 ... this issue will keep coming up folks.

The current xargs hack above is not pythonic and should be fixed if you would like to avoid more issues/requests/bugs from the python community ;)

Cheers!

MrArnoldPalmer commented 4 years ago

@brainstorm this is definitely under discussion. Keep an eye out for updates.

For others watching this issue: https://github.com/aws/jsii/pull/1143 implemented a short term solution to retry failed reads from stdin immediately after EAGAIN. This is not meant to be permanent and other issues have arisen that are related, specifically #5877. Downgrading to Node 12 is still the recommended workaround if you run into this problem on cdk >= v1.20.0.

https://github.com/aws/jsii/issues/1142 is the long term fix but it requires significant investigation.

vincent-dm commented 4 years ago

I am also experiencing an issue with the latest 12.x Node: 12.17.0. Their release notes do not contain some obvious braking change, but I'm not an expert on this.

I filed an issue #8288 but it might be the same as this one.

RomainMuller commented 4 years ago

Linking this for posterity: nodejs/help#2663, it appears to contain a lot of interesting information. Definitely reinforces me thinking it's likely due to how the pipe is set-up, triggering node to enable non-blocking mode on it. There isn't a way we can escape that, so yeah... I guess this thrashing hack is there to stay 😩

rix0rrr commented 4 years ago

Hey what if we used a custom FD? So not stdin/0), but 3 or higher? Not exactly sure how this would translate to non-UNIX platform but it's probably possible there, too...

RomainMuller commented 4 years ago

@rix0rrr - I have tried opening /dev/stdin explicitly with O_RDONLY | O_SYNC (which would be a way to go in Linux & MacOS), which typically got me using fd 20 there. However somehow the behavior was unchanged (still consistently getting EAGAIN).

As I'm saying in the comment on my PR, we could set up our own channel & control it in however way we want, but cross-platform is challenging (named pipes, and the likes... Windows could be problematic)

ronytesler commented 1 year ago

I have node 18.17.1 and I have this error: node:internal/fs/rimraf:202 throw err; ^

Error: ENOTEMPTY: directory not empty, rmdir '\?\C:\Users\ronyt\AppData\Local\Temp\jsii-kernel-etkuhF\node_modules\@aws-cdk\asset-awscli-v1\projenrc' at Object.rmdirSync (node:fs:1229:10) at _rmdirSync (node:internal/fs/rimraf:260:21) at rimrafSync (node:internal/fs/rimraf:193:7) at node:internal/fs/rimraf:253:9 at Array.forEach () at _rmdirSync (node:internal/fs/rimraf:250:7) at rimrafSync (node:internal/fs/rimraf:193:7) at node:internal/fs/rimraf:253:9 at Array.forEach () at _rmdirSync (node:internal/fs/rimraf:250:7) { errno: -4051, syscall: 'rmdir', code: 'ENOTEMPTY', path: '\\?\C:\Users\ronyt\AppData\Local\Temp\jsii-kernel-etkuhF\node_modules\@aws-cdk\asset-awscli-v1\projenrc' }