search

data-dot-all / dataall

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.
https://data-dot-all.github.io/dataall/
Apache License 2.0
232 stars 82 forks source link

Unable to run 'cdk synth' #132

Closed haljeshi closed 2 years ago

haljeshi commented 2 years ago

Hello,

I have tried following the instruction presented here to deploy on AWS several times. In every attempt, I reach to Step 5 (running cdk synth) and it's not working for me.

I have tried using Amazon Linux 2 and Ubuntu. I have tried with different versions of node, cdk and python. And Every time, it fails to run cdk synth.

During my several attempts, I noticed is that the node version is the thing that is making a difference. So, here is what I get depending on the node version I try (all output is the result of running 'cdk synth --verbose'):

On node v12.22.12

/home/ec2-user/.nvm/versions/node/v12.22.12/lib/node_modules/aws-cdk/lib/index.js:12405
    home = path.join((os.userInfo().homedir ?? os.homedir()).trim(), ".cdk");
                                             ^

SyntaxError: Unexpected token '?'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:963:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/home/ec2-user/.nvm/versions/node/v12.22.12/lib/node_modules/aws-cdk/bin/cdk.js:3:15)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
(venv) [ec2-user@ip-172-31-9-97 aws-dataall]$ cdk synth
/home/ec2-user/.nvm/versions/node/v12.22.12/lib/node_modules/aws-cdk/lib/index.js:12405
    home = path.join((os.userInfo().homedir ?? os.homedir()).trim(), ".cdk");
                                             ^

SyntaxError: Unexpected token '?'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:963:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/home/ec2-user/.nvm/versions/node/v12.22.12/lib/node_modules/aws-cdk/bin/cdk.js:3:15)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)

On node v13.14.0

/home/ec2-user/.nvm/versions/node/v13.14.0/lib/node_modules/aws-cdk/lib/index.js:12405
    home = path.join((os.userInfo().homedir ?? os.homedir()).trim(), ".cdk");
                                             ^

SyntaxError: Unexpected token '?'
    at Object.compileFunction (vm.js:344:18)
    at wrapSafe (internal/modules/cjs/loader.js:1048:15)
    at Module._compile (internal/modules/cjs/loader.js:1082:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1138:10)
    at Module.load (internal/modules/cjs/loader.js:982:32)
    at Function.Module._load (internal/modules/cjs/loader.js:875:14)
    at Module.require (internal/modules/cjs/loader.js:1022:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/home/ec2-user/.nvm/versions/node/v13.14.0/lib/node_modules/aws-cdk/bin/cdk.js:3:15)
    at Module._compile (internal/modules/cjs/loader.js:1118:30)

On node v14.20.0 (it takes a long time)

CDK toolkit version: 2.42.0 (build 7d8ef0b)
Command line arguments: {
  _: [ 'synth' ],
  verbose: 1,
  v: 1,
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  debug: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  ci: false,
  validation: true,
  quiet: false,
  q: false,
  '$0': 'cdk'
}
cdk.json: {
  "app": "python ./deploy/app.py",
  "context": {
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": false,
    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": false,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": false,
    "@aws-cdk/core:stackRelativeExports": false,
    "tooling_region": "eu-west-2",
    "DeploymentEnvironments": [
      {
        "envname": "testenv",
        "account": "****************",
        "region": "eu-west-2",
        "internet_facing": true
      }
    ]
  }
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python ./deploy/app.py',
  context: {
    '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': false,
    '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': false,
    '@aws-cdk/aws-rds:lowercaseDbIdentifier': false,
    '@aws-cdk/core:stackRelativeExports': false,
    tooling_region: 'eu-west-2',
    DeploymentEnvironments: [ [Object] ]
  },
  debug: false,
  assetMetadata: true,
  toolkitBucket: {},
  staging: true,
  bundlingStacks: [ '*' ],
  lookups: true
}
Determining if we're on an EC2 instance.
Looks like an EC2 instance.
Reading cached notices from /home/ec2-user/.cdk/cache/notices.json
Looking up AWS region in the EC2 Instance Metadata Service (IMDS).
Attempting to retrieve an IMDSv2 token.
Retrieving the AWS region from the IMDS.
AWS region from IMDS: eu-west-2
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to eu-west-2
Resolving default credentials
Retrieved account ID **************** from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to ****************
context: {
  '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': false,
  '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': false,
  '@aws-cdk/aws-rds:lowercaseDbIdentifier': false,
  '@aws-cdk/core:stackRelativeExports': false,
  tooling_region: 'eu-west-2',
  DeploymentEnvironments: [
    {
      envname: 'testenv',
      account: '****************',
      region: 'eu-west-2',
      internet_facing: true
    }
  ],
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'eu-west-2',
  CDK_DEFAULT_ACCOUNT: '****************',
  CDK_CONTEXT_JSON: '{"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId":false,"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021":false,"@aws-cdk/aws-rds:lowercaseDbIdentifier":false,"@aws-cdk/core:stackRelativeExports":false,"tooling_region":"eu-west-2","DeploymentEnvironments":[{"envname":"testenv","account":"****************","region":"eu-west-2","internet_facing":true}],"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true,"aws:cdk:version-reporting":true,"aws:cdk:bundling-stacks":["*"]}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '21.0.0',
  CDK_CLI_VERSION: '2.42.0'
}
Bundling asset dataall-main-cicd-stack/dataall-testenv-backend-stage/backend-stack/Cognito/CognitoParamsSyncHandlertestenv/Code/Stage...
Bundling asset dataall-main-cicd-stack/dataall-testenv-cloudfront-stage/cloudfront-stack/CloudFront/dataall-testenv-httpheaders-redirection/Code/Stage...
Some context information is missing. Fetching...
Reading AZs for ****************:eu-west-2
Call failed: describeAvailabilityZones(undefined) => You are not authorized to perform this operation. (code=UnauthorizedOperation)
Setting "availability-zones:account=****************:region=eu-west-2" context to {"$providerError":"You are not authorized to perform this operation.","$dontSaveContext":true}
Setting "CDK_DEFAULT_REGION" environment variable to eu-west-2
Setting "CDK_DEFAULT_ACCOUNT" environment variable to ****************
context: {
  'availability-zones:account=****************:region=eu-west-2': {
    '$providerError': 'You are not authorized to perform this operation.',
    '$dontSaveContext': true
  },
  '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': false,
  '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': false,
  '@aws-cdk/aws-rds:lowercaseDbIdentifier': false,
  '@aws-cdk/core:stackRelativeExports': false,
  tooling_region: 'eu-west-2',
  DeploymentEnvironments: [
    {
      envname: 'testenv',
      account: '****************',
      region: 'eu-west-2',
      internet_facing: true
    }
  ],
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'eu-west-2',
  CDK_DEFAULT_ACCOUNT: '****************',
  CDK_CONTEXT_JSON: '{"availability-zones:account=****************:region=eu-west-2":{"$providerError":"You are not authorized to perform this operation.","$dontSaveContext":true},"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId":false,"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021":false,"@aws-cdk/aws-rds:lowercaseDbIdentifier":false,"@aws-cdk/core:stackRelativeExports":false,"tooling_region":"eu-west-2","DeploymentEnvironments":[{"envname":"testenv","account":"****************","region":"eu-west-2","internet_facing":true}],"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true,"aws:cdk:version-reporting":true,"aws:cdk:bundling-stacks":["*"]}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '21.0.0',
  CDK_CLI_VERSION: '2.42.0'
}
Traceback (most recent call last):
  File "./deploy/app.py", line 8, in <module>
    from cdk_nag import AwsSolutionsChecks, NagSuppressions, NagPackSuppression
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/cdk_nag/__init__.py", line 491, in <module>
    from ._jsii import *
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/cdk_nag/_jsii/__init__.py", line 14, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load(
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_runtime.py", line 43, in load
    _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 300, in load
    self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 341, in load
    return self._process.send(request, LoadResponse)
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 318, in send
    self._process.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 286, in stop
    self._process.stdin.close()
BrokenPipeError: [Errno 32] Broken pipe
Reading cached notices from /home/ec2-user/.cdk/cache/notices.json

NOTICES

19179   (aws-eks): Regression in installing Helm charts from assets

    Overview: Helm charts fail to install when provided as an asset. This
              issue does not affect charts installed from a repository.

    Affected versions: framework: 2.14.0, framework: 1.146.0

    More information at: https://github.com/aws/aws-cdk/issues/19179

If you don’t want to see a notice anymore, use "cdk acknowledge <id>". For example, "cdk acknowledge 19179".
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/home/ec2-user/.nvm/versions/node/v14.20.0/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:131:23)
    at ChildProcess.emit (events.js:400:28)
    at ChildProcess.emit (domain.js:475:12)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:285:12)

On node v16.17.0

CDK toolkit version: 2.41.0 (build 6ad48a3)
Command line arguments: {
  _: [ 'synth' ],
  verbose: 1,
  v: 1,
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  debug: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  ci: false,
  validation: true,
  quiet: false,
  q: false,
  '$0': 'cdk'
}
cdk.json: {
  "app": "python ./deploy/app.py",
  "context": {
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": false,
    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": false,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": false,
    "@aws-cdk/core:stackRelativeExports": false,
    "tooling_region": "eu-west-2",
    "DeploymentEnvironments": [
      {
        "envname": "testenv",
        "account": "****************",
        "region": "eu-west-2",
        "internet_facing": true
      }
    ]
  }
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python ./deploy/app.py',
  context: {
    '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': false,
    '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': false,
    '@aws-cdk/aws-rds:lowercaseDbIdentifier': false,
    '@aws-cdk/core:stackRelativeExports': false,
    tooling_region: 'eu-west-2',
    DeploymentEnvironments: [ [Object] ]
  },
  debug: false,
  assetMetadata: true,
  toolkitBucket: {},
  staging: true,
  bundlingStacks: [ '*' ],
  lookups: true
}
Determining if we're on an EC2 instance.
Looks like an EC2 instance.
Reading cached notices from /home/ec2-user/.cdk/cache/notices.json
Looking up AWS region in the EC2 Instance Metadata Service (IMDS).
Attempting to retrieve an IMDSv2 token.
Retrieving the AWS region from the IMDS.
AWS region from IMDS: eu-west-2
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to eu-west-2
Resolving default credentials
Looking up default account ID from STS
Default account ID: ****************
Setting "CDK_DEFAULT_ACCOUNT" environment variable to ****************
context: {
  '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': false,
  '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': false,
  '@aws-cdk/aws-rds:lowercaseDbIdentifier': false,
  '@aws-cdk/core:stackRelativeExports': false,
  tooling_region: 'eu-west-2',
  DeploymentEnvironments: [
    {
      envname: 'testenv',
      account: '****************',
      region: 'eu-west-2',
      internet_facing: true
    }
  ],
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'eu-west-2',
  CDK_DEFAULT_ACCOUNT: '****************',
  CDK_CONTEXT_JSON: '{"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId":false,"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021":false,"@aws-cdk/aws-rds:lowercaseDbIdentifier":false,"@aws-cdk/core:stackRelativeExports":false,"tooling_region":"eu-west-2","DeploymentEnvironments":[{"envname":"testenv","account":"****************","region":"eu-west-2","internet_facing":true}],"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true,"aws:cdk:version-reporting":true,"aws:cdk:bundling-stacks":["*"]}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '21.0.0',
  CDK_CLI_VERSION: '2.41.0'
}
Traceback (most recent call last):
  File "./deploy/app.py", line 8, in <module>
    from cdk_nag import AwsSolutionsChecks, NagSuppressions, NagPackSuppression
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/cdk_nag/__init__.py", line 491, in <module>
    from ._jsii import *
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/cdk_nag/_jsii/__init__.py", line 14, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load(
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_runtime.py", line 43, in load
    _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 300, in load
    self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 341, in load
    return self._process.send(request, LoadResponse)
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 318, in send
    self._process.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/ec2-user/aws-dataall/venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 286, in stop
    self._process.stdin.close()
BrokenPipeError: [Errno 32] Broken pipe
Reading cached notices from /home/ec2-user/.cdk/cache/notices.json

NOTICES

19179   (aws-eks): Regression in installing Helm charts from assets

    Overview: Helm charts fail to install when provided as an asset. This
              issue does not affect charts installed from a repository.

    Affected versions: framework: 2.14.0, framework: 1.146.0

    More information at: https://github.com/aws/aws-cdk/issues/19179

If you don’t want to see a notice anymore, use "cdk acknowledge <id>". For example, "cdk acknowledge 19179".
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/home/ec2-user/.nvm/versions/node/v16.17.0/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:131:23)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess.emit (node:domain:489:12)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
dlpzx commented 2 years ago

Hi @haljeshi, thanks for your question. A couple of months ago we found issues when using older node versions (older than 15). If you check the actual CDK deployment pipeline you will see that we use 'n 16.15.1'. That should solve the first part of the issue. I have redeployed the platform and I am not encountering your issue, but one possible explanation is the site package JSII, which in general we don't have to worry about because AWS CDK manages this dependency. However, sometimes we have seen issues with conflicting new versions of this package, so a possible quick workaround would be to add it in the deploy/requirements.txt as “jsii==1.67.0". Give it a try and let us know if it works :)

haljeshi commented 2 years ago

Hi @dlpzx Thank you for your suggestion. However, it did not solved my problem.

However, the problem has been solved by something much simpler that I didn't notice before. And I'm reporting it here so that if someone else has done this same silly mistake as me, they can check their code.

In the cdk.json file, the first line is "app": "python ./deploy/app.py" by default This will invoke python2 if you didn't change the alias before. I modified this line to "app": "python3.8 ./deploy/app.py" and it worked.

Sorry to bother you and hope that this will help other people successfully install the system.