search

data-dot-all / dataall

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.
https://data-dot-all.github.io/dataall/
Apache License 2.0
232 stars 82 forks source link

Deploy dataall on AWS account is failed [Python 3.8] - Windows deployment #340

Closed vaquarkhan closed 1 year ago

vaquarkhan commented 1 year ago

Window 10 Python 3.8

Describe the bug

https://awslabs.github.io/aws-dataall/deploy-aws/

Json 

{
  "app": "python ./deploy/app.py",
  "context": {
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": false,
    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": false,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": false,
    "@aws-cdk/core:stackRelativeExports": false,
    "DeploymentEnvironments": [
      {
        "envname": "sandbox",
        "account": "XXXXXXXXXXXXXXXXX",
        "region": "us-east-1"
      }
    ]
  }
}

Logs 👍 


     ╭───────────────────────────────────────────────────────────────╮
     │                                                               │
     │     New major version of npm available! 6.14.11 -> 9.5.1      │
     │   Changelog: https://github.com/npm/cli/releases/tag/v9.5.1   │
     │               Run npm install -g npm to update!               │
     │                                                               │
     ╰───────────────────────────────────────────────────────────────╯

  (venv) C:\dataall\test\dataall>cdk --version
  2.66.1 (build 539d036)

  (venv) C:\dataall\test\dataall>cdk synth
  [2023-03-01 23:31:12,160][credentials.py:1255] Found credentials in shared credentials file: ~/.aws/credentials
  [2023-03-01 23:31:12,822][app.py       : 40] Trying to get cdkjson parameter from SSM
  [2023-03-01 23:31:13,041][app.py       : 49] SSM parameter not found - Proceeding with cdk.json and cdk.context.json in code
  [2023-03-01 23:31:13,057][app.py       : 54] Loaded context from cdk.json file in repository
  Bundling asset dataall-main-cicd-stack/dataall-sandbox-backend-stage/backend-stack/Cognito/CognitoParamsSyncHandlersandbox/Code/Stage...
  jsii.errors.JavaScriptError:
    Error: Failed to bundle asset dataall-main-cicd-stack/dataall-sandbox-backend-stage/backend-stack/Cognito/CognitoParamsSyncHandlersandbox/Code/Stage, bundle output is located at C:\dataall\test\dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979-error: @jsii/kernel.RuntimeError: Command '['cp -a C:\\dataall\\test\\dataall\\deploy\\custom_resources\\sync_congito_params/. C:\\dataall\\test\\dataall\\cdk.out\\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979/ && pip install -r C:\\dataall\\test\\dataall\\deploy\\custom_resources\\sync_congito_params\\requirements.txt -t C:\\dataall\\test\\dataall\\cdk.out\\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979']' returned non-zero exit status 1.
        at AssetStaging.bundle (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-svAMx4\node_modules\aws-cdk-lib\core\lib\asset-staging.js:2:672)
        at AssetStaging.stageByBundling (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-svAMx4\node_modules\aws-cdk-lib\core\lib\asset-staging.js:1:4168)
        at stageThisAsset (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-svAMx4\node_modules\aws-cdk-lib\core\lib\asset-staging.js:1:1675)
        at Cache.obtain (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-svAMx4\node_modules\aws-cdk-lib\core\lib\private\cache.js:1:242)
        at new AssetStaging (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-svAMx4\node_modules\aws-cdk-lib\core\lib\asset-staging.js:1:2070)
        at new Asset (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-svAMx4\node_modules\aws-cdk-lib\aws-s3-assets\lib\asset.js:1:620)
        at AssetCode.bind (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-svAMx4\node_modules\aws-cdk-lib\aws-lambda\lib\code.js:1:3506)
        at new Function (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-svAMx4\node_modules\aws-cdk-lib\aws-lambda\lib\function.js:1:2424)
        at Kernel._create (C:\Users\ADMINI~1\AppData\Local\Temp\tmptc9oebxq\lib\program.js:9937:29)
        at Kernel.create (C:\Users\ADMINI~1\AppData\Local\Temp\tmptc9oebxq\lib\program.js:9666:29)

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "./deploy/app.py", line 68, in <module>
      pipeline = PipelineStack(
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_runtime.py", line 110, in __call__
      inst = super().__call__(*args, **kwargs)
    File "C:\dataall\test\dataall\deploy\stacks\pipeline.py", line 254, in __init__
      backend_stage = self.set_backend_stage(target_env, repository_name)
    File "C:\dataall\test\dataall\deploy\stacks\pipeline.py", line 563, in set_backend_stage
      BackendStage(
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_runtime.py", line 110, in __call__
      inst = super().__call__(*args, **kwargs)
    File "C:\dataall\test\dataall\deploy\stacks\backend_stage.py", line 34, in __init__
      backend_stack = BackendStack(
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_runtime.py", line 110, in __call__
      inst = super().__call__(*args, **kwargs)
    File "C:\dataall\test\dataall\deploy\stacks\backend_stack.py", line 91, in __init__
      cognito_stack = IdpStack(
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_runtime.py", line 110, in __call__
      inst = super().__call__(*args, **kwargs)
    File "C:\dataall\test\dataall\deploy\stacks\cognito.py", line 251, in __init__
      cognito_sync_handler = _lambda.Function(
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_runtime.py", line 110, in __call__
      inst = super().__call__(*args, **kwargs)
    File "C:\dataall\aws-dataall\venv\lib\site-packages\aws_cdk\aws_lambda\__init__.py", line 18583, in __init__
      jsii.create(self.__class__, self, [scope, id, props])
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_kernel\__init__.py", line 346, in create
      obj.__jsii_ref__ = _callback_till_result(self, response, CreateResponse)
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_kernel\__init__.py", line 263, in _callback_till_result
      response = kernel.sync_complete(
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_kernel\__init__.py", line 445, in sync_complete
      return self.provider.sync_complete(
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_kernel\providers\process.py", line 396, in sync_complete
      resp = self._process.send(_CompleteRequest(complete=request), response_type)
    File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii\_kernel\providers\process.py", line 335, in send
      raise RuntimeError(resp.error) from JavaScriptError(resp.stack)
  RuntimeError: Failed to bundle asset dataall-main-cicd-stack/dataall-sandbox-backend-stage/backend-stack/Cognito/CognitoParamsSyncHandlersandbox/Code/Stage, bundle output is located at C:\dataall\test\dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979-error: @jsii/kernel.RuntimeError: Command '['cp -a C:\\dataall\\test\\dataall\\deploy\\custom_resources\\sync_congito_params/. C:\\dataall\\test\\dataall\\cdk.out\\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979/ && pip install -r C:\\dataall\\test\\dataall\\deploy\\custom_resources\\sync_congito_params\\requirements.txt -t C:\\dataall\\test\\dataall\\cdk.out\\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979']' returned non-zero exit status 1.

  Subprocess exited with error 1

  (venv) C:\dataall\test\dataall>python --version
  Python 3.8.0

How to Reproduce

*P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.*

Clone code and run "cdk synth"

Expected behavior

Should work without error

Your project

No response

Screenshots

attached logs

OS

Window 10

Python version

3.7 and 3.8

AWS data.all version

https://github.com/awslabs/aws-dataall

Additional context

tried on python 3.7 and 3.8 its not working

2023-03-01 23:31:12,160][credentials.py:1255] Found credentials in shared credentials file: ~/.aws/credentials [2023-03-01 23:31:12,822][app.py : 40] Trying to get cdkjson parameter from SSM [2023-03-01 23:31:13,041][app.py : 49] SSM parameter not found - Proceeding with cdk.json and cdk.context.json in code [2023-03-01 23:31:13,057][app.py : 54] Loaded context from cdk.json file in repository Bundling asset dataall-main-cicd-stack/dataall-sandbox-backend-stage/backend-stack/Cognito/CognitoParamsSyncHandlersandbox/Code/Stage... jsii.errors.JavaScriptError: Error: Failed to bundle asset dataall-main-cicd-stack/dataall-sandbox-backend-stage/backend-stack/Cognito/CognitoParamsSyncHandlersandbox/Code/Stage, bundle output is located at C:\dataall\test\dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979-error: @jsii/kernel.RuntimeError: Command '['cp -a C:\dataall\test\dataall\deploy\custom_resources\sync_congito_params/. C:\dataall\test\dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979/ && pip install -r C:\dataall\test\dataall\deploy\custom_resources\sync_congito_params\requirements.txt -t C:\dataall\test\dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979']' returned non-zero exit status 1.

dlpzx commented 1 year ago

Can we have a look at the cdk.context.json? Does it contain both the tooling and the deployment account?

vaquarkhan commented 1 year ago

Update :

I have setup window 10 powershell and run same commands again and getting same error

(venv) PS C:\dataall\aws-dataall> cdk synth [2023-03-10 19:37:59,112][credentials.py:1255] Found credentials in shared credentials file: ~/.aws/credentials [2023-03-10 19:37:59,734][app.py : 40] Trying to get cdkjson parameter from SSM [2023-03-10 19:38:00,016][app.py : 49] SSM parameter not found - Proceeding with cdk.json and cdk.context.json in code [2023-03-10 19:38:00,031][app.py : 54] Loaded context from cdk.json file in repository Bundling asset dataall-main-cicd-stack/dataall-sandbox-backend-stage/backend-stack/Cognito/CognitoParamsSyncHandlersandbox/Code/Stage... jsii.errors.JavaScriptError: Error: Failed to bundle asset dataall-main-cicd-stack/dataall-sandbox-backend-stage/backend-stack/Cognito/CognitoParamsSyncHandlersandbox/Code/Stage, bundle output is located at C:\dataall\aws-dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979-error: @jsii/kernel.RuntimeError: Command '['cp -a C:\dataall\aws-dataall\deploy\custom_resources\sync_congito_params/. C:\dataall\aws-dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979/ && pip install -r C:\dataall\aws-dataall\deploy\custom_resources\sync_congito_params\requirements.txt -t C:\dataall\aws-dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979']' returned non-zero exit status 1. at AssetStaging.bundle (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-zIi5m1\node_modules\aws-cdk-lib\core\lib\asset-staging.js:2:672) at AssetStaging.stageByBundling (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-zIi5m1\node_modules\aws-cdk-lib\core\lib\asset-staging.js:1:4168) at stageThisAsset (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-zIi5m1\node_modules\aws-cdk-lib\core\lib\asset-staging.js:1:1675) at Cache.obtain (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-zIi5m1\node_modules\aws-cdk-lib\core\lib\private\cache.js:1:242) at new AssetStaging (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-zIi5m1\node_modules\aws-cdk-lib\core\lib\asset-staging.js:1:2070) at new Asset (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-zIi5m1\node_modules\aws-cdk-lib\aws-s3-assets\lib\asset.js:1:620) at AssetCode.bind (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-zIi5m1\node_modules\aws-cdk-lib\aws-lambda\lib\code.js:1:3506) at new Function (C:\Users\ADMINI~1\AppData\Local\Temp\jsii-kernel-zIi5m1\node_modules\aws-cdk-lib\aws-lambda\lib\function.js:1:2424) at Kernel._create (C:\Users\ADMINI~1\AppData\Local\Temp\tmp1kx93aew\lib\program.js:9937:29) at Kernel.create (C:\Users\ADMINI~1\AppData\Local\Temp\tmp1kx93aew\lib\program.js:9666:29)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "./deploy/app.py", line 68, in pipeline = PipelineStack( File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_runtime.py", line 110, in call inst = super().call(*args, kwargs) File "C:\dataall\aws-dataall\deploy\stacks\pipeline.py", line 254, in init backend_stage = self.set_backend_stage(target_env, repository_name) File "C:\dataall\aws-dataall\deploy\stacks\pipeline.py", line 563, in set_backend_stage BackendStage( File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_runtime.py", line 110, in call inst = super().call(*args, *kwargs) File "C:\dataall\aws-dataall\deploy\stacks\backend_stage.py", line 34, in init backend_stack = BackendStack( File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_runtime.py", line 110, in call inst = super().call(args, kwargs) File "C:\dataall\aws-dataall\deploy\stacks\backend_stack.py", line 91, in init cognito_stack = IdpStack( File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_runtime.py", line 110, in call inst = super().call(*args, *kwargs) File "C:\dataall\aws-dataall\deploy\stacks\cognito.py", line 251, in init cognito_sync_handler = _lambda.Function( File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_runtime.py", line 110, in call inst = super().call(args, **kwargs) File "C:\dataall\aws-dataall\venv\lib\site-packages\aws_cdk\aws_lambda__init.py", line 18583, in init jsii.create(self.class, self, [scope, id, props]) File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_kernel__init.py", line 346, in create obj.jsii_ref = _callback_till_result(self, response, CreateResponse) File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_kernel__init.py", line 263, in _callback_till_result response = kernel.sync_complete( File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_kernel\init__.py", line 445, in sync_complete return self.provider.sync_complete( File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_kernel\providers\process.py", line 396, in sync_complete resp = self._process.send(_CompleteRequest(complete=request), response_type) File "C:\dataall\aws-dataall\venv\lib\site-packages\jsii_kernel\providers\process.py", line 335, in send raise RuntimeError(resp.error) from JavaScriptError(resp.stack) RuntimeError: Failed to bundle asset dataall-main-cicd-stack/dataall-sandbox-backend-stage/backend-stack/Cognito/CognitoParamsSyncHandlersandbox/Code/Stage, bundle output is located at C:\dataall\aws-dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979-error: @jsii/kernel.RuntimeError: Command '['cp -a C:\dataall\aws-dataall\deploy\custom_resources\sync_congito_params/. C:\dataall\aws-dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979/ && pip install -r C:\dataall\aws-dataall\deploy\custom_resources\sync_congito_params\requirements.txt -t C:\dataall\aws-dataall\cdk.out\asset.2cd6fafc2341be652b93379d7b28312db9820e80e01c47930667c0815c206979']' returned non-zero exit status 1.

Subprocess exited with error 1 (venv) PS C:\dataall\aws-dataall>

vaquarkhan commented 1 year ago

Update : Configure Ubantu 20 on window 10 and cdk worked without issue , it seems instruction provided only works for Unix not window .

dlpzx commented 1 year ago

Thank you @vaquarkhan for diving deep on the issue. I would also be interested to know if this an issue that only affects you when using Windows to deploy data.all or if you have issues running cdk synth on ther CDK applications.

vaquarkhan commented 1 year ago

I am still looking into it will share detail once fix on my window machine

dlpzx commented 1 year ago

Thank you @vaquarkhan

dlpzx commented 1 year ago

Closing this issue as it has a workaround and is inactive