search

aws / aws-toolkit-jetbrains

AWS Toolkit for JetBrains - a plugin for interacting with AWS from JetBrains IDEs
https://plugins.jetbrains.com/plugin/11349-aws-toolkit
Apache License 2.0
746 stars 214 forks source link

.npmrc file gets removed before launching the app #2733

Open aruizca opened 3 years ago

aruizca commented 3 years ago

Describe the bug When using a privately hosted npm dependency the launcher fails because it cannot resolve the dependency. We found two issues happening:

To reproduce

Expected behavior

Screenshots

Your Environment

Additional context We are implementing the AWS lambda with Typescript version 4.3.5

rli commented 3 years ago

I could not reproduce this issue. For TypeScript handlers, the toolkit will perform the following steps:

  1. invoke the typescript compiler against the source root for the handler, outputting into the aws-toolkit-ts-output folder
  2. run sam build against a temporary .yaml file, pointing at the handler in the ts output folder
    • this step runs npm install and then deletes the .npmrc file
  3. run sam local invoke

I would not expect .npmrc to be available in the docker container, but node_modules should have been pulled in as part of sam build. Is there more context or logs you can share? The toolkit also provides a typescript sample template (node10.x only at the moment) in the new project wizard that may give you some guidance.

aruizca commented 3 years ago

Hi Richard,

thanks for your quick reply. The complete log is the following:

Invoking aws-toolkit-ts-output/src/handler/publishingEnqueuerHandler.handler (nodejs14.x)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-nodejs14.x:rapid-1.27.2.

Mounting /Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/build/Function as /var/task:ro,delegated inside runtime container
START RequestId: d0559b75-42c6-406d-88ca-588f2238f225 Version: $LATEST
Debugger listening on ws://0.0.0.0:57544/16cef906-66b2-4b23-8424-706e9549cd09
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
2021-07-28T09:49:54.333Z    undefined   ERROR   Uncaught Exception  {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module '@pepr/commons'\nRequire stack:\n- /var/task/aws-toolkit-ts-output/src/handler/publishingEnqueuerHandler.js\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js","stack":["Runtime.ImportModuleError: Error: Cannot find module '@pepr/commons'","Require stack:","- /var/task/aws-toolkit-ts-output/src/handler/publishingEnqueuerHandler.js","- /var/runtime/UserFunction.js","- /var/runtime/index.js","    at _loadUserApp (/var/runtime/UserFunction.js:100:13)","    at Object.load (/var/runtime/UserFunction.js:140:17)","    at Object.<anonymous> (/var/runtime/index.js:43:30)","    at Module._compile (internal/modules/cjs/loader.js:1082:14)","    at Object..js (internal/modules/cjs/loader.js:1114:10)","    at Module.load (internal/modules/cjs/loader.js:950:32)","    at Function._load (internal/modules/cjs/loader.js:790:14)","    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)","    at internal/main/run_main_module.js:17:47"]}
time="2021-07-28T09:49:54.338" level=panic msg="ReplyStream not available"
2021/07/28 09:49:54 http: panic serving 127.0.0.1:58196: &{0xc00001e0e0 map[] 2021-07-28 09:49:54.338896876 +0000 UTC m=+4.608023110 panic <nil> ReplyStream not available <nil> <nil> }
goroutine 54 [running]:
net/http.(*conn).serve.func1(0xc00029e1e0)
    /usr/local/go/src/net/http/server.go:1800 +0x139
panic(0x866640, 0xc0001d8070)
    /usr/local/go/src/runtime/panic.go:975 +0x3e3
github.com/sirupsen/logrus.Entry.log(0xc00001e0e0, 0xc000190120, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x335
github.com/sirupsen/logrus.(*Entry).Log(0xc0001d8000, 0xc000000000, 0xc0001cf588, 0x1, 0x1)
    /go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xeb
github.com/sirupsen/logrus.(*Logger).Log(0xc00001e0e0, 0xc000000000, 0xc0001cf588, 0x1, 0x1)
    /go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:193 +0x7d
github.com/sirupsen/logrus.(*Logger).Panic(...)
    /go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:234
github.com/sirupsen/logrus.Panic(...)
    /go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/exported.go:129
go.amzn.com/lambda/rapi/rendering.RenderInteropError(0x9097c0, 0xc0001c0000, 0xc0001c8100, 0x902b60, 0xc000182090)
    /LambdaRuntimeLocal/lambda/rapi/rendering/rendering.go:292 +0x9a
go.amzn.com/lambda/rapi/handler.(*initErrorHandler).ServeHTTP(0xc00009c0a0, 0x9097c0, 0xc0001c0000, 0xc0001c8100)
    /LambdaRuntimeLocal/lambda/rapi/handler/initerror.go:52 +0x519
net/http.HandlerFunc.ServeHTTP(0xc000096380, 0x9097c0, 0xc0001c0000, 0xc0001c8100)
    /usr/local/go/src/net/http/server.go:2041 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc0000a60c0, 0x9097c0, 0xc0001c0000, 0xc0001c8100)
    /go/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/mux.go:431 +0x278
net/http.HandlerFunc.ServeHTTP(0xc00009c030, 0x9097c0, 0xc0001c0000, 0xc0001c8100)
    /usr/local/go/src/net/http/server.go:2041 +0x44
go.amzn.com/lambda/rapi/middleware.RuntimeReleaseMiddleware.func1.1(0x9097c0, 0xc0001c0000, 0xc0001c8100)
    /LambdaRuntimeLocal/lambda/rapi/middleware/middleware.go:100 +0xea
net/http.HandlerFunc.ServeHTTP(0xc000096180, 0x9097c0, 0xc0001c0000, 0xc0001c8100)
    /usr/local/go/src/net/http/server.go:2041 +0x44
go.amzn.com/lambda/rapi/middleware.AccessLogMiddleware.func1.1(0x9097c0, 0xc0001c0000, 0xc0001c8100)
    /LambdaRuntimeLocal/lambda/rapi/middleware/middleware.go:77 +0x170
net/http.HandlerFunc.ServeHTTP(0xc0000961a0, 0x9097c0, 0xc0001c0000, 0xc0001c8100)
    /usr/local/go/src/net/http/server.go:2041 +0x44
go.amzn.com/lambda/rapi/middleware.AppCtxMiddleware.func1.1(0x9097c0, 0xc0001c0000, 0xc0001c8000)
    /LambdaRuntimeLocal/lambda/rapi/middleware/middleware.go:66 +0x77
net/http.HandlerFunc.ServeHTTP(0xc00009a2d0, 0x9097c0, 0xc0001c0000, 0xc0001c8000)
    /usr/local/go/src/net/http/server.go:2041 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc0000a60c0, 0x9097c0, 0xc0001c0000, 0xc0001c8000)
    /go/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/mux.go:70 +0x513
github.com/go-chi/chi.(*Mux).Mount.func1(0x9097c0, 0xc0001c0000, 0xc0001c8000)
    /go/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/mux.go:298 +0x118
net/http.HandlerFunc.ServeHTTP(0xc0000963e0, 0x9097c0, 0xc0001c0000, 0xc0001c8000)
    /usr/local/go/src/net/http/server.go:2041 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc0000a6060, 0x9097c0, 0xc0001c0000, 0xc0001c8000)
    /go/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/mux.go:431 +0x278
net/http.HandlerFunc.ServeHTTP(0xc00009c0c0, 0x9097c0, 0xc0001c0000, 0xc0001c8000)
    /usr/local/go/src/net/http/server.go:2041 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc0000a6060, 0x9097c0, 0xc0001c0000, 0xc0002b8300)
    /go/pkg/mod/github.com/go-chi/chi@v4.1.2+incompatible/mux.go:86 +0x2b2
net/http.serverHandler.ServeHTTP(0xc0000ac000, 0x9097c0, 0xc0001c0000, 0xc0002b8300)
    /usr/local/go/src/net/http/server.go:2836 +0xa3
net/http.(*conn).serve(0xc00029e1e0, 0x90a800, 0xc0002a4280)
    /usr/local/go/src/net/http/server.go:1924 +0x86c
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2962 +0x35c
2021-07-28T09:49:54.342Z    undefined   ERROR   Uncaught Exception  {"errorType":"Error","errorMessage":"socket hang up","code":"ECONNRESET","stack":["Error: socket hang up","    at connResetException (internal/errors.js:628:14)","    at Socket.socketOnEnd (_http_client.js:499:23)","    at Socket.emit (events.js:387:35)","    at endReadableNT (internal/streams/readable.js:1317:12)","    at processTicksAndRejections (internal/process/task_queues.js:82:21)"]}
Waiting for the debugger to disconnect...
time="2021-07-28T09:49:54.371" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 129"
time="2021-07-28T09:49:54.371" level=error msg="INIT DONE failed: Runtime.ExitError"

Thanks again!

rli commented 3 years ago

Looks like a build issue more than a run issue. After a failed invocation, you can try running the following to see if there are any interesting warnings pointing to your problem:

/usr/local/bin/sam build Function --template /Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/temp-template.yaml --build-dir /Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/build --debug

If your dependency is platform-dependent, your function probably needs to be built in-container (exposed as a checkbox in the SAM CLI section of the run configuration)

aruizca commented 3 years ago

Here is the output from running the command. The build seems fine:

 ✘ ✝  workspace/pepr/pepr-4-confluence/feature/PEPR4C-27-create-P4C-publinshing-infrastructure-aws > /usr/local/bin/sam build Function --template /Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/temp-template.yaml --build-dir /Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/build --debug
2021-07-29 13:25:44,757 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-07-29 13:25:44,758 | Using config file: samconfig.toml, config environment: default
2021-07-29 13:25:44,758 | Expand command line arguments to:
2021-07-29 13:25:44,758 | --template_file=/Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/temp-template.yaml --build_dir=/Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/build --resource_logical_id=Function --cache_dir=.aws-sam/cache 
2021-07-29 13:25:44,867 | 'build' command is called
2021-07-29 13:25:44,869 | No Parameters detected in the template
2021-07-29 13:25:44,894 | 1 stacks found in the template
2021-07-29 13:25:44,894 | No Parameters detected in the template
2021-07-29 13:25:44,927 | 1 resources found in the stack 
2021-07-29 13:25:44,928 | No Parameters detected in the template
2021-07-29 13:25:44,948 | Found Serverless function with name='Function' and CodeUri='/Users/aruizca/workspace/pepr/pepr-4-confluence/aws/p4c-publishing'
2021-07-29 13:25:44,948 | --base-dir is not presented, adjusting uri /Users/aruizca/workspace/pepr/pepr-4-confluence/aws/p4c-publishing relative to /Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/temp-template.yaml
2021-07-29 13:25:44,948 | No Parameters detected in the template
2021-07-29 13:25:44,979 | Instantiating build definitions
2021-07-29 13:25:44,979 | No previous build graph found, generating new one
2021-07-29 13:25:44,979 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(nodejs14.x, /Users/aruizca/workspace/pepr/pepr-4-confluence/aws/p4c-publishing, Zip, , 105bf055-c48e-4e16-92c3-364ca9a15a8a, {}, {}, []), Function: Function(name='Function', functionname='Function', runtime='nodejs14.x', memory=128, timeout=300, handler='aws-toolkit-ts-output/src/handler/publishingEnqueuerHandler.handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/aruizca/workspace/pepr/pepr-4-confluence/aws/p4c-publishing', environment={'Variables': {'NPM_TOKEN': 'YW5nZWw6SHk2Z3NONU9kUw=='}}, rolearn=None, layers=[], events=None, metadata=None, inlinecode=None, codesign_config_arn=None, stack_path=''))
2021-07-29 13:25:44,979 | Building codeuri: /Users/aruizca/workspace/pepr/pepr-4-confluence/aws/p4c-publishing runtime: nodejs14.x metadata: {} functions: ['Function']
2021-07-29 13:25:44,979 | Building to following folder /Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/build/Function
2021-07-29 13:25:44,980 | Loading workflow module 'aws_lambda_builders.workflows'
2021-07-29 13:25:44,986 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-07-29 13:25:44,988 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-07-29 13:25:44,992 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-07-29 13:25:44,995 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-07-29 13:25:44,998 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-07-29 13:25:45,001 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-07-29 13:25:45,004 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-07-29 13:25:45,007 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-07-29 13:25:45,011 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-07-29 13:25:45,011 | Found workflow 'NodejsNpmBuilder' to support capabilities 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-07-29 13:25:45,016 | Running workflow 'NodejsNpmBuilder'
2021-07-29 13:25:45,016 | Running NodejsNpmBuilder:NpmPack
2021-07-29 13:25:45,016 | NODEJS packaging file:/Users/aruizca/workspace/pepr/pepr-4-confluence/aws/p4c-publishing to /var/folders/6z/tqx78p3d7x7ftzx_5h2jk8zm0000gn/T/tmp4y7btrab
2021-07-29 13:25:45,016 | executing NPM: ['npm', 'pack', '-q', 'file:/Users/aruizca/workspace/pepr/pepr-4-confluence/aws/p4c-publishing']
2021-07-29 13:25:45,617 | NODEJS packed to aws-0.0.1.tgz
2021-07-29 13:25:45,618 | NODEJS extracting to /var/folders/6z/tqx78p3d7x7ftzx_5h2jk8zm0000gn/T/tmp4y7btrab/unpacked
2021-07-29 13:25:45,639 | NodejsNpmBuilder:NpmPack succeeded
2021-07-29 13:25:45,639 | Running NodejsNpmBuilder:CopyNpmrc
2021-07-29 13:25:45,639 | NodejsNpmBuilder:CopyNpmrc succeeded
2021-07-29 13:25:45,639 | Running NodejsNpmBuilder:CopySource
2021-07-29 13:25:45,664 | NodejsNpmBuilder:CopySource succeeded
2021-07-29 13:25:45,664 | Running NodejsNpmBuilder:NpmInstall
2021-07-29 13:25:45,664 | NODEJS installing in: /Users/aruizca/workspace/pepr/pepr-4-confluence/.aws-sam/build/Function
2021-07-29 13:25:45,665 | executing NPM: ['npm', 'install', '-q', '--no-audit', '--no-save', '--production', '--unsafe-perm']
2021-07-29 13:25:53,352 | NodejsNpmBuilder:NpmInstall succeeded
2021-07-29 13:25:53,353 | Running NodejsNpmBuilder:CleanUpNpmrc
2021-07-29 13:25:53,353 | NodejsNpmBuilder:CleanUpNpmrc succeeded

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

2021-07-29 13:25:53,369 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '41f2c7d9-5ec7-4962-8492-e4d386753c40', 'installationId': '10a33a59-c918-4b16-a48a-67e149e3c542', 'sessionId': 'f1ad66ed-40f9-4388-a83b-c74154c7ce7f', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.11', 'samcliVersion': '1.27.2', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 8611, 'exitReason': 'success', 'exitCode': 0}}]}
2021-07-29 13:25:54,189 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Thanks a lot!

aruizca commented 3 years ago

I tried to build inside the container and the problem is the same.

aruizca commented 3 years ago

I think the problem is that it is is assuming the node module with lambda is at the root of the repo, but this is not the case. Because the module is nested under relative path aws/p4c-publishing. Do you think I could workaround that somehow?

aruizca commented 3 years ago

This is my current "Run Configuration":

image

But the relative path to the root of the repo should be inside aws/p4c-publishing, but then the validation fails:

image
aruizca commented 3 years ago

Why the build does not contain the node_modules?

image