sam build fail with "No such file or directory" for node build

Cubid commented 5 years ago

Description

When building with sam build --template sam-template.yaml the build fail due to non existent temporary folder

Steps to reproduce

run sam build --template sam-template.yaml with js source in a scub package

Observed result

Running workflow 'NodejsNpmBuilder' Running NodejsNpmBuilder:NpmPack NODEJS packaging file:/authorizer to /tmp/tmpmn3v_qa7 executing NPM: ['npm', 'pack', '-q', 'file:/authorizer'] NODEJS packed to > backlinks-authorizer@1.0.0 prepare /authorizer

npm run build

backlinks-authorizer@1.0.0 build /authorizer tsc --outDir build

backlinks-authorizer-1.0.0.tgz NODEJS extracting to /tmp/tmpmn3v_qa7/unpacked NodejsNpmBuilder:NpmPack raised unhandled exception Traceback (most recent call last): File "/home/xxx/.local/lib/python3.6/site-packages/aws_lambda_builders/workflow.py", line 248, in run action.execute() File "/home/xxx/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 67, in execute self.osutils.extract_tarfile(tarfile_path, self.artifacts_dir) File "/home/xxx/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/nodejs_npm/utils.py", line 23, in extract_tarfile with tarfile.open(tarfile_path, 'r:*') as tar: File "/usr/lib/python3.6/tarfile.py", line 1571, in open return func(name, "r", fileobj, **kwargs) File "/usr/lib/python3.6/tarfile.py", line 1636, in gzopen fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj) File "/usr/lib/python3.6/gzip.py", line 163, in init fileobj = self.myfileobj = builtins.open(filename, mode or 'rb') FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpmn3v_qa7/> backlinks-authorizer@1.0.0 prepare /authorizer\n> npm run build\n\n\n> backlinks-authorizer@1.0.0 build //authorizer\n> tsc --outDir build\n\nbacklinks-authorizer-1.0.0.tgz'

Build Failed

Expected result

The temp directory (and sub dirs) should be created for the script to continue

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Ubuntu 18.04.3
sam --version: SAM CLI, version 0.22.0

Add --debug flag to command you are running

Cubid commented 5 years ago

Seems like filename is the whole output of npm build command. Mine is:

> backlinks-authorizer@1.0.0 prepare <build-dir>/authorizer
> npm run build

> backlinks-authorizer@1.0.0 build <build-dir>/authorizer
> tsc --outDir build

backlinks-authorizer-1.0.0.tgz

it seems it is a issue in npm silent option

# npm pack -q file:<build-dir>/authorizer

> backlinks-authorizer@1.0.0 prepare <build-dir>/authorizer
> tsc --outDir build

backlinks-authorizer-1.0.0.tgz

$ npm pack -s file:<build-dir>/authorizer
backlinks-authorizer-1.0.0.tgz

it seems that changing the flag from -q to -s in aws_lambda_builders/workflows/nodejs_npm/actions.py solve the issue

awood45 commented 5 years ago

Can you expand a bit on the steps to reproduce? If the fix is simple I'd want to fast track it, but I'm not sure I understand the conditions under which I can reproduce this to confirm the fix is correct.

Cubid commented 5 years ago

I implemented a small project based on the default node sam app: https://github.com/Cubid/aws-sam-cli-1426 And I made the breaking change here: https://github.com/Cubid/aws-sam-cli-1426/commit/61d48189aa483e3b3dc53794b9b0759b4217f314

On my side I use TypeScript compiler, but it is the same for any command added in the build.

sam build --template template.yaml is crashing on this project

seriojaignat commented 5 years ago

I also work with TypeScript and my sam build fails because the dist folder is not copied to .aws-sam folder. The reason is that dist is listed in .gitignore and thereby ignored by the sam build command (why?). The ugly solution is to remove dist or any output folder from .gitignore. It is normal for dist or any output directory to be ignored by the code management tool, but it is necessary for those folders to be part of the package. How can I configure sam build to not ignore the dist folder or any other output folder used by my compiler or transpiler?

awood45 commented 5 years ago

@Cubid Thanks for the example, I'll take a look at that now.

@seriojaignat Our JS builder is specific for Node, and runs npm pack - are you compiling from TS to JS before you run sam build?

awood45 commented 5 years ago

@Cubid it looks like what you're trying to do is override the prepack step - I know what you gave me is a reproducing example, which it does reproduce, but is the idea here that you're trying to shim in TypeScript support?

awood45 commented 5 years ago

Also, what are you actually trying to put in the prepack step?

awood45 commented 5 years ago

Ok, here's a bit of background to help out. https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/nodejs_npm/DESIGN.md contains the design of the Node builder that's run by sam build.

@seriojaignat we run npm pack which I understand ignores files in your .gitignore by default, so you may want to add an .npmignore file that has a narrower scope and see if that includes the files you want built.

@Cubid What version of node are you running? I'm doing some research and it does look like prepack can have issues with pack depending on version (though I'm not a node expert myself so I'm researching in parallel here).

Cubid commented 5 years ago

@awood45 In the script section of my package.json, I have the following line: "prepare": "tsc --outDir build" It's not link to prepack, but to any script that will output something Since the whole output is taken as file name, the file name is not correct.

awood45 commented 5 years ago

Understood - we have some features in our backlog that should make this easier in general, but I'm looking in to what to recommend for this use case.

mildaniel commented 2 years ago

We have released native TypeScript support through esbuild. This should resolve this issue. Please re-open if anything else comes up.

aws / aws-sam-cli