Package lambdas locally without uploading to S3

[tomislacker]

Background

Given the current paradigm as of v1.13, the Lambdas get packaged then uploaded to S3 and the source template is modified and outputted to a new destination. This may have the possibility of causing a problem with any changes that AWS implements with the SAM extensions.

Proposal

Instead of immediately zipping and uploading contents to S3 of the packaged Lambda, they are written to disk. For example the following source contents of a project:

$ tree
.
|-- src
|   |-- mything
|   |   `-- __init__.py
|   `-- requirements.txt
`-- template.yml

2 directories, 3 files

Assume I had two functions in my template.yml named FuncA and FuncB. The possum entrypoint would optionally take something like --output-directory dist/ and then it creates something like:

$ tree
.
|-- dist
|   |-- FuncA
|   |   |-- mything
|   |   |   `-- __init__.py
|   |   `-- requirements.txt
|   `-- FuncB
|       |-- mything
|       |   `-- __init__.py
|       `-- requirements.txt
|-- src
|   |-- mything
|   |   `-- __init__.py
|   `-- requirements.txt
`-- template.yml

7 directories, 7 files

This would then allow the use of possum first, then aws cloudformation package to operate normally without interfering with it (such as #7 ). It would also reduce the scope of liability on this project in modifying the original template contents.

[brysontyrrell]

This is planned. It ties into #4

[tomislacker]

With sam local functionality, the template would need a special version created for it, at least as far as how I've proposed the output directory structure in this issue. Would you be okay with that?

I'd also be proposing to change up the usage. Something along these lines:

1a2
>               [-b s3_bucket] [-d destination_dir]
9,10c10,17
< positional arguments:
<   s3_bucket             The S3 bucket to upload artifacts
---
> required arguments:
>   -b s3_bucket          The S3 bucket to upload artifacts
>   -d destination_dir    The directory to locally package artifacts
>                         Within here, the logical identifier of each
>                         Serverless::Function will have a respective
>                         subdirectory where its contents are loaded
> 
>   At least one of -b, -d is required.

I'd like to take on this effort if you're okay with all the above (or we can determine and reasonable design). I think this would get you most of the way towards #4.

[brysontyrrell]

@tomislacker for use with the SAM CLI we'll still need an output template that references the location of the zipped Lambda packages for invoking tests.

My original thought for supporting testing was to specify a tests directory within the Lambda's directory that contains the JSON for the test event to invoke the function with and then provide a --test option that will loop through all Lambdas in the template invoking them using the events found within that directory.

Thoughts?

[tomislacker]

At this point, I've been only using aws cloudformation package and not the actual sam CLI utility so I'd have to mull on it a bit and try it out. Sorry I don't have much of an opinion at the moment.

[brysontyrrell]

Won't the same still be true for using aws cloudformation package/deploy? The referenced template would need to contain the paths to the generated packages. I think testing and deployment for CI/CD will require the same work/changes.

[tomislacker]

Oh yea it would. I was imagining that the template would be already prepared for that. So something like:

FunctionName:
  # ...
  CodeUri: ./output/FunctionName/

[brysontyrrell]

The workflow I'm thinking about would preserve the current expectation of a directory path as the template is how Possum determines the location of the source code it is building packages for:

Source: template.yaml

FunctionName:
    # ...
    CodeUri: ./src/FunctionName/

Output from Possum: packaged-template.yaml

FunctionName:
    # ...
    CodeUri: ./dist/FunctionName/{Package}.zip

[tomislacker]

Understood. I was coming from the perspective that with my proposal, Possum wouldn't need to manipulate the source template at all. It would just be a tool for reading in the dependencies and outputting a directory structure like what SAM/CloudFormation are already able to package and ship to S3.

[brysontyrrell]

I feel that going that route would require developers using Possum to adhere to a directory structure and/or naming convention for directories in their projects so the utility is able to locate and package the functions.

I did that in my original version of this script and then switched to following the SAM spec of referencing the path in CodeUri to allow choice and flexibility in source location. I want to iterate features that keep that flexibility.