Are you considering a workflow with stages?

[jbrook]

I am still struggling a bit with how best to set up our development workflow with Lambda.

It seems that some tools provide some basic functionality to test the Lambda code with a command that executes it locally with event data being passed in from a local JSON file.

I haven't used it or really even researched it but JAWS seems to have support for stages and their documentation mentions:

The Framework uses Stages to partition sets of Lambdas, Resources and API Gateway stages. For clarity, every Stage gets its own:

Lambdas replication Resources replication (DynamoDb, IAM, etc.) API Gateway Stage on the Project’s REST API

The Framework helps you manage these stages, and even point to the Lambdas or Resources in one Stage within another. Further, new Lambda Versioning and Lambda Alias features allow easy promotion of each of your Lambdas to production, when they are ready.

Perhaps all of that is a little more complex than you have in mind for the Sparta project but it would be curious to hear your thoughts.

The personal use case that I have in mind is sending SNS notifications from some monolithic REST API code running on an EC2 instance inside a VPC. The SNS notifications have a lambda function subscribing to them to send transactional e-mails to users via SES. I am not really sure if I want to have separate Lambda functions and topics for each 'stage' but I need to work out how to make sure that the topics, lambdas and subscriptions are all properly provisioned and connected together. Sparta is helping with that by registering the lambda function and adding a permission, but it assumes that the SNS topic already exists. I suppose my development and testing Lambda's need to behave differently so that we don't accidentally try to e-mail a real person.

[mweagle]

Hi James. I agree - Sparta needs to be better support for Lambda development workflows. IIUC, there are at least three interrelated areas:

• Local dev/mock event testing
• Multiple geography support
• Promotion within an AWS (account/geo) pair

Also in case it helps address the SNS topic provisioning, v0.0.6 supports Template Decorators.

Can you describe your ideal workflow and where Sparta currently makes that difficult? Do you use multiple AWS accounts? Do you support/need multiple AWS geographies? Thanks.

[mweagle]

Hi @jbrook - JAWS just rebranded & posted docs on Stages that I'm taking a look at.

[jbrook]

I am going to miss their animated Jaws logo! It looks like they are putting a lot of hours into the project.

I see that they are dropping CloudFormation.

I think the versioning and alias features of Lambda are a powerful concept, although I haven’t even played with them yet. I guess the goal for a framework that supports stages would to be steer the user into a workflow that harnesses the power of Lambda and API Gateway but without getting them into a mess. It looks like Serverless makes each Lambda deployment 'stage-specific' by aliasing the version to a specific stage and including stage specific environment variables in the lambda function. They never alias the LATEST version to a stage. That makes the deployment unit immutable, which seems to make sense and works more or less like task definitions do in AWS ECS - something that has worked well for my container deployments. Do you think the concept of environment variables for configuration makes sense? I guess with a Go app you would want to compile the config into the binary with something like ‘esc'. Would it make sense to have the concept of project level config that could be overridden at the stage level or would that be too opinionated?

Also, what about some tooling to be able to run things locally? Or do you think with Lambda and AWS API Gateway it’s not going to be possible to emulate things locally?

What’s your general vision for Sparta?

On 9 Dec 2015, at 04:01, Matt Weagle notifications@github.com wrote:

Hi @jbrook https://github.com/jbrook - JAWS just rebranded & posted docs on Stages http://docs.serverless.com/docs/project-structure that I'm taking a look at.

— Reply to this email directly or view it on GitHub https://github.com/mweagle/Sparta/issues/2#issuecomment-163090751.

[mweagle]

I am going to miss their animated Jaws logo! It looks like they are putting a lot of hours into the project.

Yes - they've been very busy recently. It's great to see, since the larger the community behind AWS Lambda the better off we'll all be.

I see that they are dropping CloudFormation.

I do agree that CF isn't the most performant system, but it does have well defined semantics to maintain correctness & communicate state. My general perspective is that CloudFormation will continue to add new Resource Types over time. In the short term, Sparta uses Lambda-backed custom resources to fill the CloudFormation gaps. See the provision directory. The API Gateway custom resource is particularly thorny :)

I plan to either migrate these implementations to CloudFormation (which is ideal) or Go, whichever comes first.

I think the versioning and alias features of Lambda are a powerful concept, although I haven’t even played with them yet. I guess the goal for a framework that supports stages would to be steer the user into a workflow that harnesses the power of Lambda and API Gateway but without getting them into a mess.

Agreed - it's all too easy to get things into a mess.

It looks like Serverless makes each Lambda deployment 'stage-specific' by aliasing the version to a specific stage and including stage specific environment variables in the lambda function. They never alias the LATEST version to a stage. That makes the deployment unit immutable, which seems to make sense and works more or less like task definitions do in AWS ECS - something that has worked well for my container deployments.

Sparta doesn't use Lambda aliases, although it does have the ability to isolate resources by provisioning different ServiceName CloudFormation stacks. For instance, if you provision:

* `sparta.Main("jbrook", ...)`
* `sparta.Main("stage", ...)``
* `sparta.Main("prod", ...)`

From the "same" codebase, you'll get multiple Lambda functions deployed. I think an advantage of this approach is that the different stacks can subscribe to different event triggers. If those event triggers have different semantics (eg, DymamoDB schema changes), theres a spot to hook that up.

IMO, an even better form of isolation is to have multiple AWS Accounts (devs, stage, prod),

Do you think the concept of environment variables for configuration makes sense? I guess with a Go app you would want to compile the config into the binary with something like ‘esc'.

It does, but I agree that esc or something similar that allows pushing as much config/discovery into the binary is my default position. In my experience, minimizing external configuration produces more stable systems.

Would it make sense to have the concept of project level config that could be overridden at the stage level or would that be too opinionated?

Could something like that be handled at the application init phase?

Also, what about some tooling to be able to run things locally? Or do you think with Lambda and AWS API Gateway it’s not going to be possible to emulate things locally?

I've stubbed out local support but haven't had a time to finish it. Would testing over localhost HTTP work for your usecase?

What’s your general vision for Sparta?

To be a complete framework for deploying Go-based serverless apps. That will likely include features that aren't Lambda-specific (eg, provision an S3 bucket with static resources, R53 management, CloudWatch dashboard provisioning), but I definitely need it to support all core Lambda features. And to do so with DevOps & SecOps in mind.

I'm also looking for feedback to help grow it in the right directions.

Serverless does have a Trello board and I need to set one of those up :)

[mweagle]

@jbrook - I've pushed some changes that enable the explore command line option (go run main.go explore) as well as httptest for go test:

func TestIt(t *testing.T) {
    logger, _ := sparta.NewLogger("info")
    ts := httptest.NewServer(sparta.NewLambdaHTTPHandler(spartaLambdaFunctions(), logger))
    defer ts.Close()

    // Test code goes here, where the list of valid URLs is http://localhost::{PORT}/{LambdaFunctionName}
}

The LambdaFunctionName is available via the explore options list, or programmatically via:

    lambdaPtr := runtime.FuncForPC(reflect.ValueOf(myLambdaFunction).Pointer())
    lambdaFnName :=  lambdaPtr.Name(),

Would an approach like this be helpful for local testing?

[jbrook]

@mweagle. I haven't been keeping up with you. Your goals for Sparta are exciting.

I am going to play with the explore and httptest commands really soon. Thank you.

In answer to some of your earlier questions - we are operating in a single AWS region with a single AWS account and our current needs are really rather simple at the moment. We are just triggering a couple of Lambda functions from an application that runs on EC2 instances. We do that by publishing events to SNS topics.

I need to do a crash course in Cloudformation. I played with the template decorator functionality and managed to create some topics but I failed to work out how to subscribe my Lambdas to them. An example would be really useful if it's a relatively easy thing to do.

When I feel like I know what I am talking about a bit more I think I'll write an article about my experiences with Lambda and Sparta.

[mweagle]

I am going to play with the explore and httptest commands really soon. Thank you.

LMK if it helps. I need to write up some docs on the HTTP request body, but in the short term at a minimum you'll need something like:

{
"context": {...},
"event" : {...}
}

which matches the proxying request.

In answer to some of your earlier questions - we are operating in a single AWS region with a single AWS account and our current needs are really rather simple at the moment. We are just triggering a couple of Lambda functions from an application that runs on EC2 instances. We do that by publishing events to SNS topics.

Good to know - I'd like to hold off multiple geo support until workflows & signatures stabilize a bit.

I need to do a crash course in Cloudformation. I played with the template decorator functionality and managed to create some topics but I failed to work out how to subscribe my Lambdas to them. An example would be really useful if it's a relatively easy thing to do.

That's a good use case and not one that is straightforwardly exposed right now. The problem is that the SNS APIs are required to manage the subscription, but the SNSPermission type doesn't support dynamically created topics. I have some ideas on how to address that - for now is it ok to use preexisting topic ARNs?

When I feel like I know what I am talking about a bit more I think I'll write an article about my experiences with Lambda and Sparta.

I would really appreciate that. Looking forward to reading more - thanks!

[mweagle]

@jbrook -Release 0.1.1 adds support for dynamic provisioning with Lambda subscription. See the SpartaApplication for an example.

[mweagle]

@jbrook - Release 0.1.2 provides better support for localhost testing. See explore_test for an example.

It still needs to be a bit easier to determine exactly what to send in the event payload.

Also, this issue touches on a lot of good points, but it's getting difficult to track them all. I'm going to close this, with specific issues/Trello cards to follow for outstanding issues. Please open new issues in the meantime if you're running into problems/feature gaps.

[mweagle]

@jbrook - Checking back to see if you've had a chance to write down any thoughts about Sparta use/future/pain points. Would appreciate your feedback. Thanks.