search

elastic / elastic-serverless-forwarder

Elastic Serverless Forwarder
Other
36 stars 36 forks source link

SAR: Terraform support #22

Closed aspacca closed 2 years ago

aspacca commented 3 years ago

We should include a way to add the required IAM permissions/policies.

damien commented 2 years ago

My company is in the process of provisioning ESF via Terraform this weekend. We've found a few friction points that may be worth considering:

Happy to chat more about this. My company is really interested in making deployment of ESF via Terraform an easy process. You can reach me at damien.wilson at humaninterest dot com.

aspacca commented 2 years ago

hello @damien , glad to hear you are using ESF.

  • ESF requires a bucket to read from for the sar-application.yml configuration file. Terraform deployment would be much simpler if these configurations could be specified as part of the Lambda environment and via AWS secrets manager.

how does exactly your terraform definition look like? is it a single step deployment or one the goes through an initial deployment and importing the state of the deployed lambda?

so far I've only experimented a little with the terraform support for ESF, and from what I can see the aws_serverlessapplicationrepository_cloudformation_stack (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/serverlessapplicationrepository_cloudformation_stack) resource doesn't allow the same level of customisation/automation as the lambda_function one (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function)

I have to figure out how to solve this problem but I would expect a similar solution to what's currently documented for deployment through cloudformation:

  1. apply the SAR stack definition
  2. import the lambda state
  3. customise definition from the state
  4. apply the customised definition

this will probably requires some scripting, without being able to rely only on terraform apply

  • Deploying ESF as a SAR via Terraform is not straightforward, nor is it clear how to pin our ESF deployment to a specific revision. We ended up building this repo and bundling a zip of EST at 7fff40d to avoid changes to master or the SAR application from breaking any live ESF instances we have within our AWS accounts.

for concerns pinning the terraform definition to a specific version you can rely on semantic_version argument in the definition (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/serverlessapplicationrepository_cloudformation_stack#semantic_version)

currently releases are not tagged in the repo, and we still have to decide a strategy regarding this, since when we will extended support to serverless from different cloud providers, a single version fitting all of them will not be most likely be available.

damien commented 2 years ago

Hey @aspacca, thanks for the response!

how does exactly your terraform definition look like? is it a single step deployment or one the goes through an initial deployment and importing the state of the deployed lambda?

so far I've only experimented a little with the terraform support for ESF, and from what I can see the aws_serverlessapplicationrepository_cloudformation_stack (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/serverlessapplicationrepository_cloudformation_stack) resource doesn't allow the same level of customisation/automation as the lambda_function one (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function)

We created our own Terraform modules loosely based off of the Cloudformation template you mentioned in the docs, though we opted to try to deploy the lambda directly rather than provision it via SAR. It was a long busy weekend, so I'll need to get back to you on the specific reasons why we did that after consulting some notes, but we ran into some kind of issue wiring the SAR app with other infrastructure components when done through Terraform code that deploying the forwarder via a lambda_function resolved for us.

Sadly, to deploy as a lambda_function resource we needed to specify a function payload. I (debatably) got around by using .internal/aws/scripts/dist.sh#L31-L50 from commit 58d81b12ffc0630f0000af73316649b43db8f1ae (master at the time).

The problem with this approach today is that I have no idea if this is the correct revision to build from to end up with a payload identical to the one used by the public SAR application.

is it a single step deployment or one the goes through an initial deployment and importing the state of the deployed lambda?

The benefit to this approach is that it is single step deploy and doesn't depend on multiple terraform apply steps; we just need to know the appropriate lambda payload ahead of time. :)

currently releases are not tagged in the repo, and we still have to decide a strategy regarding this, since when we will extended support to serverless from different cloud providers, a single version fitting all of them will not be most likely be available.

Adding a commit SHA to the metadata of the SAR distribution or some kind of "published-$TIMESTAMP" style tag on deployed commits would help me out here if decisions around semver and release management are going to take more time to figure out.

Thanks for your time @aspacca! Let me know if there's any more info I can share that would be interesting to you. Based on your commentary I think we're addressing this problem in similar ways. Looking forward to hearing more of your thoughts!

aspacca commented 2 years ago

@damien

We started tagging lambda versions from main (https://github.com/elastic/elastic-serverless-forwarder/tags)

It is a temporary solution to pin to a specific version of the lambda flow picture in the SAR readme (https://github.com/elastic/elastic-serverless-forwarder/blob/main/docs/lambda-flow.png), please, consider it a strategy that might change once we go GA.

damien commented 2 years ago

@aspacca Thank you for the response. The tagging has been useful to derive the appropriate version of ESF to deploy with our own Terraform code, though there are still some gaps here I suspect you and the fine folks working on this project address as you near GA (outlined in my previous responses).

Ideally a GA release will be deployable via Terraform in a single run without needing to resort to state manipulation. Doing this is possible, it just requires a bit more formalization of how peripheral dependencies and configurations of ESF are deployed (AWS IAM roles, definition and association of relevant SQS queues and events, etc).

Happy to help if you have new releases or changes you'd like feedback on.

Cheers!

aspacca commented 2 years ago

hello @damien the problem in defining AWS IAM roles in the template is that users must opt-in in SAR to access applications that define such roles.

I'm currently looking to have same Params for the ARNs of event sources and subscriptions, according to the type (sqs, kinesis, cloudwatch logs) and with a macro expand them directly in the template: I this case I would expect default AWS IAM roles for the Lambda will be added and therefore no explicit opt-in in required to access the application. I still have to test this is the case.

Having that it should be possible to use https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/serverlessapplicationrepository_cloudformation_stack to deploy the application with terraform

aspacca commented 2 years ago

hello @damien

I'm closing this issue as terraform deployment is now supported: please, see the instructions at https://github.com/elastic/elastic-serverless-forwarder/blob/lambda-v0.30.0/docs/README-AWS.md#terraform