Automate deployments to SAR

girodav commented 1 year ago

Create a Github Workflow that automatically release Elastic Serverless Forwarder to SAR, when a new version tag is pushed

Relevant doc: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#running-your-workflow-only-when-a-push-of-specific-tags-occurs

v1v commented 1 month ago

I think we can help with this. We have configured OIDC access to AWS accounts in GitHub actions (see https://github.com/elastic/observability-robots/blob/main/docs/teams/ci/keyless/README.md), so there is no need to use any secrets but configure the automation.

If you can provide how you deploy to SAR currently and the steps you do now, then we can figure out the bits and pieces that are required.

Thanks

zmoog commented 1 month ago

Hey @v1v, thanks! I'm sharing this with the team.

bturquet commented 1 month ago

@v1v that would be good news if you could help us on that. Would it be possible to have this automation for both "regular" SAR and GovCloud SAR ? We are struggling a lot to have the proper permissions for the latest.

@zmoog @kaiyan-sheng would it make sense to move this issue to our current iteration to make sure that we could provide Victor the relevant information ?

zmoog commented 1 month ago

would it make sense to move this issue to our current iteration to make sure that we could provide Victor the relevant information ?

Yep, definitely. We already have the information Victor needs; I'll pick them all and share them here.

zmoog commented 1 month ago

@v1v, I have a public note https://github.com/zmoog/public-notes/issues/15 that describes how to publish ESF on SAR as a private application (a good option for testing).

The process for publishing ESF as a public application remains the same. We need to use the same .internal/aws/scripts/dist.sh script with different parameters.

The parameters to change are (at least):

AWS account (we possibly need different account IDs for the non-GovCloud public version)
AWS region (we published ESF on eu-central-1 first, so we need to keep using it)

We may need to repeat the publishing with different settings for AWS GovCloud (US): @kaiyan-sheng probably knows more than me.

I see we already have a GitHub workflow to upload the .zip file for the Terraform installation method at https://github.com/elastic/elastic-serverless-forwarder/blob/main/.github/workflows/upload-dependencies.yml.

We can probably change the like https://github.com/elastic/elastic-serverless-forwarder/blob/196240566f7f59bad7bc952063cc80b563b5a3a2/.internal/aws/scripts/dist.sh#L73-79 since we are already building a .zip file in an S3 bucket for the Terraform installation method.

The release workflow should publish the artifacts on Terraform and SAR channels, right?

zmoog commented 1 month ago

Thanks to @girodav for double-checking with me and integrating this info before posting ❤️

v1v commented 1 month ago

Hi folks,

I'm now back from PTO and we just started a new iteration, aka sprint, so I plan to work on this in the upcoming days, I'll keep you posted.

zmoog commented 1 month ago

I'm now back from PTO and we just started a new iteration, aka sprint, so I plan to work on this in the upcoming days, I'll keep you posted.

Hey @v1v, thank you for taking this task! Let us know if you need anything!

v1v commented 1 month ago

I've just created https://github.com/elastic/elastic-serverless-forwarder/pull/758

I need some help with a few things:

the expected parameters when running .internal/aws/scripts/dist.sh for a release.
when to run the SAR deployment for a release. I assumed I could use the release workflow and include a new job to run for the given git tag.

Would it be possible to have this automation for both "regular" SAR and GovCloud SAR ? We are struggling a lot to have the proper permissions for the latest.

I'm unfamiliar with GovCloud SAR, so it will probably be postponed in a follow-up if we can even automate it.

I see we already have a GitHub workflow to upload the .zip file for the Terraform installation method at main/.github/workflows/upload-dependencies.yml.

How does it work at the moment? IIUC, upload-dependencies can accidentally publish a version since the checkout is not fixed to the tag but main, so someone could accidentally push a new commit while the release workflow runs.

Maybe it can be combined with the release workflow so we can ensure the sha commit when the release runs is the one used for generating the artifacts in the s3 bucket.

We can probably change the like 1962405/.internal/aws/scripts/dist.sh#L73-79 since we are already building a .zip file in an S3 bucket for the Terraform installation method. The release workflow should publish the artifacts on Terraform and SAR channels, right?

Can you contribute to my above-mentioned PR with the required changes please?

v1v commented 4 weeks ago

Status update

I enabled the OIDC targeting the elastic-observablity dev AWS account and create my s3 bucket for testing my changes.

Then I created a branch based on my PR but removing some bits and pieces that were not needed, and I finally ran:

https://github.com/elastic/elastic-serverless-forwarder/actions/runs/10286873394/job/28470615564

It worked fine after discovering the required permissions to publish the application with sam publish and

https://us-east-1.console.aws.amazon.com/serverlessrepo/home?region=us-east-1#/published-applications/arn:aws:serverlessrepo:us-east-1:627286350134:applications~elastic-serverless-forwarder

Therefore, the only missing bits and pieces are the questions I asked above.

Cheers

zmoog commented 2 weeks ago

Would it be possible to have this automation for both "regular" SAR and GovCloud SAR ? We are struggling a lot to have the proper permissions for the latest.

I'm unfamiliar with GovCloud SAR, so it will probably be postponed in a follow-up if we can even automate it.

Yeah, we discussed this during a team meeting and agreed it's better to focus on automating SAR first and then evaluate the gap later.

v1v commented 2 weeks ago

Thanks @zmoog

Do you happen to know who can help me with some of the questions I have (see https://github.com/elastic/elastic-serverless-forwarder/issues/279#issuecomment-2271594305)? If it helps, we can syncup in a zoom call

Thanks

zmoog commented 2 weeks ago

Do you happen to know who can help me with some of the questions I have (see #279 (comment))?

Yeah, I can help with this.

I'm sorry for the delay. I started working on this but got interrupted, so I'm resuming today!

If it helps, we can syncup in a zoom call

Thanks! Let's see how far I can go. I won't hesitate to contact you for a call if it helps!

zmoog commented 2 weeks ago

I need some help with a few things:

the expected parameters when running .internal/aws/scripts/dist.sh for a release.

I have a public note https://github.com/zmoog/public-notes/issues/15 describing the publishing process of ESF on SAR as a private application. IIRC, the process is identical, and we need to use a specific AWS account and region for publishing it as a public application.

Here's the command line format:

.internal/aws/scripts/dist.sh \
    sar-app-name \
    version-in-sar
    s3-bucket-for-sar \
    account-id \
    sar-region \
    "author name"

Here is the parameter info I remember or found in my notes. I'll update this table as I get more details.

Parameter	Notes	Example values	Public release value
sar-app-name	The SAR application name.	`elastic-serverless-forwarder` or `elastic-serverless-forwarder-<YOUR NAME>`	Probably the name is `elastic-serverless-forwarder` — @constanca-m, you was probably the last person to release ESF, did you use this app name?
version-in-sar	The SAR application version.	Possible values are `1.8.0`.	IIRC, the latest version can be found in the `share/version.py` file.
s3-bucket-for-sar	S3 bucket that the publishing process uses to store artifacts.	`zmoog-elastic-serverless-forwarder-artifacts`
account-id	AWS account ID where to publish the SAR application.
sar-region	`eu-west-1`	`eu-central-1` — @constanca-m, did you use this region, right?
author name	The SAR author name	Elastic	Elastic

zmoog commented 2 weeks ago

when to run the SAR deployment for a release. I assumed I could use the release workflow and include a new job to run for the given git tag.

Yep, I would extend the current workflow for releasing the Terraform artifacts with a job for SAR!

zmoog commented 2 weeks ago

I see we already have a GitHub workflow to upload the .zip file for the Terraform installation method at main/.github/workflows/upload-dependencies.yml.

How does it work at the moment?

@constanca-m, please correct me if I'm wrong. This workflow packages ESF as a release .zip file. It is designed to deploy ESF using the Terraform module at https://github.com/elastic/terraform-elastic-esf and is unrelated to SAR.

IIUC, upload-dependencies can accidentally publish a version since the checkout is not fixed to the tag but main, so someone could accidentally push a new commit while the release workflow runs.

Oh, good point! We need to fix this then!

Maybe it can be combined with the release workflow so we can ensure the sha commit when the release runs is the one used for generating the artifacts in the s3 bucket.

Sounds great.

zmoog commented 2 weeks ago

We can probably change the like 1962405/.internal/aws/scripts/dist.sh#L73-79 since we are already building a .zip file in an S3 bucket for the Terraform installation method. The release workflow should publish the artifacts on Terraform and SAR channels, right?

Can you contribute to my above-mentioned PR with the required changes please?

I need to check whether we can use the exact .zip file for SAR and Terraform or if they differ.

constanca-m commented 2 weeks ago

Yes to your questions, I followed the script the last time I updated SAR like this:

AWS_PROFILE=elastic-observability-prod .internal/aws/scripts/[dist.sh](http://dist.sh/) elastic-serverless-forwarder 1.14.0 elastic-serverless-forwarder 267093732750 eu-central-1

v1v commented 1 week ago

Status update

https://github.com/elastic/elastic-serverless-forwarder/pull/758 is ready for review. I kept what we discussed in https://github.com/elastic/elastic-serverless-forwarder/issues/279#issuecomment-2304870149; I mentioned in the PR that it might be done in a follow-up. I prefer to keep it simple for now and iterate in small PRs.

v1v commented 1 day ago

Hi folks,

Can I get some more eyes for https://github.com/elastic/elastic-serverless-forwarder/pull/758? I don't know if it can be merged as is or if you have any concerns, but I need you to validate if the implementation is what you had in mind. In addition, if the arguments are the expected ones when using .internal/aws/scripts/dist.sh

Thanks

elastic / elastic-serverless-forwarder

Automate deployments to SAR #279

Status update

Status update