elastic / elastic-serverless-forwarder

Elastic Serverless Forwarder
Other
36 stars 36 forks source link

Make Elastic Serverless Forwarder available in AWS GovCloud regions #181

Open ravikesarwani opened 2 years ago

ravikesarwani commented 2 years ago

Currently, Elastic Serverless Forwarder(ESF) is not available in AWS GovCloud regions.

From the AWS docs "Applications that are publicly shared in other AWS Regions are not automatically available in AWS GovCloud (US) Regions. To make applications available in AWS GovCloud (US) Regions, you must publish and share them independently of other AWS Regions".

This issue is to publish ESF in AWS SAR to be made available on the AWS GovCloud(US) Regions.

There are certain requirements around who can have access to AWS GovCloud(US) Regions. For example: "AWS GovCloud (US-East) and (US-West) Regions are operated by employees who are U.S. citizens on U.S. soil. AWS GovCloud (US) is only accessible to U.S. entities and root account holders who pass a screening process. Customers must confirm that they will only use a U.S. person (green card holder or citizen as defined by the U.S. Department of State) to manage and access root account keys to these regions."

### Related
- [ ] https://github.com/elastic/enhancements/issues/17413
- [ ] https://github.com/elastic/enhancements/issues/18231
michaelmagyar commented 1 year ago

Adding some thoughts as Functionbeat is deprecated with support ending in less than a year, and federal clients and their service providers will need extensive time to plan, implement, re-document, and possibly get re-audited on the change (i.e., they may have to add SQS, Secrets Manager, etc. to their packages if they weren't using them before).

Elastic currently has a GovCloud account cluster that supports the FedRAMP offering. Because Elastic is already approved for GovCloud, the team that handles that account cluster should be able to either add ESF publishing directly to the existing GovCloud account(s) or create a new GovCloud account specifically for ESF publishing. The issue is compliance, and that needs to be thought of first in this case.

It may be possible to immediately add ESF to the existing GovCloud accounts. However, there might be some compliance issues with immediately adding additional functionality to that environment because that might change the ATO package and require a significant change request/re-audit.

If that is going to take some time (it could easily be 6-12 months), then an alternative is to:

If that flow ends up happening and ESF is published to GovCloud outside of the existing federal offering, some entities still may not be able to leverage it for compliance reasons, but they would at least have the ability to make the risk-based decision and/or test that it works with time to implement it properly.

I think the steps are likely:

I hope this happens quickly and that it can be immediately added to the existing FedRAMP accounts/package, but I am not optimistic.

aspacca commented 1 year ago

@michaelmagyar I'd need to pair with someone with access to the GovCloud account cluster with the proper permission to publish an app on SAR.

They'll have to publish the forwarder and maintain it on the GovCloud account. From the technical requirements there should not be much to do, but I cannot do anything on my side without pairing with them and transfer them the knowledge they require.

The same if another GovCloud account is requested.

Let's just arrange the proper point of contact and I don't see, from the technical point of view, any great blockers on this.

michaelmagyar commented 1 year ago

I see that this issue has not been resolved for almost a year. What is the current plan to support serverless log forwarding in GovCloud? Go Lambda support is being discontinued at the end of this year, so functionbeats will no longer be viable, even as a deprecated option, without a lot of changes.

Is the plan really to force GovCloud clients to add VMs just to collect logs? What about clients that don't currently use/authorize VMs in their environment?

bturquet commented 1 year ago

Hey @michaelmagyar, we still have the plan to support serverless log forwarding in GovCloud. We are working with Legal Team to have the Sponsor approval to make it happen.

More info here: https://github.com/elastic/infosec/issues/14266

michaelmagyar commented 10 months ago

Hello Elastic. Can you please provide an update to this? I see that PR #510 was merged. However, although there is a published ESF package on AWS SAR, the publisher is not verified and the helper/nested applications are missing.

What is the current status/timeline?

We were told that our current spend of 5 figures a month is not large enough to have access to a technical account manager except for renewals, so this appears to be the only route for us to get updates on this outside of support.

Note: functionbeat is no longer deployable given that AWS has disabled the Go runtime for new functions, so continuing to use that would require rebuilding the package using AL2 runtime and adding Go on top.

bturquet commented 9 months ago

Hi @michaelmagyar , we are still waiting for internal permissions to be granted for us, to deploy last ESF version in GovCloud SAR. The ETA is in 2 weeks.

michaelmagyar commented 8 months ago

Hello @bturquet , are there any updates on the ETA for ESF in GovCloud?

aspacca commented 8 months ago

hi @michaelmagyar , we are still blocked on having proper access to GovCloud in order to make ESF available there. We'll send an update next week

kaiyan-sheng commented 8 months ago

@michaelmagyar I was able to get the proper permission this week and I changed helper applications to public. So you should be able to see them now.

However, although there is a published ESF package on AWS SAR, the publisher is not verified and the helper/nested applications are missing.

For the publisher not being verified, we are still working with the AWS side to get that solved.

We have not published a new version of ESF yet due to some other issues. Will update this ticket once we are able to publish a new version.

kaiyan-sheng commented 1 day ago

Last time I had proper permission to access govcloud, I was able to change ESF helper applications to public. But since then, we had multiple releases so we should push the latest release to govcloud and also might be a good time to start looking into how we can automate this process.