[AC-4/SC-7] Manage egress traffic from SSB Broker Apps

[nickumia-reisys]

User Story

In order to increase our security posture, the Data.gov SSB Team wants to implement an egress-proxy in conjunction with our SSB Apps (ssb-solr, ssb-eks and ssb-smtp).

Acceptance Criteria

• [ ] GIVEN the egress proxy has been configured \ WHEN a request is sent from the SSB Broker \ THEN it is blocked/allowed in accordance with egress policies

Background

Historical Issues/References

• https://github.com/GSA/data.gov/issues/3755
• https://github.com/GSA/data.gov/issues/3651

Security Considerations (required)

To satisfy POAM

Sketch

• [x] Create design for deploying https://github.com/GSA/cg-egress-proxy with https://github.com/GSA/datagov-ssb/
• [x] Decide whether we can de-couple the EKS broker and remove it, so that we don't need the extra egress-proxy
• [ ] ~Modify Terraform code to deploy egress-proxy (other solutions are welcome, however, this is the most integrated)~

[jbrown-xentity]

my bad, wrong ticket

[nickumia-reisys]

Having explored a few options, I've decided that my original sketch is too complicated. It might be more logical to keep the same deployment procedure that we've been using for our other apps. Irrespective of this fact, the following steps need to be taken (admin -- @btylerburton @Jin-Sun-tts @jbrown-xentity):

• [x] Create development-ssb-egress space
• [x] Create management-staging-egress space
• [x] Create management-egress space

---

Since we have the infrastructure already setup and optimized in this repo, I would suggest to add the ssb apps to this repo, but the two options are:

• [ ] ~Extend the github actions in data.gov repo to include ssb-eks, ssb-smtp and ssb-solrcloud apps in the development-ssb-egress, management-staging-egress and management-egress spaces~
• [x] Re-create the github action infrastructure in the datagov-ssb repo.
  • This would help to abstract the different aspects of "Data.gov" and "SSB Team", but it just makes it a bit more complex to maintain.

---

Create space-deployer accounts in egress spaces (@btylerburton @Jin-Sun-tts)

• [x] cf create-service cloud-gov-service-account space-deployer ci-deployer (development-ssb-egress)
• [x] cf create-service cloud-gov-service-account space-deployer ci-deployer (management-staging-egress)
• [x] cf create-service cloud-gov-service-account space-deployer ci-deployer (management-egress)
• [x] cf create-service-key ci-deployer datagov-ssb-deployer (development-ssb-egress)
• [x] cf create-service-key ci-deployer datagov-ssb-deployer (management-staging-egress)
• [x] cf create-service-key ci-deployer datagov-ssb-deployer (management-egress)

Each key needs to be added to the Github Secrets in datagov-ssb

• [x] @btylerburton @Jin-Sun-tts Please also verify that there is another CF user/pass combo for development-ssb, management-staging and management.

---

Next step (regardless of last decision) is to:

• [ ] Determine the allow + deny lists for ssb-eks
• [ ] Determine the allow + deny lists for ssb-smtp
• [ ] Determine the allow + deny lists for ssb-solrcloud

*Note: I would consolidate these lists if practical.

---

Deploy all egress apps:

• [ ] ssb-eks (management-egress)
• [ ] ssb-eks (management-staging-egress)
• [ ] ssb-eks (development-ssb-egress)
• [ ] ssb-smtp (management-egress)
• [ ] ssb-smtp (management-staging-egress)
• [ ] ssb-smtp (development-ssb-egress)
• [ ] ssb-solrcloud (management-egress)
• [ ] ssb-solrcloud (management-staging-egress)
• [ ] ssb-solrcloud (development-ssb-egress)

---

Once the apps are deployed and functioning, I believe all that's left to do is:

• [ ] Create network policy for ssb-eks (management)

  cf add-network-policy ssb-eks proxy-gsa-datagov-management-ssb-eks -s management-egress -o gsa-datagov --protocol tcp --port 8080

• [ ] Create network policy for ssb-eks (management-staging)

  cf add-network-policy ssb-eks proxy-gsa-datagov-management-staging-ssb-eks -s management-staging-egress -o gsa-datagov --protocol tcp --port 8080

• [ ] Create network policy for ssb-eks (development-ssb)

  cf add-network-policy ssb-eks proxy-gsa-datagov-development-ssb-ssb-eks -s development-ssb-egress -o gsa-datagov --protocol tcp --port 8080

• [ ] Create network policy for ssb-smtp (management)

  cf add-network-policy ssb-smtp proxy-gsa-datagov-management-ssb-smtp -s management-egress -o gsa-datagov --protocol tcp --port 8080

• [ ] Create network policy for ssb-smtp (management-staging)

  cf add-network-policy ssb-smtp proxy-gsa-datagov-management-staging-ssb-smtp -s management-staging-egress -o gsa-datagov --protocol tcp --port 8080

• [ ] Create network policy for ssb-smtp (development-ssb)

  cf add-network-policy ssb-smtp proxy-gsa-datagov-development-ssb-ssb-smtp -s development-ssb-egress -o gsa-datagov --protocol tcp --port 8080

• [ ] Create network policy for ssb-solrcloud (management)

  cf add-network-policy ssb-solrcloud proxy-gsa-datagov-management-ssb-solrcloud -s management-egress -o gsa-datagov --protocol tcp --port 8080

• [ ] Create network policy for ssb-solrcloud (management-staging)

  cf add-network-policy ssb-solrcloud proxy-gsa-datagov-management-staging-ssb-solrcloud -s management-staging-egress -o gsa-datagov --protocol tcp --port 8080

• [ ] Create network policy for ssb-solrcloud (development-ssb)

  cf add-network-policy ssb-solrcloud proxy-gsa-datagov-development-ssb-ssb-solrcloud -s development-ssb-egress -o gsa-datagov --protocol tcp --port 8080

• [ ] Remove public egress security group (development-ssb)

  cf unbind-security-group public_networks_egress gsa-datagov development-ssb --lifecycle running

• [ ] Remove public egress security group (management-staging)

  cf unbind-security-group public_networks_egress gsa-datagov management-staging --lifecycle running

• [ ] Remove public egress security group (management)

  cf unbind-security-group public_networks_egress gsa-datagov management --lifecycle running

[nickumia-reisys]

As a foreboding warning, we may have memory usages following this deployment. We have 39G of constant app allocation, with rolling restarts and recurring tasks, the usage gets really close to our 50G limit. This work will add ~1GB to our constant app allocation. I fear it being just enough to be causing more random failures.

[nickumia-reisys]

Documenting here, but I'm sure it will get lost again. For the current egress deployments to work, the ci-deployer key that is created in the non-egress space needs to be given additional permission to the egress space. @btylerburton was able to assign the correct permissions and we did it through the cloud.gov dashboard UI.

[robert-bryson]

After the huddle yesterday, I poked around a bit and found that (for Terraform Enterprise at least) there is a list of domains that need egress. I added those to the allow.acl but it still had the same issue creating the service.

I have still been able to ping domains not on the allow list from the ssb-solrcloud app so I am suspicious that egress is working correctly.

[nickumia-reisys]

As a summary of where this stands...

• There are concerns about whether egress is actually doing what egress was fundamentally designed to do. (@robert-bryson can offer additional context while I'm away, but see Appendix A below)
• There are issues with determining the finite set of domains that each broker (eks, smtp, solr) needs to operate (see Appendix B below)

General Workflow/Design

The ssb egress deployment is supposed to be an identical copy of the data.gov egress deployment. While the deployment process was not formally documented anywhere (including the historical issue mentioned above), the following statements highlight the key points:

• There is an app in the main space.
• There is an egress-proxy in the main-egress space.
• The main space has the cloud.gov trusted_local_networks_egress security group applied to it.
  • This allows only internal cloud.gov traffic.
• The main-egress space has the cloud.gov public_networks_egress security group applied to it.
  • This allows both internal cloud.gov traffic and external public internet traffic.
• There is a network-policy between the app and the egress-proxy.
  • All outbound traffic from the app must go through the egress-proxy
  • There is an environment variable proxy_url that informs the app of the url/username/password/port of the egress-proxy. On startup, the app converts the proxy_url to https_proxy to start enforcing it. The reason is because if https_proxy is set then it will cause problems during staging of the app. Below are a few relevant PRs/Issues:
  • https://github.com/GSA/data.gov/issues/3755#issuecomment-1310598529
  • https://github.com/GSA/cg-egress-proxy/pull/10
  • https://github.com/GSA/data.gov/pull/4064
  • https://github.com/GSA/data.gov/issues/4055
• The source code of the egress-proxy is: https://github.com/GSA/cg-egress-proxy/tree/main/proxy
• Management of the egress-proxy is scripted, but not automated.
  • The deployment is handled by
  • An enable workflow which creates the egress-proxy, the network-policy and sets the proxy_url in the app
    • As mentioned above, the user that is used in this workflow must have privileges to both the main and main-egress spaces.
  • A disable workflow which unsets the proxy_url variable in the app, deletes the network-policy and then restarts the app for the changes to take effect
  • A restart workflow ... I'm not entirely sure of the use-case of this workflow
  • Typically, if there are changes to the egress-proxy source code or (allow/deny) acls, the disable workflow MUST be run and then the enable workflow MUST be run. (This is a terrible design, but it's the current situation)
  • For incremental workarounds, the following process can be used:
  • SSH into the egress-proxy
  • Update the (allow/deny) acl with vim.
  • Run ./caddy stop
  • Run ./caddy start
  • ⭐ Make sure the app does not get restarted, re-deployed or otherwise changed (If it does, all of your changes will be lost)⭐
  • SSH into the app test the url or process that needs to talk to the domain.
• The acls are tracked in this directory: https://github.com/GSA/datagov-ssb/tree/main/egress/acl
  • There is one allow file and one deny file. These two files apply to all of the apps (to decrease complexity).

SSB Specific Details

(Each broker app lives in cloud.gov) to create (services that live outside of cloud.gov) to be made available to (other apps in cloud.gov). For the following explanation, I will use the ssb-solrcloud broker app in development-ssb space as an example, but the same details apply to the rest of the brokers and all of the other spaces.

• There is a ssb-solrcloud app that exists in the development-ssb space.
• There is a proxy-gsa-datagov-development-ssb-ssb-solrcloud app that exists in the development-ssb-egress space.
• There is a test-solr service that exists (or will be created) in the development-ssb space.
• test-solr is created by running cf create-service solr-on-ecs base "test-solr" -c service-config.json -b "ssb-solrcloud-gsa-datagov-development-ssb" --wait in the development-ssb space.
  • service-config.json is copied from either catalog or inventory.
  • When this command is run, a request is sent to the ssb-solrcloud app.
  • The ssb-solrcloud app receives the request via the cloud-service-broker binary.
  • The cloud-service-broker binary then does some housekeeping and creates a terraform workspace in the attached DB.
  • Within the terraform workspace, a terraform process is kicked off that does everything that is predefined in the datagov-brokerpak-solr code.
  • This new terraform process then sends out requests to wherever it needs to (terraform.io, aws, et cetera) via the proxy-gsa-datagov-development-ssb-ssb-solrcloud
  • When the terraform process is complete, there will be a Solr that exists on AWS that is represented by the test-solr service on cloud.gov and accessible from other apps within its cloud.gov space.
• ⭐⭐ The proxy-gsa-datagov-development-ssb-ssb-solrcloud proxy is for the ssb-solrcloud app and NOT the resources that represent Solr. ⭐ ⭐
  • The proxy-gsa-datagov-development-ssb-ssb-solrcloud needs to allow the ssb-solrcloud app to talk to whoever to create the Solr, but it does not manage the Solr itself.

Appendix A

The egress-proxy uses caddy as a reverse proxy to control egress traffic. There are few modules and plugins that help caddy do its job. The five main components that are relevant are:

• The Caddyfile .... not really relevant, just need to know it exists and is the configuration file.
• The start.sh ... also not really relevant, does some certificate magic and stuff haha..
• The .profile does some acl stuff on startup.
• The allow.acl are all of the domains that are allowed through.
• The deny.acl are all of the domains explicitly denied.
  • Anything not mentioned on the allow or deny are implicitly denied, but there needs to be at least something in the deny.acl (learned from @robert-bryson)

If you attempt to access something on the allowed list, you should get a 200. If you attempt to access something on the denied list, you should get a 403. When we say that the egress proxy is not doing what it was designed to do, essentially all domains are returning a 200.

Appendix B

Through trial and error, the farthest we've gotten is that the ssb-solrcloud app needs to talk to registry.terraform.io to download the vpc module. This most likely occurs during a terraform init. We've added all of the terraform specific routes, but have not seen the process get any further. This could be a compound issue with the problem highlighted by Appendix A, but it may or may not be related. We can access everything from the ssb-solrcloud app (container). But as I'm writing this, I'm wondering if there is a problem with the cloud-service-broker binary abstraction where it needs to be configured to talk to the egress proxy properly. The terraform process runs in the ssb-solrcloud app itself, it's not like a second layer of virtualization (at least I don't think so), so it shouldn't have any additional layers of abstraction to jump through (although it is a possibility). The reason I think this is unlikely is because sometimes the service retrieves some bytes, the connection just gets closed early... this is technically a lead, but not much to work with.

Given that we have not made it past the terraform init, we still don't know what's necessary for the terraform apply and as equally as important the terraform destroy. For eks, there may be additional dependencies since it uses helm and kubectl which downloads charts and other resources. @robert-bryson comment above highlights part of a solution, but it is not the entire solution.

• https://github.com/GSA/data.gov/issues/4118#issuecomment-1379124748

All in all, this is a very complex technology stack with a lot of peculiarities, I will not call it elegant, but it is a wonder how it all works. I wish I could offer more help, but this is all that I have about the problem currently.

[nickumia-reisys]

List of possible issues (none of which I have been able to verify or deny)

• Certificate issues
• Domain specific issues
• Cloud Service Broker binary issues
• Terraform issues
• HTTP vs HTTPS issues

[nickumia-reisys]

Investigated TF Possible Environment Variable workarounds.. Nothing seems to inhibit or support the use of an egress proxy. I have a hunch that terraform is not using the egress proxy properly. It seems like the egress proxy is working properly now in that domains in the allowed list are allowed and everything else is denied.

[nickumia-reisys]

I fear we may have to implement something like this 😢 😭 ... https://github.com/jasonwbarnett/terraform-registry-proxy

[nickumia-reisys]

Although looking through the code, it doesn't seem to have anything special in it... For the sake of trying the only known option found above, the following steps will be taken as a proof-of-concept:

• [x] Create a CF deployment manifest for the terraform-registry-proxy.
  • [x] Use go buildpak as the base.
  • [x] Figure out what else is necessary to get the proxy functioning.
• [x] Deploy app to development-ssb-egress space.
• [x] Ensure proxy-gsa-datagov-development-ssb-ssb-solrcloud can talk to terraform-registry-proxy.
• [x] Modify the solr brokerpak to point to the terraform-registry-proxy for the terraform required providers.
• [x] Deploy the modified solr brokerpak to ssb-solrcloud in development-ssb
• [ ] Test to see if requests can get through!

[nickumia-reisys]

Ssoooooo.... super complicated proof-of-concept (shoutout to @btylerburton for helping me focus through a critical part!)

• https://github.com/jasonwbarnett/terraform-registry-proxy

  • I modified the source code to add a /health route to pass cloud.gov healthchecks

    
    (in func main)
    http.HandleFunc("/health", health)

  (somewhere at root) func health(w http.ResponseWriter, req *http.Request) { fmt.Fprintf(w, "hello\n") }

  - I vendored go dependencies by using either `godep` or `dep` or some form of `mod` vendoring?
  ```bash
  # https://github.com/golang/dep#installation
  sudo apt-get install go-dep
  dep init
  dep ensure
  
  # https://github.com/tools/godep#install
  go get github.com/tools/godep
  
  # mod vendoring?
  go get ./...
  go mod vendor

  • I created a cf deployment using the following manifest.yml (cf push terraform-registry)

    ---
    applications:
    - name: terraform-registry
    buildpacks:
      - https://github.com/cloudfoundry/go-buildpack
    routes:
      - route: terraform-registry.apps.internal
      - route: hashicorp-releases.apps.internal
    health-check-type: http
    health-check-http-endpoint: /health
    health-check-invocation-timeout: 60
    instances: 1
    disk_quota: 2G
    memory: 250M
    # (note: the :8080 is super important to get it to work)
    command: server -registry-proxy-host terraform-registry.apps.internal:8080 -release-proxy-host hashicorp-releases.apps.internal:8080
    env:
      GO_INSTALL_PACKAGE_SPEC: terraform-registry/server

  • Added network policies for lots of apps (not sure if all of them are necessary)

    cf add-network-policy proxy-gsa-datagov-development-ssb-ssb-solrcloud terraform-registry -s development-ssb-egress -o gsa-datagov --protocol tcp --port 8080
    cf add-network-policy ssb-solrcloud terraform-registry -s development-ssb-egress -o gsa-datagov --protocol tcp --port 8080

• proxy-gsa-datagov-development-ssb-ssb-solrcloud
  • I added all of the following routes to the allow.acl

    api.replicated.com
    get.replicated.com
    registry-data.replicated.com
    registry.replicated.com
    *.quay.io
    cdn.quay.io
    quay-registry.s3.amazonaws.com
    *.cloudfront.net
    hub.docker.com
    index.docker.io
    auth.docker.io
    registry-1.docker.io
    download.docker.com
    production.cloudflare.docker.com
    install.terraform.io
    releases.hashicorp.com
    yy0ffni7mf-dsn.algolia.net
    *.apps.internal

  • I tested that it can talk to the (real) terraform registry through the (intermediary) terraform registry app and it succeeded.

    curl -I http://terraform-registry.apps.internal:8080/.well-known/terraform.json

• ssb-solrcloud
  • I ran the same curl test to make sure that this app could talk to terraform registry and it succeeded.
  • In the datagov-brokerpak-solr source code,
  • I changed terraform urls from "hashicorp/aws" to "terraform-registry.apps.internal/hashicorp/aws"
  • I changed terraform urls from "hashicorp/template" to "terraform-registry.apps.internal/hashicorp/template"
  • I changed terraform urls from "terraform-aws-modules/vpc/aws" to "terraform-registry.apps.internal/terraform-aws-modules/vpc/aws"
  • ~Added insecure = true to (aws) provider definition~

~The terraform-registry proxy app is setup only for http use, so I tried to get the terraform code to send http requests to it, not https requests through the insecure = true configuration. It did not work. If we get terraform to send http requests, I have pretty high confidence that this will get it to work.~ However, I figured I would document this much for now while I still try other ways of getting it to work. All in all, if this does eventually work, we'll have to come up with a deployment framework for this new app terraform-registry which will be a separate headache altogether...

~This seems so close...!~

A list of references:

• https://docs.cloudfoundry.org/buildpacks/go/index.html
• https://golang.github.io/dep/docs/introduction.html
• https://golang.github.io/dep/docs/new-project.html
• https://stackoverflow.com/questions/23164547/golang-reverseproxy-not-working
• https://gobyexample.com/http-servers
• https://github.com/golang/dep/issues/2057
• https://stackoverflow.com/a/67195422
• https://stackoverflow.com/a/50842202
• https://registry.terraform.io/providers/hashicorp/aws/latest/docs#insecure

[nickumia-reisys]

Well... learning about terraform internals.. It seems like the discovery service can only be used with https.

It is strongly recommended to provide the discovery document for a hostname on the standard HTTPS port 443. However, in development environments this is not always possible or convenient, so Terraform allows a hostname to end with a port specification consisting of a colon followed by one or more decimal digits.

When a custom port number is present, the service on that port is expected to implement HTTPS and respond to the same fixed discovery path.

For day-to-day use it is strongly recommended not to rely on this mechanism and to instead provide the discovery document on the standard port, since this allows use of the most user-friendly hostname form.

With this information, it seems like the terraform-registry must be reworked to run on https... I don't think this is worth the effort (but I'll still try for now I guess). We would not only be incorporating another third-party service, but we would need to fork it to get it to work for us.

I don't know where to start with the alternative path of trying to include the vpc module so that terraform doesn't need to download anything. From my preliminary analysis, it seem like modules cannot be packaged in the buildpak in the same way as terraform sources and binaries.

[nickumia-reisys]

As more context, the thing that we are trying get to terraform is displayed in the following screenshot: As opposed to the providers which are all packaged in the broker, the modules are not..

Also note: This is just for the solr brokerpak. This would need to be done for the eks brokerpak too which has more modules..

[nickumia-reisys]

We could manually install the modules dir and disable the terraform call to HashiCorp-provided network services using the .terraformrc. This seems a bit hack-y though.. And then I'm not sure if there will be other things terraform will try to talk to and get blocked.

[nickumia-reisys]

As a summary of possible next steps:

• [ ] Continue working with terraform-registry and make it operate on https
• [ ] Try to add the functionality from terraform-registry into egress-proxy (since egress-proxy is already handling ssl certs)
• [ ] Try to "vendor" terraform modules alongside terraform providers.
  • Note: From my research so far, there's no official way of doing this. We would need a custom solution.
• [ ] ~Try to do more research into what is happening to the terraform requests in the current egress-proxy and figure out if there's another solution to this problem... (I guess we could enable more logging in Caddy, but I'm still not convinced that we will determine the root of the problem)~
  • Caddy only supports INFO as the lowest log level and it is already set to that level.

[mogul]

From way above:

But as I'm writing this, I'm wondering if there is a problem with the cloud-service-broker binary abstraction where it needs to be configured to talk to the egress proxy properly.

From googling, it looks like Terraform understands and uses HTTPS_PROXY. I'm wondering if this could be resolved by ensuring the HTTPS_PROXY env var is propagated all the way into the CSB's invocation of Terraform. (It looks like no env vars are propagated currently; I asked about that in the CF Slack.)

[hkdctol]

Awaiting some information from upstream on vendoring modules

[nickumia-reisys]

Ongoing discussion with upstream CSB Team: https://cloudfoundry.slack.com/archives/C0164KKEZGX/p1675103503770689

[nickumia-reisys]

As an update on discoveries and current sticking points:

• The main problem with the CSB is that modules are not currently packaged with the binaries which causes it to try to connect to the world and download them (giving the Failed to request discovery document: Get https://x.y.z/.well-known/terraform.json error)
• The CSB doesn't have a method of including modules.
  • Possible solutions are to:
  • Re-write the brokerpak without module
    • Not sustainable because there are other brokerpaks with more complex modules that are harder to operate without.
  • Package the module into a provider somehow
    • This is a hack-y type of solution, it's just recreating the file structure (with a custom script) as if the module had been downloaded properly.
  • Create a CSB feature to support modules
    • This is the most community-friendly option, but definitely a big chunk of work.
  • Accept that the module cannot be packaged and work on the actual HTTPS_PROXY issue
    • This requires more debugging of the CSB and analyzing what specific requests are being blocked/not handled properly or otherwise lost in translation.

None of those solutions seem enticing to me, but I'll leave it up to the team to come up with more or decide from these.

[jbrown-xentity]

Summary Doc @hkdctol

[hkdctol]

ISSO advises we start work on AOR, at https://docs.google.com/document/d/11iNXSSk556VA1fhx6KaeYmw-qCktypjh/edit?usp=sharing&ouid=113511966954817069922&rtpof=true&sd=true

[hkdctol]

Draft AOR sent to ISSO

[hkdctol]

Met with ISSM on 8/9, this is pending CISO return from leave in a few weeks

[rpalmer-gsa]

@nickumia-reisys

Couple of questions:

1. Is this still going though the proxy with an Allow * from a domain standpoint, or not going though the proxy at all?

2. _"Accept that the module cannot be packaged and work on the actual HTTPSPROXY issue This requires more debugging of the CSB and analyzing what specific requests are being blocked/not handled properly or otherwise lost in translation."

If you send it though an Allow Any proxy and log the URLs would this be a means to populate the list of all URLS that are needed? Is this due to blocks, or is it due to usage of the proxy itself, its a little unclear from above.

[nickumia-reisys]

@rpalmer-gsa

Is this still going though the proxy with an Allow * from a domain standpoint, or not going though the proxy at all?

Currently, there are no egress applications set up. Is there a desire to have an Allow * proxy app?

If you send it though an Allow Any proxy and log the URLs would this be a means to populate the list of all URLS that are needed?

The issue is not "what are the domains/urls that need to get through".

Is this due to blocks, or is it due to usage of the proxy itself, its a little unclear from above.

The issue is that the CSB is not recognizing the proxy environment variables, so it doesn't know to send traffic through the proxy. Because all egress is disallowed except through the proxy, the requests do not leave the container at all.. I assume they get stuck at some network level "waiting to be sent" ...or... the container thinks it sent it and is waiting for a response that never comes.

[hkdctol]

Met with ISSM on 1/22 - he will follow up on AOR status

[gujral-rei]

Is this item up for review as part of the assessment?