Ngrok - VA CSOC Security Alert

[gracekretschmer-metrostar]

User Story or Problem Statement

As a member of VA CSOC, I need to understand how CMS is using ngrok and how it plans to remove it from usage.

Description or Additional Context

Relevant Links

• Jira ticket: https://va-gov.atlassian.net/browse/VAHELP-8139

Steps for Implementation

Acceptance Criteria

• [ ] The scope of ngrok usage within CMS is understood and the appropriate team (either CMS or sitewide) is communicating with VA CSOC about the remediation plan.

[7hunderbird]

The three TL;DR to know

1. We are now using Datadog on the training site's EC2 host to watch for any ngrok traffic
2. The CMS application (which runs the training site) has no runtime dependencies that use ngrok
3. Because any developer can install and run ngrok we need to communicate that it's not a tool to use

Infrastructure

• The report came in that the https://training.cms.va.gov website (aka https://training-qtpbfivdfzrlrlyiqwmanznaobpek1ca.demo.cms.va.gov/) was linked to an ngrok instance
• Reviewed what was being recorded to datadog
  • No network information was being recorded
  • To turn on network information, process monitoring also needed to be turned on
• With network monitoring enabled, a Datadog DNS query is added

  (network.dns_query:*.ngrok-free.app OR network.dns_query:*.ngrok-free.dev OR network.dns_query:*.ngrok.io OR network.dns_query:*.ngrok.com)

• Then ran a call and caught my request from Tugboat EC2 instance out to api.ngrok.com

  curl https://api.ngrok.com -H "authorization: Bearer {MY_API_TOKEN}" -H "ngrok-version: 2"

• We are now recording any DNS lookups of ngrok related domains

Application

• the cms app 2 years ago used lando for a local development environment
  • lando used ngrok
  • but we've not used lando and have transitioned to ddev 2 years ago
• the cms app uses the content-build and vets-website repos
• both content-build and vets-website use cypress
• cypress uses blob-util
  • blob-util has a development dependency for ngrok
  • blob-util doesn't use ngrok when compiled

Ngrok CLI Usage

• Any developer can install a binary CLI on their local laptop
  • This has been used as a work around when Tugboat is not working
  • https://github.com/department-of-veterans-affairs/va.gov-cms/issues/17306
• Anyone can start a tunnel with a command like

  ngrok http -–domain=panda-new-kit.ngrok-free.app 80

• the *.ngrok-free.app domain is for free-tier users
  • https://ngrok.com/blog-post/new-ngrok-domains
  • https://www.infoq.com/news/2023/08/ngrok-free-static-domain/