Indicator of Compromise Scanner for CVE-2019-19781

This repository contains a utility for detecting compromises of Citrix ADC Appliances related to CVE-2019-19781. The utility, and its resources, encode indicators of compromise collected during FireEye Mandiant investigations. To learn more, please read the blog announcing this tool's release.

In summary the utility will:

• do a best effort job at identifying existing compromise.

It will not:

• identify a compromise 100% of the time, or
• tell you if a device is vulnerable to exploitation.

With community feedback, the tool may become more thorough in its detection. Please open an issue, submit a PR, or contact the authors if you have problems, ideas, or feedback.

Download the standalone tool from the Releases tab of this repository.

Features

This scanner can identify:

• web server log entries indicating successful exploitation
• file system paths of known malware
• post-exploitation activity in shell history
• known malicious terms in NetScaler directories
• unexpected modification of NetScaler directories
• unexpected crontab entries
• unexpected processes
• ports used by known malware

Details

The Indicator of Compromise (IoC) Scanner for CVE-2019-19781 was jointly developed by FireEye Mandiant and Citrix based on knowledge gleaned from incident response engagements related to exploitation of CVE-2019-19781. The goal of the scanner is to analyze available log sources and system forensic artifacts to identify evidence of successful exploitation of CVE-2019-19781. There are limitations in what the tool will be able to accomplish, and therefore, executing the tool should not be considered a guarantee that a system is free of compromise. For example, log files on the system with evidence of compromise may have truncated/rolled, the system may have been rebooted, an attacker may have tampered with the system to remove evidence of compromise, and/or installed a rootkit that masks evidence of compromise, etc.

The output of the this tool will fall into one of three categories:

1. Evidence of compromise. This is the default. Any evidence that falls into this category indicates that a device was successfully compromised. This could be anything from executing commands that disclosure information (e.g. view the ns.conf or smb.conf configuration files), to installing a backdoor (e.g. NOTROBIN, a coin miner, etc.), or dropping a Perl-based web shell.

2. Evidence of successful vulnerability scanning (this could be authorized system administrator or unauthorized attacker). Any evidence that falls into this category indicates the system was in a vulnerable state (e.g. the mitigation had not been applied) and that at least the first step to exploit CVE-2019-19781 was successful.

3. Evidence of failed vulnerability scanning. Any evidence that falls into this category indicates that attempts to scan or exploit the system failed.

This tool is not guaranteed to find all evidence of compromise, or all evidence of compromise related to CVE-2019-19781. If indications of compromise are identified on systems, organizations should perform a forensic examination of the compromised system to determine the scope and extent of the incident. This tool is offered AS IS and without warranty.

Usage

You should download the standalone Bash script from the Releases tab of this repository. Copying the source directory to a Citrix ADC Appliance is possible but not recommended.

The IoC Scanner can be run directly on a Citrix ADC Appliance. In this mode, the tool will scan files, processes, and ports for known indicators. The tool writes diagnostic messages to the STDERR stream and results to the STDOUT stream. In typical usage, you should redirect STDOUT to a file for review. The tool must be run as root in live mode on a Citrix ADC Appliance.

For example:

$ sudo bash ./ioc-scanner-CVE-2019-19781-v1.1.sh > "/tmp/results-$(date).txt"

The tool is designed to be used with the following products:

• Citrix ADC and Citrix Gateway version 13.0
• Citrix ADC and Citrix Gateway version 12.1
• Citrix ADC and Citrix Gateway version 12.0
• Citrix ADC and Citrix Gateway version 11.1
• Citrix ADC and Citrix Gateway version 10.5
• Citrix SD-WAN WANOP software and appliance models 4000, 4100, 5000, and 5100

The IoC Scanner can also inspect a mounted forensic image. In this scenario, pass a command line argument specifying the path to the image root directory. You don't have to be root to run in offline mode.

For example:

$ bash ./ioc-scanner-CVE-2019-19781-v1.1.sh /mnt/path/to/evidence/root/

In both modes, the tool will extract supporting code into a temporary directory; this directory will be deleted upon termination of the script. The tool does not make further changes to the system, although it may cause log entries to be generated.

Like all forensic analysis, prefer offline analysis against a dd image to live response. This will eliminate the likelihood that the tool causes relevant evidence to be overwritten.

Please review the Frequently Asked Questions for further details.

Contributing

As you invent further ways to identify compromise, please consider contributing to this IoC Scanner. We would like to provide the most thorough, correct scanner as possible.

The primary goal is to report high confidence indicators of compromise. Because users may rely on the output of this tool to initiate further investigation, it's important that we don't send them on a wild goose chase. Therefore, activity such as simple scanning should not be reported in the default mode. Any evidence of an actor gaining access to the system, fetching information, or creating content should always be reported.

Design

We provide this tool as a Bash script because it's a common denominator across Citrix ADC Appliances. Here's the feature matrix for Citrix ADC releases:

Although we've seen malware use Go to target FreeBSD/NetScaler, Go does not support FreeBSD 6.x.

Testing

We maintain sparse file system images containing evidence of compromise in the ./tests/ directory. As you add IoCs to this tool, such as known paths or blacklisted content, please provide examples of the evidence for testing.

You can run the unit tests on a Linux or macOS system like so:

$ bash ./tests/test.sh
runnning test:  access-logs
runnning test:    xml-template
runnning test:  crontab
runnning test:    var-cron-tabs-nobody
runnning test:  error-logs
runnning test:  file-system
runnning test:    netscalerd
runnning test:    notrobin-tmp-init
runnning test:    notrobin-var-nstmp-nscache
runnning test:  ns-content
runnning test:    chr-encoded-template
runnning test:    copied-ns-conf
runnning test:    curl-in-template
runnning test:    perms
runnning test:    var-tmp-netscaler-portal-templates
runnning test:    var-vpn-bookmark
runnning test:    webshell-in-scripts
runnning test:  shell-history
runnning test:    bash_log
runnning test:    notice_log

Building

Once you've checked out the source repository, you can build a standalone script using the ./build.sh tool. This packages the primary script and supporting resources into a single bundle. Upon execution, it will extract to a temporary directory, execute from there, and then clean up.

To build:

$ bash ./build.sh > ioc-scanner-CVE-2019-19781-rev$(git rev-parse HEAD | cut -c 1-8).sh

Further Reading

For additional information from FireEye regarding CVE-2019-19781 and in-the-wild exploitation, please see:

• Rough Patch: I Promise It'll Be 200 OK (Citrix ADC CVE-2019-19781) - published on January 14, 2020
• 404 Exploit Not Found: Vigilante Deploying Mitigation for Citrix NetScaler Vulnerability While Maintaining Backdoor - published on January 16, 2020

The response to CVE-2019-19781 has been a significant effort across the security industry and these blogs heavily cite additional contributions that will be of value to users of this tool. We recommend reading the linked material from these posts to best understand activity in your environment. As always, the FireEye Mandiant team is available to answer follow-up questions or to further assist on an investigation by contacting us here.