recog_standardize: adjust logic

tsellers-r7 commented 3 years ago

Description

This PR addresses what I believe to a bug in bin/recog_standardize in which a string index is compared to -1 instead of nil. Since -1 shouldn't be a valid response to this check all entries were skipped.

I noticed this when working on PR #301 and not seeing some of the changes I made. I then added a random vendor name and recog_standardize didn't report it.

NOTE: In this PR I have not included the changes to the files in identifiers/ since I need some of the changes in PR #301 to land first. When the issue is addressed I'll open a PR to clean up.

You can see the results either of the following commands before and after the changes in this PR.

CC @hdm

# Across http_servers.xml
ruby bin/recog_standardize xml/http_servers.xml

# Across all of the files
for db in xml/*.xml; do ruby bin/recog_standardize $db; done

Output when running the updated script against `http_servers.xml`

$ ruby bin/recog_standardize xml/http_servers.xml 
VENDOR MISSING: CentOS WebPanel
SERVICE PRODUCT MISSING: Web Cache
SERVICE PRODUCT MISSING: Application Server Web Cache
SERVICE PRODUCT MISSING: Serv-U FTP Server
VENDOR MISSING: WFTPServer
SERVICE FAMILY MISSING: Wing FTP
SERVICE PRODUCT MISSING: Wing FTP Server
SERVICE PRODUCT MISSING: GlassFish Server
VENDOR MISSING: lighttpd
SERVICE PRODUCT MISSING: NetWeaver Application Server Java
SERVICE PRODUCT MISSING: NetWeaver Application Server
SERVICE PRODUCT MISSING: NetWeaver Internet Communication Manager
SERVICE PRODUCT MISSING: NetWeaver AS ABAP
SERVICE PRODUCT MISSING: Internet Graphics Server
SERVICE PRODUCT MISSING: SAP Message Server
SERVICE PRODUCT MISSING: SQL Anywhere
SERVICE PRODUCT MISSING: OpenVPN Access Server
SERVICE PRODUCT MISSING: Expressway
SERVICE PRODUCT MISSING: STUN Server
VENDOR MISSING: TigerVNC
SERVICE PRODUCT MISSING: TigerVNC
VENDOR MISSING: EmbedThis
VENDOR MISSING: Zed Shaw
VENDOR MISSING: Ruby-Lang
VENDOR MISSING: Aspen
VENDOR MISSING: Boa
VENDOR MISSING: Genivia
SERVICE FAMILY MISSING: Endpoint Protection Manager
VENDOR MISSING: SMA
SERVICE PRODUCT MISSING: Sunny WebBox
HW PRODUCT MISSING: Sunny WebBox
VENDOR MISSING: Kong
SERVICE FAMILY MISSING: Gateway
SERVICE PRODUCT MISSING: Gateway
VENDOR MISSING: MiniUPnP Project
SERVICE PRODUCT MISSING: alphapd
VENDOR MISSING: Xiph
SERVICE PRODUCT MISSING: Icecast
VENDOR MISSING: Couchbase
SERVICE PRODUCT MISSING: Sync Gateway
SERVICE PRODUCT MISSING: Couchbase Server
SERVICE PRODUCT MISSING: Kestrel web server
VENDOR MISSING: Tencent
SERVICE PRODUCT MISSING: Secure Tencent Gateway
VENDOR MISSING: axTLS Project
SERVICE PRODUCT MISSING: axTLS
VENDOR MISSING: Tinyproxy Project
SERVICE PRODUCT MISSING: Tinyproxy
VENDOR MISSING: Comcast
HW PRODUCT MISSING: Xfinity Broadband Router
HW PRODUCT MISSING: Univerge
VENDOR MISSING: CaddyServer
SERVICE PRODUCT MISSING: Caddy
SERVICE PRODUCT MISSING: httpd
VENDOR MISSING: Facebook
SERVICE PRODUCT MISSING: Proxygen
VENDOR MISSING: GFI
SERVICE PRODUCT MISSING: Kerio Connect
SERVICE PRODUCT MISSING: Kerio Control
VENDOR MISSING: Cesanta
SERVICE PRODUCT MISSING: Mongoose
VENDOR MISSING: Bangteng
SERVICE PRODUCT MISSING: Kangle
VENDOR MISSING: PalletsProjects
SERVICE PRODUCT MISSING: Werkzeug

Motivation and Context

Bug fix

How Has This Been Tested?

Local testing using the commands above after adding a product I to not exist in identifiers/service_product.txt.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have updated the documentation accordingly (or changes are not required).
[x] I have added tests to cover my changes (or new tests are not required).
[x] All new and existing tests passed.

tsellers-r7 commented 3 years ago

@hdm - CCing you because you're the most likely to use this code on a regular basis.

hdm commented 3 years ago

Thanks! Looks good to me!

tsellers-r7 commented 3 years ago

Note: After PR #301 lands I will create a clean up PR to address any issues identified by running this tool. I know of at least a couple at this point.

rapid7 / recog