search

rfc-st / humble

A humble, and 𝗳𝗮𝘀𝘁, security-oriented HTTP headers analyzer.
https://github.com/rfc-st/humble
MIT License
261 stars 18 forks source link
analysis checklist cybersecurity header-parser headers http infosec kali-linux owasp python3 security security-audit security-scanner security-tools

readme

humble

A humble, and fast, security-oriented HTTP headers analyzer






A quick analysis with 'humble'!

"千里之行，始於足下 - 老子"
("A journey of a thousand miles begins with a single step. - Lao Tzu")

"And if you don't keep your feet, there's no knowing where you might be swept off to. - Bilbo Baggins"

### Table of contents [Features](#features)
[Screenshots](#screenshots)
[Installation & Update](#installation--update)
[Installation & Maintenance (Docker)](#installation--maintenance-docker)
[Usage](#usage)
[Advanced Usage](#advanced-usage)
[Checks: Missing Headers](#checks-missing-headers)
[Checks: Fingerprint Headers](#checks-fingerprint-headers)
[Checks: Deprecated Headers and Insecure Values](#checks-deprecated-headersprotocols-and-insecure-values)
[Checks: Empty Values](#checks-empty-values)
[Guidelines included](#guidelines-included-to-enable-security-http-headers)
[To-Do](#to-do)
[Further Reading](#further-reading)
[Contribute](#contribute)
[Acknowledgements](#acknowledgements)
[License](#license)

## Features :heavy_check_mark: 14 [checks](#checks-missing-headers) of missing HTTP response headers.
:heavy_check_mark: 1186 [checks](#checks-fingerprint-headers) of fingerprinting through HTTP response headers.
:heavy_check_mark: 113 [checks](#checks-deprecated-headersprotocols-and-insecure-values) of deprecated HTTP response headers/protocols or with insecure/wrong values.
:heavy_check_mark: SSL/TLS checks: requires the **amazing** https://testssl.sh/.
:heavy_check_mark: Browser support references for enabled HTTP security headers: provided by https://caniuse.com/.
:heavy_check_mark: Two types of analysis: brief and detailed, along with HTTP response headers.
:heavy_check_mark: Can exclude specific HTTP response headers from the analysis.
:heavy_check_mark: Can export each analysis to CSV, HTML5, JSON, PDF 1.4 and TXT (and in a filename and path of your choice).
:heavy_check_mark: Can analyze '_raw response files_': text files with HTTP response headers and values. Ex: curl option '--dump-header'.
:heavy_check_mark: Highlights experimental headers in each analysis.
:heavy_check_mark: Each detailed analysis may include up to dozens of official links, references and technical articles.
:heavy_check_mark: l10n: can display each analysis, the messages and almost all errors in English or Spanish.
:heavy_check_mark: Saves each analysis, showing at the end the improvements or deficiencies in relation to the last one.
:heavy_check_mark: Can display analysis statistics: either against a specific URL or all of them.
:heavy_check_mark: Can display fingerprint statistics: either against a specific term or the Top 20.
:heavy_check_mark: Code reviewed via Bandit, Flake8, pyinstrument, SonarLint, Sourcery and vermin.
:heavy_check_mark: Tested, one by one, on thousands of URLs.
:heavy_check_mark: Tested on Docker 26.1, Kali Linux 2021.1, macOS 14.2.1 and Windows 10 20H2.
:heavy_check_mark: Almost all the code under one of the most permissive licenses: MIT.
:heavy_check_mark: Regularly updated.
:heavy_check_mark: Minimal dependencies required.
:heavy_check_mark: Featured on OWASP, Kali Linux, Artemis, DefectDojo and HackTricks.
:heavy_check_mark: Developed entirely in my spare time, no strings attached: feel free to try it out and integrate it into your projects!.
:heavy_check_mark: And with the approval of several AI :smile:!.
## Screenshots .: (Windows) - Brief analysis.

(Windows) - Brief analysis


.: (Linux) - Brief analysis along with HTTP response headers.

(Linux) - Brief analysis along with HTTP response headers


.: (Linux) - Detailed analysis, in Spanish.

(Linux) - Detailed analysis in Spanish


.: (Linux) - Analysis of a "raw response file". Example.

(Linux) - Analysis of a raw response file


.: (Linux) - SSL/TLS checks.

```bash Options used: -f -g -p -U -s --hints ```

(Linux) - SSL/TLS checks (requires https://testssl.sh/ and Linux/Unix client)


.: (Linux) - List of HTTP fingerprint headers based on a specific term.

(Linux) - List of HTTP fingerprint headers based on a specific term


.: (Linux) - Brief analysis saved as CSV. Example.

(Linux) - Brief analysis saved as CSV


.: (Windows) - Detailed analysis saved as PDF. Example.

(Windows) - Detailed analysis saved as PDF


.: (Linux) - Detailed analysis saved as HTML. Example.

(Linux) - Detailed analysis saved as HTML


.: (Linux) - Brief analysis saved as JSON. Example.

(Linux) - Brief analysis saved as JSON


.: (Linux) - Analysis history file: Date, URL, Missing, Fingerprint, Deprecated/Insecure, Empty headers & Total warnings (the four previous totals).

(Linux) - Analysis history file: Date, URL, Missing, Fingerprint, Deprecated/Insecure, Empty headers & Total warnings (the four previous totals)


.: (Linux) - Statistics of the analysis performed against a specific URL.

(Linux) - Statistics of the analysis performed against a specific URL


.: (Linux) - Statistics of the analysis performed against all URLs, in Spanish.

(Linux) - Statistics of the analysis performed against all URLs in Spanish


## Installation & Update > [!NOTE] > Python 3.8 or higher is required. ```bash # Requirements: python3 and python3-pip (Windows) https://www.python.org/downloads/windows/ (Linux) if not installed by default, install them via, e.g. Synaptic, apt, dnf, yum ... (macOS) https://www.python.org/downloads/macos/ # Requirement: Git (Windows) https://git-scm.com/download/win (Linux) https://git-scm.com/download/linux (macOS) https://git-scm.com/download/mac # Setting up a virtual environment in Python (pending how to do it in Windows) Note: '/home/bluesman/humble_venv' is a example path for the virtual environment. $ python3 -m venv /home/bluesman/humble_venv $ source /home/bluesman/humble_venv/bin/activate $ cd /home/bluesman/humble_venv/ $ git clone https://github.com/rfc-st/humble.git $ cd humble $ pip3 install -r requirements.txt # Good practice: deactivate the virtual environment after you have finished using 'humble' $ deactivate # Activate the virtual environment to analyze URLs again with 'humble' $ cd /home/bluesman/humble_venv/ $ source /home/bluesman/humble_venv/bin/activate $ cd humble # Updating (weekly): activate the virtual environment and from 'humble' folder $ git pull # Updating (Release): activate the virtual environment, download the source code # .zip file of the most recent Asset and unzip it in the 'humble' folder, overwriting files. https://github.com/rfc-st/humble/releases ``` ## Installation & Maintenance (Docker) > [!NOTE] > Python 3.8 will be used to [build](https://github.com/rfc-st/humble/blob/master/Dockerfile) the image. ```bash # Install Docker, and make sure it's running. # E.g. (Linux): https://www.kali.org/docs/containers/installing-docker-on-kali/ # E.g. (macOs): https://docs.docker.com/desktop/install/mac-install/ # E.g. (Windows): https://docs.docker.com/desktop/install/windows-install/ # Build the image, providing the TAG as the latest Release of 'humble': '1.42' in this example. # https://github.com/rfc-st/humble/releases # (Windows may require elevated console privileges) $ docker build -t humble:1.42 . # Run the analysis specifying the above TAG, along with the specific options for 'humble'. # '-it', required: allocate a pseudo-TTY and keep the input interactive # '-rm', required: automatically remove the container and associated anonymous volumes when it exits # (Linux/macOS) $ docker run -it --rm --name humble humble:1.42 /bin/bash -c "python3 humble.py -u https://facebook.com -b" # (Windows) $ docker run -it --rm --name humble humble:1.42 python3 humble.py -u https://facebook.com -b # Removing (and untagging) previous images of 'humble' after upgrading to the latest release. $ docker rmi humble:1.42 ``` ## Usage ```console (Windows) $ py humble.py (Linux) $ python3 humble.py (macOS) $ python3 humble.py usage: humble.py [-h] [-a] [-b] [-df] [-e [TESTSSL_PATH]] [-f [FINGERPRINT_TERM]] [-g] [-grd] [-if INPUT_FILE] [-l {es}] [-lic] [-o {csv,html,json,pdf,txt}] [-of OUTPUT_FILE] [-op OUTPUT_PATH] [-r] [-s [SKIP_HEADERS ...]] [-u URL] [-ua USER_AGENT] [-v] 'humble' (HTTP Headers Analyzer) | https://github.com/rfc-st/humble | v.2024-11-01 options: -h, --help show this help message and exit -a Shows statistics of the performed analysis; if the '-u' parameter is ommited they will be global -b Shows overall findings; if omitted detailed ones will be shown -df Do not follow redirects; if omitted the last redirection will be the one analyzed -e [TESTSSL_PATH] Shows TLS/SSL checks; requires the PATH of https://testssl.sh/ -f [FINGERPRINT_TERM] Shows fingerprint statistics; if 'FINGERPRINT_TERM' (e.g., 'Google') is omitted the top 20 results will be shown -g Shows guidelines for enabling security HTTP response headers on popular servers/services -grd Shows the checks to grade an analysis, along with advice for improvement -if INPUT_FILE Analyzes 'INPUT_FILE': must contain HTTP response headers and values separated by ': '; E.g. 'server: nginx'. -l {es} Defines the language for displaying analysis, errors and messages; if omitted, will be shown in English -lic Shows the license for 'humble', along with permissions, limitations and conditions. -o {csv,html,json,pdf,txt} Exports analysis to 'humble_scheme_URL_port_yyyymmdd_hhmmss_language.ext' file; csv/json will have a brief analysis -of OUTPUT_FILE Exports analysis to 'OUTPUT_FILE'; if omitted the default filename of the parameter '-o' will be used -op OUTPUT_PATH Exports analysis to 'OUTPUT_PATH'; must be absolute. If omitted the PATH of 'humble.py' will be used -r Shows HTTP response headers and a detailed analysis; '-b' parameter will take priority -s [SKIP_HEADERS ...] Skips 'deprecated/insecure' and 'missing' checks for the indicated 'SKIP_HEADERS' (separated by spaces) -u URL Scheme, host and port to analyze. E.g. https://google.com -ua USER_AGENT User-Agent ID from 'additional/user_agents.txt' file to use. '0' will show all and '1' is the default -v, --version Checks for updates at https://github.com/rfc-st/humble examples: -u URL -a Shows statistics of the analysis performed against the URL -u URL -b Analyzes URL and reports overall findings -u URL -b -o csv Analyzes URL and exports overall findings to CSV format -u URL -l es Analyzes URL and reports (in Spanish) detailed findings -u URL -o pdf Analyzes URL and exports detailed findings to PDF format -u URL -o html -of test Analyzes URL and exports detailed findings to HTML format and 'test' filename -u URL -o pdf -op D:/Tests Analyzes URL and exports detailed findings to PDF format and 'D:/Tests' path -u URL -r Analyzes URL and reports detailed findings along with HTTP response headers -u URL -s ETag NEL Analyzes URL and skips 'deprecated/insecure' and 'missing' checks for 'ETag' and 'NEL' headers -u URL -ua 4 Analyzes URL using the fourth User-Agent of 'additional/user_agents.txt' file -a -l es Shows statistics (in Spanish) of the analysis performed against all URLs -f Google Shows HTTP fingerprint headers related to the term 'Google' ``` ## Advanced Usage .: (Linux) - Show only the analysis summary.
``` $ python3 humble.py -u https://www.spacex.com | grep -A 8 "\!." | sed $'1i \n' ``` Show only the analysis summary (Linux) .: (Windows) - Show only the analysis summary, in Spanish. PowerShell >= 7 required.
``` $ py humble.py -u https://www.spacex.com -l es | Select-String -Pattern '!.' -Context 1,8 -NoEmphasis ``` Show only the analysis summary (Windows, in Spanish. PowerShell >= 7 required) .: (Linux) - Show only the URL, date and analysis summary.
``` $ python3 humble.py -u https://www.spacex.com | grep -A7 -E "0. Info|\!." | grep -v "^\[1\." | sed 's/[--]//g' | sed -e '/./b' -e :n -e 'N;s/\n$//;tn' | sed $'1i \n' ``` Show URL, date and the analysis summary (Linux) .: (Linux) - Show only the deprecated headers/protocols and insecure values.
``` $ python3 humble.py -u https://www.spacex.com | sed -n '/\[3/,/^\[4/ { /^\[4/!p }' | sed '$d' | sed $'1i \n' ``` Show only the deprecated headers/protocols and insecure values (Linux) .: (Linux) - Check for HTTP client errors (4XX).
``` $ python3 humble.py -u https://my.prelude.software/demo/index.pl | grep -A1 -B5 'Note : \|Nota : ' --color=never ``` Check for HTTP client errors (4XX) (Linux) .: (Linux) - Analyze multiple URLs and save the results as PDFs.
``` $ datasets=('https://facebook.com' 'https://github.com' 'https://www.spacex.com'); for dataset in "${datasets[@]}"; do python3 humble.py -u "$dataset" -o pdf; done ``` Analyze multiple URLs and save the results as PDFs ## Checks: Missing Headers Check this file. ## Checks: Fingerprint headers Check this file. ## Checks: Deprecated headers/protocols and insecure values Check this file. > [!NOTE] > _humble_ tries to be **strict**: both in checking HTTP response headers and their values; some of these headers may be experimental and you may not agree with all the results after analysis. > > And that's **OK**! :smiley:; you should **never** blindly trust the results of security tools: there should be further work to decide whether the risk is non-existent, potential or real depending on the analyzed URL (its exposure, environment, etc). ## Checks: Empty values Any HTTP response header. ## Guidelines included to enable security HTTP headers * Amazon Web Services * Apache HTTP Server * Cloudflare * LiteSpeed Web Server * Microsoft Internet Information Services * Nginx * Node.js * WordPress ## To-Do - [ ] Add more Header/Value checks (only security-oriented) - [ ] A new detailed analysis of all CSP directives/values (W3C Level 2 & 3) - [ ] Google Style Python Docstrings and documentation via Sphinx ## Further reading https://caniuse.com/
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
https://github.com/search?q=http+headers+analyze
https://github.com/search?q=http+headers+secure
https://github.com/search?q=http+headers+security
https://owasp.org/www-project-secure-headers/
https://securityheaders.com/
https://scotthelme.co.uk/
https://webtechsurvey.com/common-response-headers
https://www.w3.org
## Contribute * Read this first!. * Report a Bug. * Create a Feature request. * Report a Security Vulnerability. * Send me your suggestions: rafael.fcucalon@gmail.com * Or use that email to tell me about integrations of this tool in others! * And to recommend me a good Blues! :sunglasses: Thanks for downloading _'humble'_, for trying it and for your time!. ## Acknowledgements * Bandit, colorama, Flake8, fpdf2, pyinstrument, requests, SonarLint, Sourcery, testssl.sh, tldextract and Vermin authors/teams: you rock :metal:!. * Aniket Navlur for this gem. * Azathothas for reporting this bug. * bulaktm for this suggestion. * confuciussayuhm for this suggestion. * cr4zyfish for some of these suggestions. * danterolle for this. * David for believing in the usefulness of this tool. * Eduardo for the first Demo and the example "(Linux) - Analyze multiple URLs and save the results as PDFs". * gl4nce for this suggestion. * İDRİS BUDAK for reporting the need to this check. * Julio for testing on macOS. * kazet for this suggestion. * manuel-sommer for this, this and this!. * MikeAnast for several suggestions. * n3bojs4, ehlewis and dkadev for this and this. * sophie for keeping 'humble' updated in Kali Linux and for this. * stanley101music for this, this and this!. ## License MIT © 2020-2024 Rafa 'Bluesman' Faura (rafael.fcucalon@gmail.com)
Original Creator - Rafa 'Bluesman' Faura (rafael.fcucalon@gmail.com)