Closed Spidle closed 2 months ago
both of these requests seem pretty reasonable, and easy to implement (pretty sure, haven't looked yet). pinning for now
howdy! started poking at this. when you say json logs, i assume you mean those produced by -o ... --json
and do NOT mean those inside --debug-log ...
. You may be interested in having the configuration
entry as well for analysis.
I'm looking into what it'll take code-wise to make the requested changes. lmk if the config entry is useful
{"type":"configuration","wordlist":"common.txt","config":"ferox-config.toml","proxy":"","replay_proxy":"","server_certs":[],"client_cert":"","client_key":"","target_url":"http://localhost:8000","status_codes":[100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,421,422,423,424,426,428,429,431,451,500,501,502,503,504,505,506,507,508,510,511,103,425],"replay_codes":[100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,421,422,423,424,426,428,429,431,451,500,501,502,503,504,505,506,507,508,510,511,103,425],"filter_status":[],"threads":50,"timeout":7,"verbosity":3,"silent":false,"quiet":false,"auto_bail":false,"auto_tune":false,"json":true,"output":"test-log2.log","debug_log":"test-debug.log","user_agent":"feroxbuster/2.10.2","random_agent":false,"redirects":false,"insecure":false,"extensions":[],"methods":["GET"],"data":[],"headers":{},"queries":[],"no_recursion":false,"extract_links":true,"add_slash":false,"stdin":false,"depth":4,"scan_limit":0,"parallel":0,"rate_limit":0,"filter_size":[],"filter_line_count":[],"filter_word_count":[],"filter_regex":[],"dont_filter":false,"resumed":false,"resume_from":"","save_state":false,"time_limit":"","filter_similar":[],"url_denylist":[],"regex_denylist":[],"collect_extensions":false,"dont_collect":["tif","tiff","ico","cur","bmp","webp","svg","png","jpg","jpeg","jfif","gif","avif","apng","pjpeg","pjp","mov","wav","mpg","mpeg","mp3","mp4","m4a","m4p","m4v","ogg","webm","ogv","oga","flac","aac","3gp","css","zip","xls","xml","gz","tgz"],"collect_backups":false,"backup_extensions":["~",".bak",".bak2",".old",".1"],"collect_words":false,"force_recursion":false}
PS: Can the json statistics be added on exit if ctrl-c is pressed?
this actually happens already, if you have save-state
enabled, the statistics
field should be the last entry in the .state
file (the filename itself denotes the original target url -> ferox-http_localhost:8000-1711710683.state
as well as the config
entry within the state file).
However, with large distributed scans (which is what it sounds like you're doing), I THINK you should get a state file for any single target that was killed, but I haven't checked that for sure.
here's what things are looking like at the moment
{"type":"response","url":"http://localhost:8000/stuff/nthings","original_url":"http:localhost:8000","path":"/stuff/nthings","wildcard":false,"status":401,"method":"GET","content_length":120,"line_count":1,"word_count":5,"headers":{"content-type":"text/plain; charset=utf-8","content-length":"120","x-content-type-options":"nosniff","server":"Rocket","x-frame-options":"SAMEORIGIN","date":"Sat, 30 Mar 2024 19:32:05 GMT","permissions-policy":"interest-cohort=()"},"extension":"","timestamp":1711827125.7913597}
single scan
{"type":"statistics","timeouts":0,"requests":28389,"expected_per_scan":4724,"total_expected":28344,"errors"
:0,"successes":5811,"redirects":5257,"client_errors":11962,"server_errors":4593,"total_scans":6,"initial_ta
rgets":0,"links_extracted":0,"extensions_collected":0,"status_200s":737,"status_301s":785,"status_302s":784
,"status_401s":775,"status_403s":4584,"status_429s":4426,"status_500s":2305,"status_503s":769,"status_504s"
:0,"status_508s":0,"wildcards_filtered":344,"responses_filtered":344,"resources_discovered":15990,"url_form
at_errors":0,"redirection_errors":0,"connection_errors":0,"request_errors":0,"directory_scan_times":[10.584
397651,10.981936694,12.019307243,11.967351518,11.079414173,9.05110747],"total_runtime":[15.644809645],"targ
ets":["http:localhost:8000"]}
two targets, not parallel
{"type":"statistics","timeouts":0,"requests":36,"expected_per_scan":2,"total_expected":17,"errors":0,"successes":11,"redirects":4,"client_errors":19,"server_errors":2,"total_scans":2,"initial_targets":0,"links_extracted":13,"extensions_collected":0,"status_200s":8,"status_301s":0,"status_302s":1,"status_401s":0,"status_403s":1,"status_429s":2,"status_500s":2,"status_503s":0,"status_504s":0,"status_508s":0,"wildcards_filtered":9,"responses_filtered":9,"resources_discovered":8,"url_format_errors":0,"redirection_errors":0,"connection_errors":0,"request_errors":0,"directory_scan_times":[0.028152285,2.2592075019999998],"total_runtime":[3.424981309],"targets":["http:localhost:8000","https://bitdiscovery.com"]}
two targets, parallel, creates a new directory, with a log file per target. each log file has the same array with a single entry as the single target shown above.
❯ lt stuff2_json-1711827125.logs
total 88
drwxrwxr-x 17 epi epi 65536 Mar 30 15:32 ..
drwxrwxr-x 2 epi epi 4096 Mar 30 15:32 .
-rw-rw-r-- 1 epi epi 1659 Mar 30 15:32 ferox-http:localhost:8000-1711827125.log
-rw-rw-r-- 1 epi epi 10894 Mar 30 15:32 ferox-https_bitdiscovery_com-1711827125.log
added configuration
entry to -o outputfile
(with and without --json
)
{"type":"configuration","wordlist":"common.txt","config":"ferox-config.toml","proxy":"","replay_proxy":"","server_certs":[],"client_cert":"","client_key":"","target_url":"http:localhost:8000","status_codes":[100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,421,422,423,424,426,428,429,431,451,500,501,502,503,504,505,506,507,508,510,511,103,425],"replay_codes":[100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,421,422,423,424,426,428,429,431,451,500,501,502,503,504,505,506,507,508,510,511,103,425],"filter_status":[],"threads":50,"timeout":7,"verbosity":0,"silent":false,"quiet":false,"auto_bail":false,"auto_tune":false,"json":true,"output":"stuff.json","debug_log":"","user_agent":"feroxbuster/2.10.2","random_agent":false,"redirects":false,"insecure":false,"extensions":[],"methods":["GET"],"data":[],"headers":{},"queries":[],"no_recursion":false,"extract_links":true,"add_slash":false,"stdin":false,"depth":4,"scan_limit":0,"parallel":0,"rate_limit":0,"filter_size":[],"filter_line_count":[],"filter_word_count":[],"filter_regex":[],"dont_filter":false,"resumed":false,"resume_from":"","save_state":false,"time_limit":"","filter_similar":[],"url_denylist":[],"regex_denylist":[],"collect_extensions":false,"dont_collect":["tif","tiff","ico","cur","bmp","webp","svg","png","jpg","jpeg","jfif","gif","avif","apng","pjpeg","pjp","mov","wav","mpg","mpeg","mp3","mp4","m4a","m4p","m4v","ogg","webm","ogv","oga","flac","aac","3gp","css","zip","xls","xml","gz","tgz"],"collect_backups":false,"backup_extensions":["~",".bak",".bak2",".old",".1"],"collect_words":false,"force_recursion":false}
Epi, This is amazing, sorry I was out this weekend and didn't get back to you to provide my gratitude and 2-cents.
So much gratitude.
@all-contributors add @Spidle for ideas
@epi052
I've put up a pull request to add @Spidle! :tada:
Is your feature request related to a problem? Please describe. When looking at my feroxbuster json logs, I'm unable to relate the statistics to a particular host, and additionally, I dont know when the feroxbuster was run. I'm trying to use statistical analysis on a massive amount of ferox JSON files.
Describe the solution you'd like I would like each JSON entry to have a timestamp. Additionally, statistics entries should have the base URL that was passed in when then scan was initiated included in the data.
PS: Can the json statistics be added on exit if ctrl-c is pressed?
Describe alternatives you've considered Post processing. It works (sorta).