Zapotek
commented
9 years ago When configuring Arachni's proxy settings where the env variables unset? Because that should have worked.
Closed bricex closed 9 years ago
Zapotek
commented
9 years ago When configuring Arachni's proxy settings where the env variables unset? Because that should have worked.
bricex
commented
9 years ago Yes, I have been trying a variety of combinations including the various proxy types, and thinking the "?" character in the password may need to be URL encoded. So far I get "couldn't get a response after 5 tries" when using Arachni's proxy settings.
2015-01-28 12:26:35 -0600 --------------------------------------------------------------------------------
ENV:
---
REMOTEHOST: brice-dev.bricex.com
XDG_SESSION_ID: c3
rvm_bin_path: "/home/labs/.rvm/bin"
GEM_HOME: "/home/labs/arachni-1.0.6-0.5.6/system/gems"
TERM: xterm
SHELL: "/bin/bash"
IRBRC: "/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/lib/ruby/.irbrc"
HUSHLOGIN: 'FALSE'
MY_RUBY_HOME: "/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/lib/ruby"
USER: labs
http_proxy: ''
LD_LIBRARY_PATH: "/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/lib"
LS_COLORS: 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'
_system_type: Linux
rvm_path: "/home/labs/.rvm"
rvm_prefix: "/home/labs"
PATH: "/home/labs/arachni-1.0.6-0.5.6/system/gems/bin:/home/labs/arachni-1.0.6-0.5.6/bin/../system/../bin:/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/bin:/home/labs/arachni-1.0.6-0.5.6/bin/../system/gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/labs/.rvm/bin"
MAIL: "/var/mail/labs"
QT_QPA_PLATFORMTHEME: appmenu-qt5
PWD: "/home/labs/arachni-1.0.6-0.5.6/bin"
ARACHNI_WEBUI_LOGDIR: "/home/labs/arachni-1.0.6-0.5.6/bin/../system/logs/webui"
LANG: en_US.UTF-8
ARACHNI_FRAMEWORK_LOGDIR: "/home/labs/arachni-1.0.6-0.5.6/bin/../system/logs/framework"
_system_arch: x86_64
_system_version: '14.04'
https_proxy: ''
rvm_version: 1.26.9 (master)
HOME: "/home/labs"
SHLVL: '1'
RAILS_ENV: production
DYLD_LIBRARY_PATH: "/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/lib:"
LOGNAME: labs
GEM_PATH: "/home/labs/arachni-1.0.6-0.5.6/bin/../system/gems"
LESSOPEN: "| /usr/bin/lesspipe %s"
XDG_RUNTIME_DIR: "/run/user/1000"
RUBYLIB: "/home/labs/arachni-1.0.6-0.5.6/system/gems/gems/bundler-1.7.8/lib:/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/lib/ruby:/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/lib/ruby/site_ruby/2.1.0:/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/lib/ruby/2.1.0:/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/lib/ruby/2.1.0/x86_64-linux:/home/labs/arachni-1.0.6-0.5.6/bin/../system/usr/lib/ruby/site_ruby/2.1.0/x86_64-linux"
RUBY_VERSION: ruby-2.1.5
LESSCLOSE: "/usr/bin/lesspipe %s %s"
_system_name: Ubuntu
RACK_ENV: development
BUNDLE_GEMFILE: "/home/labs/arachni-1.0.6-0.5.6/system/arachni-ui-web/Gemfile"
_ORIGINAL_GEM_PATH: "/home/labs/arachni-1.0.6-0.5.6/bin/../system/gems"
BUNDLE_BIN_PATH: "/home/labs/arachni-1.0.6-0.5.6/system/gems/gems/bundler-1.7.8/bin/bundle"
RUBYOPT: "-rbundler/setup"
MANPATH: "/home/labs/arachni-1.0.6-0.5.6/system/gems/gems/kramdown-1.4.1/man"
BUNDLE_ORIG_MANPATH: "/home/labs/arachni-1.0.6-0.5.6/system/gems/gems/kramdown-1.4.1/man"
--------------------------------------------------------------------------------
OPTIONS:
---
datastore:
token: b47346b371f5fea3f5fea5cf8dde4a8a
input:
values:
"(?i-mx:name)": arachni_name
"(?i-mx:user)": arachni_user
"(?i-mx:usr)": arachni_user
"(?i-mx:pass)": 5543!%arachni_secret
"(?i-mx:txt)": arachni_text
"(?i-mx:num)": '132'
"(?i-mx:amount)": '100'
"(?i-mx:mail)": arachni@email.gr
"(?i-mx:account)": '12'
"(?i-mx:id)": '1'
default_values:
"(?i-mx:name)": arachni_name
"(?i-mx:user)": arachni_user
"(?i-mx:usr)": arachni_user
"(?i-mx:pass)": 5543!%arachni_secret
"(?i-mx:txt)": arachni_text
"(?i-mx:num)": '132'
"(?i-mx:amount)": '100'
"(?i-mx:mail)": arachni@email.gr
"(?i-mx:account)": '12'
"(?i-mx:id)": '1'
without_defaults: true
force: false
browser_cluster:
pool_size: 6
job_timeout: 120
worker_time_to_live: 100
ignore_images: false
screen_width: 1600
screen_height: 1200
audit:
exclude_vector_patterns: []
include_vector_patterns: []
link_templates: []
links: true
forms: true
cookies: true
headers: false
with_both_http_methods: false
cookies_extensively: false
http:
user_agent: Arachni/v1.0.6
request_timeout: 50000
request_redirect_limit: 5
request_concurrency: 10
request_queue_size: 100
request_headers: {}
cookies: {}
proxy_host: 10.10.5.18
proxy_port: 8080
proxy_username: A320060
proxy_password: "-4t?67*swechawR!zeme"
scope:
redundant_path_patterns: {}
dom_depth_limit: 10
exclude_path_patterns: []
exclude_content_patterns: []
include_path_patterns: []
restrict_paths: []
extend_paths: []
url_rewrites: {}
include_subdomains: false
https_only: false
session: {}
checks:
- xss
- xss_dom
- xss_dom_inputs
- xss_dom_script_context
- xss_event
- xss_path
- xss_script_context
- xss_tag
platforms: []
plugins:
autothrottle:
discovery:
healthmap:
timing_attacks:
uniformity:
no_fingerprinting: false
authorized_by:
url: https://usa.cg.na.bricex.com/
--------------------------------------------------------------------------------
[2015-01-28 12:26:35 -0600] Giving up trying to audit: https://usa.cg.na.bricex.com/
[2015-01-28 12:26:35 -0600] Couldn't get a response after 5 tries.I can see the proxy env vars causing issues as libcurl might use them. And since Arachni uses libcurl to communicate with its browsers as well as with the remote server that might cause issues.
Unseating the env vars should have worked though so I'm not sure what's going on, could you please try setting the --http-proxy-type option?
bricex
commented
9 years ago I seem to get the same "couldn't get a response after 5 tries" when using --http-proxy-type. I tried all the possible options for it just to make sure.
2015-01-28 14:22:09 -0600 --------------------------------------------------------------------------------
ENV:
---
GEM_HOME: "/usr/bin/arachni/system/gems"
SHELL: "/bin/bash"
TERM: xterm
IRBRC: "/usr/bin/arachni/bin/../system/usr/lib/ruby/.irbrc"
MY_RUBY_HOME: "/usr/bin/arachni/bin/../system/usr/lib/ruby"
USER: root
http_proxy: ''
LD_LIBRARY_PATH: "/usr/bin/arachni/bin/../system/usr/lib"
SUDO_USER: labs
SUDO_UID: '1000'
USERNAME: root
PATH: "/usr/bin/arachni/system/gems/bin:/usr/bin/arachni/bin/../system/../bin:/usr/bin/arachni/bin/../system/usr/bin:/usr/bin/arachni/bin/../system/gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
MAIL: "/var/mail/root"
PWD: "/usr/bin/arachni/bin"
ARACHNI_WEBUI_LOGDIR: "/usr/bin/arachni/bin/../system/logs/webui"
LANG: en_US.UTF-8
ARACHNI_FRAMEWORK_LOGDIR: "/usr/bin/arachni/bin/../system/logs/framework"
https_proxy: ''
SHLVL: '0'
SUDO_COMMAND: "/bin/bash arachni_web"
HOME: "/root"
RAILS_ENV: production
DYLD_LIBRARY_PATH: "/usr/bin/arachni/bin/../system/usr/lib:"
LOGNAME: root
GEM_PATH: "/usr/bin/arachni/bin/../system/gems"
SUDO_GID: '1000'
RUBYLIB: "/usr/bin/arachni/system/gems/gems/bundler-1.7.8/lib:/usr/bin/arachni/bin/../system/usr/lib/ruby:/usr/bin/arachni/bin/../system/usr/lib/ruby/site_ruby/2.1.0:/usr/bin/arachni/bin/../system/usr/lib/ruby/2.1.0:/usr/bin/arachni/bin/../system/usr/lib/ruby/2.1.0/x86_64-linux:/usr/bin/arachni/bin/../system/usr/lib/ruby/site_ruby/2.1.0/x86_64-linux"
RUBY_VERSION: ruby-2.1.5
RACK_ENV: development
BUNDLE_GEMFILE: "/usr/bin/arachni/system/arachni-ui-web/Gemfile"
_ORIGINAL_GEM_PATH: "/usr/bin/arachni/bin/../system/gems"
BUNDLE_BIN_PATH: "/usr/bin/arachni/system/gems/gems/bundler-1.7.8/bin/bundle"
RUBYOPT: "-rbundler/setup"
MANPATH: "/usr/bin/arachni/system/gems/gems/kramdown-1.4.1/man"
BUNDLE_ORIG_MANPATH: "/usr/bin/arachni/system/gems/gems/kramdown-1.4.1/man"
--------------------------------------------------------------------------------
OPTIONS:
---
datastore:
token: 4bed8cc1e645e9697d4cd2ef8b9cf9d4
input:
values:
"(?i-mx:name)": arachni_name
"(?i-mx:user)": arachni_user
"(?i-mx:usr)": arachni_user
"(?i-mx:pass)": 5543!%arachni_secret
"(?i-mx:txt)": arachni_text
"(?i-mx:num)": '132'
"(?i-mx:amount)": '100'
"(?i-mx:mail)": arachni@email.gr
"(?i-mx:account)": '12'
"(?i-mx:id)": '1'
default_values:
"(?i-mx:name)": arachni_name
"(?i-mx:user)": arachni_user
"(?i-mx:usr)": arachni_user
"(?i-mx:pass)": 5543!%arachni_secret
"(?i-mx:txt)": arachni_text
"(?i-mx:num)": '132'
"(?i-mx:amount)": '100'
"(?i-mx:mail)": arachni@email.gr
"(?i-mx:account)": '12'
"(?i-mx:id)": '1'
without_defaults: true
force: false
browser_cluster:
pool_size: 6
job_timeout: 120
worker_time_to_live: 100
ignore_images: false
screen_width: 1600
screen_height: 1200
audit:
exclude_vector_patterns: []
include_vector_patterns: []
link_templates: []
links: true
forms: true
cookies: true
headers: false
with_both_http_methods: false
cookies_extensively: false
http:
user_agent: Arachni/v1.0.6
request_timeout: 50000
request_redirect_limit: 5
request_concurrency: 10
request_queue_size: 100
request_headers: {}
cookies: {}
proxy_host: 10.10.5.18
proxy_port: 8080
proxy_username: A320060
proxy_password: "-4t?67*swechawR!zeme"
proxy_type: http
scope:
redundant_path_patterns: {}
dom_depth_limit: 10
exclude_path_patterns: []
exclude_content_patterns: []
include_path_patterns: []
restrict_paths: []
extend_paths: []
url_rewrites: {}
include_subdomains: false
https_only: false
session: {}
checks:
- xss
- xss_dom
- xss_dom_inputs
- xss_dom_script_context
- xss_event
- xss_path
- xss_script_context
- xss_tag
platforms: []
plugins:
autothrottle:
discovery:
healthmap:
timing_attacks:
uniformity:
no_fingerprinting: false
authorized_by:
url: https://usa.cg.na.bricex.com/
--------------------------------------------------------------------------------
[2015-01-28 14:22:09 -0600] Giving up trying to audit: https://usa.cg.na.bricex.com/
[2015-01-28 14:22:09 -0600] Couldn't get a response after 5 tries.OK then, time to setup a proxy and start digging. Thanks for the feedback man.
Zapotek
commented
9 years ago The password wasn't the issue but I did notice browser comms go over the proxy when the http_proxy env variable is set.
Can you try the patch at 26f991e695af61984465b7a8d1b47d4176b40de8 and retry? It's a tiny change, you can do it manually.
The relevant file can be found at: system/gems/gems/arachni-1.0.6/lib/arachni/selenium/webdriver/remote/http/typhoeus.rb
If that doesn't fix the issue I guess I'll have to keep digging.
Cheers
bricex
commented
9 years ago Thank you for looking into this. I applied this patch and now get more consistent results. I found that I can now scan both intranet and internet sites just fine with either the http_proxy vars set, or the proxy settings specified in Archani (or both). I realized after more tests that I had another issue that was confusing things: the primary internal site I wanted to test was using a very old self-signed SSL certificate with an MD5 signature algorithm. I think this is causing connection issues because I tried with another self-signed cert using SHA1 without issue. No need to make any changes in Arachni, I will just update the old cert to get around this. If interested I set up a test environment you can scan at https://briceisapi.cloudapp.net/ where I was able to reproduce this issue by using an MD5 cert.
Again, thank you for your help with this. Do you have a donations page or similar?
Zapotek
commented
9 years ago Good to hear this fixed the issue. :)
v1.1 will include much more fine-grained control over SSL options so that may help you sort out the other issue as well.
There's no donation page though, but I do appreciate the sentiment. :)
Cheers man and thanks for all the feedback, if anything else comes up do let me know.
I am seeing a consistent issue with all scans failing right away with the message '[Selenium::WebDriver::Error::WebDriverError] unexpected response, code=301, content-type=""'. I believe this might be proxy related as the server Arachni is installed on has the http_server/https_server environment variables set to an authenticated proxy server. When I unset these values I am able to scan internal HTTP addresses, but not HTTPS (presumably because of CRL checks requiring outbound proxy support). With no proxy set I see the error "Giving up trying to audit" and "couldn't get a response after 5 tries".
I have confirmed that the proxy settings when set are correct. I am also using a self-signed certificate on the primary site I wish to scan, but this doesn't seem to be a factor from what I have observed.
The output below is from a scan when the proxy values are set. I also tried setting the proxy settings within Arachni (http options) but got the same results.
Any help with troubleshooting this further would be greatly appreciated.