search

Arachni / arachni

Web Application Security Scanner Framework
http://www.arachni-scanner.com
Other
3.76k stars 761 forks source link

Scanner seems to get stuck on certain check #650

Closed sbehrens closed 8 years ago

sbehrens commented 8 years ago

Hi Tasos,

I'm not sure exactly which plugin but lately spits out these messages indefinitely (after crawling the site for maybe 5-10 minutes). These messages continue to be printed until my scan times out (I have it set to timeout after 8 hours). The scanner finds no issues (unlike a few weeks ago where it was picking up CSRF as an example)

 [*] [HTTP: 200] http://www.netflix.com/browse
 [~] Identified as: unix, linux, mysql, pgsql, apache, jetty, nginx, tomcat, java
 [~] Analysis resulted in 0 usable paths. 
[*] Harvesting HTTP responses...
 [~] DOM depth: 1 (Limit: 3)
 [~]   Transitions:
 [~]     -- [1.1881s] load     => page (http://www.netflix.com/browse)
 [~] DOM depth: 1 (Limit: 3)
 [~]   Transitions:
 [~]     -- [1.1881s] load     => page (http://www.netflix.com/browse)
 [~] DOM depth: 1 (Limit: 3)
 [~]   Transitions:
 [~]     -- [1.1881s] load     => page (http://www.netflix.com/browse)
 [~]         * [0.7548s] request  => http://www.netflix.com/browse
 [~] Check code_injection does not support: unix + linux + mysql + pgsql + apache + jetty + nginx + tomcat + java
 [~] Check no_sql_injection does not support: unix + linux + mysql + pgsql + apache + jetty + nginx + tomcat + java
 [~] Check no_sql_injection_differential does not support: unix + linux + mysql + pgsql + apache + jetty + nginx + tomcat + java

Any idea on what this check is and why it's not able to continue scanning/crawling?

Zapotek commented 8 years ago

It looks like there's a lot of page snapshot to be audited but just contain stuff already seen. The messages you see are platform related, those checks don't support the identified platforms so the system just lets you know.

Does the scan actually get stuck or consumes its workload until the given timeout is reached and then exits cleanly?

sbehrens commented 8 years ago

it consumes its workload until the given timeout is reached and then exits cleanly.

Zapotek commented 8 years ago

That's good, if you just get a lot of snapshots for the /browse page you probably need to specify redundancy rules for it.

I'm assuming that page changes a lot during the scan, right?

sbehrens commented 8 years ago

Good point, yes, it can change a ton. I'll try tuning that and will re-open the issue if I see anything odd.

On Mon, Dec 14, 2015 at 12:11 PM, Tasos Laskos notifications@github.com wrote:

That's good, if you just get a lot of snapshots for the /browse page you probably need to specify redundancy rules for it.

I'm assuming that page changes a lot during the scan, right?

— Reply to this email directly or view it on GitHub https://github.com/Arachni/arachni/issues/650#issuecomment-164545644.

sbehrens commented 8 years ago

Hi Tasos,

The scanner isn't turning on /browse anymore however it's finishing scanning www.netflix.com in 50 seconds to 2 minutes. I have been using a config to reduce redundant paths and what not but it seems to not be able to crawl the majority of the site (as an example it never finds www.netflix.com/YourAccount).

the config I've been using is below (slightly redacted with users/passwords)

---
audit:
  exclude_vector_patterns:
  - "(?-mix:authURL)"
  include_vector_patterns: []
  link_templates: []
  links: true
  forms: true
  cookies: false
  headers: false
  with_both_http_methods: false
  cookies_extensively: false
browser_cluster:
  pool_size: 6
  job_timeout: 120
  worker_time_to_live: 100
  ignore_images: true
  screen_width: 1600
  screen_height: 1200
datastore: {}
http:
  user_agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML,
    like Gecko) Chrome/37.0.2049.0 Safari/537.36
  request_timeout: 50000
  request_redirect_limit: 5
  request_concurrency: 25
  request_queue_size: 700
  request_headers: {}
input:
  values:
    "(?i-mx:name)": arachni_name
    "(?i-mx:user)": ACCOUNT@netflix.com
    "(?i-mx:usr)": ACCOUNT@netflix.com
    "(?i-mx:pass)": PASSWORD
    "(?i-mx:password)": PASSWORD
    "(?i-mx:txt)": arachni_text
    "(?i-mx:num)": '132'
    "(?i-mx:amount)": '100'
    "(?i-mx:mail)": ACCOUNT@netflix.com
    "(?i-mx:account)": '12'
    "(?i-mx:id)": '1'
  without_defaults: true
  force: false
scope:
  redundant_path_patterns:
    "(?-mix:Movie)": '10'
    "(?-mix:load.story)": '5'
    "(?-mix:beacons)": '5'
    "(?-mix:browse)": '5'
    "(?-mix:Redeem)": '5'
    "(?-mix:userprefs)": '5'
    "(?-mix:api/desktop/account)": '5'
    "(?-mix:api/desktop/notifications/load)": '5'
    "(?-mix:api/ecapi/account/myserviceinfo)": '5'
    "(?-mix:api/ecapi/account/v2/payment)": '5'
    "(?-mix:SubGenreList)": '5'
    "(?-mix:WiRecentAdditionsGallery)": '2'
    "(?-mix:WiGenre)": '3'
    "(?-mix:BOB)": '15'
    "(?-mix:beacons)": '5'
    "(?-mix:TasteCategoryExamples)": '5'
    "(?-mix:WiPlayer)": '10'
    "(?-mix:WiMovie)": '10'
    "(?-mix:bob)": '15'
    "(?-mix:WiMemberReviews)": '10'
    "(?-mix:RaitingsWizard)": '10'
    "(?-mix:LogCustomerEvent)": '10'
    "(?-mix:KidsAltGenre)": '10'
    "(?-mix:KidsMovie)": '10'
    "(?-mix:support)": '10'
  dom_depth_limit: 14
  exclude_path_patterns:
  - "(?-mix:sign out)"
  - "(?-mix:secure.netflix.com)"
  - "(?-mix:pr.netflix.com)"
  - "(?-mix:blog)"
  - "(?-mix:Sign out)"
  - "(?-mix:signout)"
  - "(?-mix:SignOut)"
  - "(?-mix:Signout)"
  - "(?-mix:Logout)"
  - "(?-mix:CancelPlan)"
  - "(?-mix:Foreign_Languages)"
  exclude_content_patterns: []
  include_path_patterns: []
  restrict_paths: []
  extend_paths: []
  url_rewrites: {}
  include_subdomains: false
  auto_redundant_paths: 4
  https_only: false
checks:
- code_injection
- code_injection_timing
- csrf
- file_inclusion
- no_sql_injection
- no_sql_injection_differential
- os_cmd_injection
- os_cmd_injection_timing
- path_traversal
- response_splitting
- rfi
- source_code_disclosure
- sql_injection
- sql_injection_differential
- sql_injection_timing
- trainer
- unvalidated_redirect
- xpath_injection
- xss
- xss_dom
- xss_dom_script_context
- xss_event
- xss_path
- xss_script_context
- xss_tag
- allowed_methods
- credit_card
- cvs_svn_users
- directory_listing
- form_upload
- htaccess_limit
- html_objects
- http_only_cookies
- http_put
- insecure_cookies
- interesting_responses
- mixed_resource
- origin_spoof_access_restriction_bypass
- private_ip
- unencrypted_password_forms
- webdav
- xst
platforms:
- unix
- linux
- apache
- mysql
- pgsql
- jetty
- nginx
- tomcat
- java
no_fingerprinting: false
plugins:
  autologin:
    url: https://www.netflix.com/globallogin
    parameters: email=redacted@netflix.com&password=redacted
    check: "."
authorized_by:

I have tried using different versions of the nightlies, and I'm still unsure why. A few weeks ago with this config I was getting a ton more results/findings and the sitemap looked more complete.

Zapotek commented 8 years ago

I'll have a look at it, I remember getting to the account page when I tried it a few days ago. It's probably an issue with fine-tuning the redundancy rules.

sbehrens commented 8 years ago

Thanks Tasos, I wasn't sure if it was a change to the crawler or something else. Please let me know what you find out.

Zapotek commented 8 years ago

There have been changes but in the underlying libraries, the crawl behavior is mostly the same. Coverage has actually improved, which is probably why you're getting more workload and some tweaking is necessary to get to the pages you're interested in within the given timeout period.

For now, I'd suggest experimenting with the redundancy counters, try to increase them and see if you're getting the desired results that way.

sbehrens commented 8 years ago

Hi tasos,

I went ahead and removed all redundant path rules "except limiting /browse since the scanner never gets past that link", and scans are still finishing in 2 minutes. The config now looks like this:

---
audit:
  exclude_vector_patterns:
  - "(?-mix:authURL)"
  include_vector_patterns: []
  link_templates: []
  links: true
  forms: true
  cookies: false
  headers: false
  with_both_http_methods: false
  cookies_extensively: false
browser_cluster:
  pool_size: 6
  job_timeout: 120
  worker_time_to_live: 100
  ignore_images: true
  screen_width: 1600
  screen_height: 1200
datastore: {}
http:
  user_agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML,
    like Gecko) Chrome/37.0.2049.0 Safari/537.36
  request_timeout: 50000
  request_redirect_limit: 5
  request_concurrency: 25
  request_queue_size: 700
  request_headers: {}
input:
  values:
    "(?i-mx:name)": arachni_name
    "(?i-mx:user)": ACCOUNT@netflix.com
    "(?i-mx:usr)": ACCOUNT@netflix.com
    "(?i-mx:pass)": PASSWORD
    "(?i-mx:password)": PASSWORD
    "(?i-mx:txt)": arachni_text
    "(?i-mx:num)": '132'
    "(?i-mx:amount)": '100'
    "(?i-mx:mail)": ACCOUNT@netflix.com
    "(?i-mx:account)": '12'
    "(?i-mx:id)": '1'
  without_defaults: true
  force: false
scope:
  redundant_path_patterns:
    "(?-mix:browse)": '15'
  dom_depth_limit: 14
  exclude_path_patterns:
  - "(?-mix:sign out)"
  - "(?-mix:secure.netflix.com)"
  - "(?-mix:pr.netflix.com)"
  - "(?-mix:blog)"
  - "(?-mix:Sign out)"
  - "(?-mix:signout)"
  - "(?-mix:SignOut)"
  - "(?-mix:Signout)"
  - "(?-mix:Logout)"
  - "(?-mix:CancelPlan)"
  - "(?-mix:Foreign_Languages)"
  exclude_content_patterns: []
  include_path_patterns: []
  restrict_paths: []
  extend_paths: []
  url_rewrites: {}
  include_subdomains: false
  auto_redundant_paths: 4
  https_only: false
checks:
- code_injection
- code_injection_timing
- csrf
- file_inclusion
- no_sql_injection
- no_sql_injection_differential
- os_cmd_injection
- os_cmd_injection_timing
- path_traversal
- response_splitting
- rfi
- source_code_disclosure
- sql_injection
- sql_injection_differential
- sql_injection_timing
- trainer
- unvalidated_redirect
- xpath_injection
- xss
- xss_dom
- xss_dom_script_context
- xss_event
- xss_path
- xss_script_context
- xss_tag
- allowed_methods
- credit_card
- cvs_svn_users
- directory_listing
- form_upload
- htaccess_limit
- html_objects
- http_only_cookies
- http_put
- insecure_cookies
- interesting_responses
- mixed_resource
- origin_spoof_access_restriction_bypass
- private_ip
- unencrypted_password_forms
- webdav
- xst
platforms:
- unix
- linux
- apache
- mysql
- pgsql
- jetty
- nginx
- tomcat
- java
no_fingerprinting: false
plugins:
  autologin:
    url: https://www.netflix.com/globallogin
    parameters: email=redacted@netflix.com&password=redacted
    check: "."
authorized_by:

When proxying the scan and viewing what it's hitting, it seems to indicate that it's not hitting the majority of the site (sitemap below):

http://www.netflix.com/
http://www.netflix.com/browse
http://www.netflix.com/api/shakti
http://www.netflix.com/api
http://www.netflix.com/PrivacyPolicy
http://www.netflix.com/TermsOfUse
https://www.netflix.com/TermsOfUse
https://www.netflix.com/api/shakti/4649b6b7/esn?authURL=1449867506405.fKVOhs+zNpBNuxIm8lOVgXxaK44=
http://www.netflix.com/watch/80044545?trackId=13462260&tctx=0,0,cadc6a00-b8db-4b75-bc23-3aba61946c4b-23351925
http://www.netflix.com/browse/audio-description
http://www.netflix.com/browse/genre/11714
http://www.netflix.com/browse/genre/6548
http://www.netflix.com/browse/set-cookies-7272ea52ede3e3f43cc0dddf43485e32
https://www.netflix.com/api/shakti/4649b6b7/esn?authURL=1449867506322.luleJLTpNgP3HJUvVVjvzraoHk4=
https://www.netflix.com/api/shakti/4649b6b7/esn?authURL=1449867510759.SilanCJTWdf8UIzGsxMT4UcdJWU=
http://www.netflix.com/player/silverlight/Player-SL5-4.2338.000.0.xap

The sitemap from the scans I ran before upgrading to the nightlies is below:

http://www.netflix.com/
http://www.netflix.com/Activate
http://www.netflix.com/AddPlan
http://www.netflix.com/EditProfiles
http://www.netflix.com/EmailPreferences
http://www.netflix.com/Gift/Select
http://www.netflix.com/Kids
http://www.netflix.com/ManageFB
http://www.netflix.com/MoviesYouveSeen
http://www.netflix.com/MyList
http://www.netflix.com/NotFound
http://www.netflix.com/NotFound/
http://www.netflix.com/NotFound/%3Cmy_tag_58c24c67a0fc4ea87cfb4a768e64ff40/%3E
http://www.netflix.com/NotFound/%3E%22'%3E%3Cmy_tag_58c24c67a0fc4ea87cfb4a768e64ff40/%3E
http://www.netflix.com/NotFound/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
http://www.netflix.com/Privacy
http://www.netflix.com/PrivacyPolicy
http://www.netflix.com/PrivacyPolicy/%3Cmy_tag_58c24c67a0fc4ea87cfb4a768e64ff40/%3E
http://www.netflix.com/PrivacyPolicy/%3E%22'%3E%3Cmy_tag_58c24c67a0fc4ea87cfb4a768e64ff40/%3E
http://www.netflix.com/PrivacyPolicy/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
http://www.netflix.com/Reviews
http://www.netflix.com/Subtitles
http://www.netflix.com/TermsOfUse
http://www.netflix.com/TermsOfUse/%3Cmy_tag_58c24c67a0fc4ea87cfb4a768e64ff40/%3E
http://www.netflix.com/TermsOfUse/%3E%22'%3E%3Cmy_tag_58c24c67a0fc4ea87cfb4a768e64ff40/%3E
http://www.netflix.com/TermsOfUse/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
http://www.netflix.com/Watch
http://www.netflix.com/WiAltGenre
http://www.netflix.com/WiGenre
http://www.netflix.com/WiHome
http://www.netflix.com/WiSearch
http://www.netflix.com/WiViewingActivity
http://www.netflix.com/YourAccount
http://www.netflix.com/api
http://www.netflix.com/api/shakti
http://www.netflix.com/browse
http://www.netflix.com/browse/audio-description
http://www.netflix.com/browse/genre/10375
http://www.netflix.com/browse/genre/11559
http://www.netflix.com/browse/genre/1365
http://www.netflix.com/browse/genre/7424
http://www.netflix.com/browse/genre/83
http://www.netflix.com/browse/my-list
http://www.netflix.com/browse/subtitle/en
http://www.netflix.com/confirm
http://www.netflix.com/gift
http://www.netflix.com/kid
http://www.netflix.com/kid/search
http://www.netflix.com/player/silverlight/Player-SL5-4.2338.000.0.xap
http://www.netflix.com/privacy
http://www.netflix.com/privacy/%3E%22'%3E%3Cmy_tag_58c24c67a0fc4ea87cfb4a768e64ff40/%3E
http://www.netflix.com/search/null
http://www.netflix.com/uitracking/users/presentationtracking
https://www.netflix.com/
https://www.netflix.com/.member-header.2014
https://www.netflix.com/.nav-item
https://www.netflix.com/.nav-wrap
https://www.netflix.com/Activate
https://www.netflix.com/AddPlan
https://www.netflix.com/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
https://www.netflix.com/BillingActivity
https://www.netflix.com/ChangePlan
https://www.netflix.com/DoNotTest
https://www.netflix.com/EULA
https://www.netflix.com/EditProfiles
https://www.netflix.com/EmailPreferences
https://www.netflix.com/FilePackageGetter/sharedSystem/pkg-nflxrsrc-3a9180a2866041c0200000000000080020016043e00008093404e3100001a1c-s-1446590188-15-15.css
https://www.netflix.com/FilePackageGetter/sharedSystem/pkg-nflxrsrc-3a9180a28866041c0262000000000280c20016043e00008093404e1100001a1c-s-1446590188-15-15.css
https://www.netflix.com/FilePackageGetter/sharedSystem/pkg-nflxrsrc-fff4161e29843b1c38bffc00008000001c00f2b07bd0560c11c41d0dcc5ec00a8a138924600a010040613f1fc00-s-1446590188-15-15.js
https://www.netflix.com/FilePackageGetter/sharedSystem/pkg-nflxrsrc-fff4161e29843b5e38bffc06008000001c00f2b07bd1560c01c41d0dcc5ec00a8a1389247e01ca010040013f1fc00-s-1446590188-15-15.js
https://www.netflix.com/Gift/Select
https://www.netflix.com/GiftHistory
https://www.netflix.com/GiftTerms
https://www.netflix.com/HdToggle
https://www.netflix.com/LanguagePreferences
https://www.netflix.com/Login
https://www.netflix.com/LoginHelp
https://www.netflix.com/ManageDevices
https://www.netflix.com/ManageFB
https://www.netflix.com/MoviesYouveSeen
https://www.netflix.com/MyListOrder
https://www.netflix.com/NotFound
https://www.netflix.com/Notices
https://www.netflix.com/PopupDetails
https://www.netflix.com/Privacy
https://www.netflix.com/Privacy%0A
https://www.netflix.com/Privacy%0A/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
https://www.netflix.com/PrivacyPolicy
https://www.netflix.com/Redeem
https://www.netflix.com/Reviews
https://www.netflix.com/SocialTerms
https://www.netflix.com/SocialTerms/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
https://www.netflix.com/SubtitlePreferences
https://www.netflix.com/TermsOfUse
https://www.netflix.com/TermsOfUseChanges
https://www.netflix.com/Watch
https://www.netflix.com/WhySecure
https://www.netflix.com/WiHome
https://www.netflix.com/WiMovie/70044256
https://www.netflix.com/WiMovie/70204316
https://www.netflix.com/WiMovie/80029368
https://www.netflix.com/WiMovie/80043890
https://www.netflix.com/WiSearch
https://www.netflix.com/WiViewingActivity
https://www.netflix.com/YourAccount
https://www.netflix.com/YourAccountPayment
https://www.netflix.com/a
https://www.netflix.com/api
https://www.netflix.com/api/account/subscription/restartplan
https://www.netflix.com/api/desktop/search/instantsearch
https://www.netflix.com/api/ecapi/account/giftsubhistory
https://www.netflix.com/api/servicecode
https://www.netflix.com/api/shakti
https://www.netflix.com/api/shakti/0022ef63/densityKids/searchAutocomplete
https://www.netflix.com/appsyouraccount/secure/index.do
https://www.netflix.com/beacons
https://www.netflix.com/browse
https://www.netflix.com/cardlocator
https://www.netflix.com/confirm
https://www.netflix.com/copyrights
https://www.netflix.com/copyrights/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
https://www.netflix.com/css/include
https://www.netflix.com/dvdterms
https://www.netflix.com/email
https://www.netflix.com/fblogin
https://www.netflix.com/gift
https://www.netflix.com/help
https://www.netflix.com/ichnaea/log
https://www.netflix.com/js/include
https://www.netflix.com/login
https://www.netflix.com/modernizr
https://www.netflix.com/notfound
https://www.netflix.com/password
https://www.netflix.com/patents
https://www.netflix.com/patents/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
https://www.netflix.com/phonenumber
https://www.netflix.com/pin
https://www.netflix.com/priorterms
https://www.netflix.com/priorterms/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
https://www.netflix.com/privacy
https://www.netflix.com/privacypolicy
https://www.netflix.com/privacypolicychanges
https://www.netflix.com/privacyupdates
https://www.netflix.com/privacyupdates/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
https://www.netflix.com/redeem
https://www.netflix.com/rememberme
https://www.netflix.com/storelocator
https://www.netflix.com/storelocator/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
https://www.netflix.com/tel:1-800-585-7265
https://www.netflix.com/tel:1-800-585-7265/Arachni-58c24c67a0fc4ea87cfb4a768e64ff40
https://www.netflix.com/termsofuse
https://www.netflix.com/torii/layouts/basicLayout.jsx
https://www.netflix.com/torii/nonmemberHome.jsx
https://www.netflix.com/watch
https://www.netflix.com/yourAccount

This seems to indicate either a change on our site or some issue with the crawler.

Zapotek commented 8 years ago

Running scans now to see what's going on.

Zapotek commented 8 years ago

Any chance you got cut off by an IDS? I'm getting this:

            <h1>Netflix Site Error</h1>
            <p>We were unable to process your request.</p>
            <p>Please go to the Netflix home page by clicking the button below.</p>
            <div class="nflxButton">
                <a href="/">Netflix Home</a>
            </div>

First scan worked, then I got throttled way down and aborted, when I retried but all I kept getting was the above.

sbehrens commented 8 years ago

I am white listing my scan ip. If you want to send me an IP via email I'll white list you to

On Tuesday, December 15, 2015, Tasos Laskos notifications@github.com wrote:

Any chance you got cut off by an IDS? I'm getting this:

        <h1>Netflix Site Error</h1>
        <p>We were unable to process your request.</p>
        <p>Please go to the Netflix home page by clicking the button below.</p>
        <div class="nflxButton">
            <a href="/">Netflix Home</a>
        </div>

First scan worked, then I got throttled way down and aborted, when I retried but all I kept getting was the above.

— Reply to this email directly or view it on GitHub https://github.com/Arachni/arachni/issues/650#issuecomment-164928480.

Zapotek commented 8 years ago

Sent an e-mail yesterday btw.

sbehrens commented 8 years ago

The IP you sent is now whitelisted.

On Tue, Dec 15, 2015 at 4:11 PM, Scott Behrens sbehrens@gmail.com wrote:

I am white listing my scan ip. If you want to send me an IP via email I'll white list you to

On Tuesday, December 15, 2015, Tasos Laskos notifications@github.com wrote:

Any chance you got cut off by an IDS? I'm getting this:

        <h1>Netflix Site Error</h1>
        <p>We were unable to process your request.</p>
        <p>Please go to the Netflix home page by clicking the button below.</p>
        <div class="nflxButton">
            <a href="/">Netflix Home</a>
        </div>

First scan worked, then I got throttled way down and aborted, when I retried but all I kept getting was the above.

— Reply to this email directly or view it on GitHub https://github.com/Arachni/arachni/issues/650#issuecomment-164928480.

Zapotek commented 8 years ago

I managed to reproduce the issue; you mentioned the site changing, is that possible?

The current drop-down links wouldn't have been visible to Arachni, ever. They don't just change visibility onmouseover (wouldn't have been an issue), the HTML gets written onmouseover. Problem is, that event won't be triggered because it's not currently seen -- and won't be until #652.

So, if previously the menu's entries just changed visibility they would have been seen and followed, and that would explain the difference in coverage.

sbehrens commented 8 years ago

HI Tasos,

Any update on this? No matter what I try the scanner just hits www.netflix.com/browse forever and then the scan finishes. never crawls 90% of the site.

] Depending on server responsiveness and network conditions this may take a while.
 [*] Got new page from the browser-cluster: http://www.netflix.com/browse
 [~] DOM depth: 1 (Limit: 14)
 [~]   Transitions:
 [~]     -- [1.2561s] load     => page (http://www.netflix.com/browse)
 [~]         * [0.8356s] request  => http://www.netflix.com/browse
 [*] Got new page from the browser-cluster: http://www.netflix.com/browse
 [~] DOM depth: 1 (Limit: 14)
 [~]   Transitions:
 [~]     -- [1.2561s] load     => page (http://www.netflix.com/browse)
 [~]         * [0.8356s] request  => http://www.netflix.com/browse
 [~] Check code_injection does not support: unix + linux + mysql + pgsql + apache + jetty + nginx + tomcat + java
 [~] Check no_sql_injection does not support: unix + linux + mysql + pgsql + apache + jetty + nginx + tomcat + java
 [~] Check no_sql_injection_d

Is there a way I can disable whatever check is causing the issue? I tried tweaking auto-redundancy but that didn't help either.

Zapotek commented 8 years ago

Did you miss my previous reply?

sbehrens commented 8 years ago

Hi Tasos,

I guess I don't understand. So basically a change is needed to stop it from constantly hitting /browse? Any way to work around the issue until you get that feature in?

On Tue, Dec 29, 2015 at 9:55 AM, Tasos Laskos notifications@github.com wrote:

Did you miss my previous reply?

— Reply to this email directly or view it on GitHub https://github.com/Arachni/arachni/issues/650#issuecomment-167843690.

Zapotek commented 8 years ago

The /browse thing can be resolved with a redundancy rules. The problem is with the fact that something probably changed with the way the user drop-down menu is rendered in the page (the one that points to http://www.netflix.com/YourAccount), causing Arachni to no longer see the links it contains.

For now, you can specify the paths in the drop-down menu via --scope-extend-paths and set redundancy limits for /browse, that combo should work.