Browser crawl not working

r3t4k3r commented 1 month ago

Hi, i have a problem when i use max_depth param in my config.

I use owasp benchmark to test this scanner. i run this command wscan --config /requirements/wscan-config.yaml ws --browser https://bench:8443/benchmark Crawler found this page and pages on it without problems.

But cannot go to this pages

I use this config flags

browser-crawler:
    exec_path: "/usr/sbin/chromium"
    disable_headless: false
    force_sandbox: true
    enable_image: true
    parent_path_detect: true
    proxy: ""
    user_agent: "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/124.0.6344.218 Chrome/124.0.6344.218 Safari/537.36"
    domain_headers: []
    max_depth: 100
    navigate_timeout_second: 100
    load_timeout_second: 100
    retry: 3
    page_analyze_timeout_second: 100
    max_interactive: 100
    max_interactive_depth: 100
    max_page_concurrent: 10
    max_page_visit: 100000
    max_page_visit_per_site: 100000
    element_filter_strength: 0
    restriction:
        hostname_allowed: []
        hostname_disallowed:
            - '*google*'
            - '*github*'
            - '*.gov.cn'
            - '*.edu.cn'
        port_allowed: []
        port_disallowed: []
        path_allowed: []
        path_disallowed: []
        query_key_allowed: []
        query_key_disallowed: []
        fragment_allowed: []
        fragment_disallowed: []
        post_key_allowed: []
        post_key_disallowed: []

U can see my console log here: wscan.log

r3t4k3r commented 1 month ago

After some tests, i understand this no run browser at all. I use archlinux docker container with last chromium package, but it still no working, with /usr/sbin/chromium and /usr/sbin/cromedriver in config exec_path. Then i use /usr/sbin/chromedirver app exit with error [FTAL] 2024-05-23 15:36:54 websocket url timeout reached Then I use /usr/sbin/chromium just exit with 0 found urls

If no hard to you, can you write me correct Dockerfile where --browser option work correct? Thanks for you work.

chushuai commented 1 month ago

@r3t4k3r Is there a public test website available? I would like to test it. Alternatively, you can use the --log-level=debug parameter to output debug logs.

r3t4k3r commented 1 month ago

@chushuai thanks for you answer

@r3t4k3r Is there a public test website available?

Yes, but you need start it locally, I use this docker-compose.yml to test it

services:
  bench:
    image: owasp/benchmark
    container_name: bench
    hostname: bench
    command: '/owasp/BenchmarkJava/runRemoteAccessibleBenchmark.sh'
    ports:
      - 8443:8443

Run docker-compose up -d wait 1 minute, after this open https://127.0.0.1:8443/benchmark/ More about owasp benchmark you can read here: https://owasp.org/www-project-benchmark/

Alternatively, you can use the --log-level=debug parameter to output debug logs.

I know, i already do it, but it's no provided me more useful information

r3t4k3r commented 1 month ago

I finally managed to launch the browser (Chromium 125.0.6422.60 Arch Linux). But crawl still doesn't work, the browser starts, but no action happens

chushuai / wscan

Browser crawl not working #14