zrashwani / arachnid

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
MIT License
253 stars 60 forks source link

Response 401 - Authentification #15

Open merolhack opened 8 years ago

merolhack commented 8 years ago

Hi, i need authentification against LDAP vía HTTP Auth, and it gives me an 401 status code.

How i can do this using 'CookieJar'? Like in the comment: http://zrashwani.com/simple-web-spider-php-goutte/#comment-92

It gives me:

Array
(
    [http://somehost] => Array
        (
            [links_text] => Array
                (
                    [0] => BASE_URL
                )
            [absolute_url] => http://somehost
            [frequency] => 1
            [visited] => 
            [external_link] => 
            [original_urls] => Array
                (
                    [0] => http://somehost
                )
            [status_code] => 401
        )
)