search

rflechner / ScrapySharp

reborn of https://bitbucket.org/rflechner/scrapysharp
MIT License
346 stars 75 forks source link

302 Redirect after Login #15

Open imukai opened 5 years ago

imukai commented 5 years ago

I can successfully populate and submit a login form with Scrapy. As can be common with logins the server sends back a 302 redirect to a main page that Scrapy sees as an error (a WebException).

I tried enabling .AllowAutoRedirect as well as AllowMetaRedirect on the ScrapingBrowser object with no change in behavior.

Please advise.

imukai commented 5 years ago

My current workaround is to trap for the exception and manually call the next page, but this seems less than ideal as relying on exceptions for normal flow is usually bad.

catch (AggregateException ex) {
    var inner = ex.InnerExceptions.First() as WebException;
    redoUrl = inner?.Response?.Headers["Location"];
    }
dararish commented 5 years ago

@imukai as far as I can tell, HttpWebRequest throws an exception whenever a non-200 status code is returned. See #19 for a fix.