search

TeamHG-Memex / autologin

A project to attempt to automatically login to a website given a single seed
Apache License 2.0
123 stars 44 forks source link

Splash support #6

Closed lopuhin closed 8 years ago

lopuhin commented 8 years ago

Will solve #5 and will also allow to support phpbb3-style sessions that are tied to user-agent and ip.

I see two ways to implement it:

I'm still not sure which is better... The first option is more self-contained.

madisonb commented 8 years ago

I dont understand why Splash is needed in order to support phpbb3 style cookies? If autologin requires Splash, then it is no longer really a python module and requires greater architecture for it to function. While I am not well versed in phpbb3 style cookies - I do not see why faking a header request with all of the proper information cannot be done - which is pretty easy in Scrapy.

We have been very happy with integrating autologin in our scraping architecture, and I think the best use of the module will be to make it standalone as much as possible.

lopuhin commented 8 years ago

Thanks for the feedback, @madisonb! Do you use autologin as a library to get the request data and then send it with Scrapy?

The situation where splash support is helpful is when we use autologin as a service, perhaps even on a different host, and also crawl via a separate splash instance. In this case by using the same splash instance both in autologin and in the crawler we get the same ip and the same user-agent, and can also log in on sites that are hard to handle without splash (js heavy or tor).

lopuhin commented 8 years ago

Just to clarify - splash support it intended to be optional, not a requirement.

madisonb commented 8 years ago

Precisely, we use autologin/formasaurus in library form and but could switch over to autologin as a service if needed, and then use the cookies generated within Scrapy. We dont use Splash instances to crawl the open web, and for Tor we have our spiders configured to work with the network.

Most sites in the past have not cared whether the cookie comes from a different IP, but the phpbb3 sites may and we may need extra engineering for work with that.

lopuhin commented 8 years ago

This is done in #8 by using scrapy and scrapy-splash.