Merge crawler-integration

TeamHG-Memex / autologin

A project to attempt to automatically login to a website given a single seed

Apache License 2.0

123 stars 44 forks source link

Merge crawler-integration #9

Closed lopuhin closed 8 years ago

lopuhin commented 8 years ago

This branch brings some improvements:

search for login and registration forms
UI for managing login credentials
HTTP API with the ability to get login credentials by url
optional splash support, which adds support for login on Tor sites, and simplifies using autologin from a splash-enabled crawler

Still, there are some unresolved issues:

[x] Usage as a library (see discussion in #6)
[x] Update documentation, check installation
[x] Add tests, add to travis
[x] Cleanup, remove dead code

lopuhin commented 8 years ago

The most non-obvious issue is library support: right now we use scrapy to fetch pages, find login and registration forms, and log in. It is possible to hide it all behind a synchronous API, the only caveat is that AFAIK there should be only one twisted event loop in a thread, so we'll have to offload all autologin operation into a different thread. Does it sound like a sane approach?

lopuhin commented 8 years ago

Done in #13