New URLs plugin - Githubissues

UltrosBot / Ultros

Connecting communities, one squid at a time! Ultros is a multi-protocol chat bot written in Python, designed with both the user and developer in mind

http://ultros.io

Artistic License 2.0

23 stars 7 forks source link

New URLs plugin #75

Closed gdude2002 closed 8 years ago

gdude2002 commented 9 years ago

Oh boy, this won't merge nicely will it.. That's going to be annoying.

A much-improved URLs plugin with a proper handler and shortener system, asynchronous requests using txrequests, and full sessions/cookies support.

I'll attempt to merge this myself at some point, but for now, I'm collecting opinions.

rakiru commented 9 years ago

One merge conflict is tox.ini, where the same change was made in both branches. Either version can be taken for the same result. The other conflict is due to a bug patch in the old URLs plugin in master, and shouldn't be a problem as long as the empty header value bug wasn't reintroduced.

[x] plugins/urls/__init__old.py needs removed.
[x] Add version: 2 to config.yml. This will help alleviate any problems if it should change in future.
[x] catcher/old config warning should use a single call to the logger, not one per line.
[x] Update URL-Tools.
[x] Handle non-200 status codes. (g edit: Handles redirects)

TODO by g:

[x] Handler priority constants
[x] Channels using default shortener should use changed default
[ ] Logger support for Failures
[x] Asynchronous DNS

gdude2002 commented 9 years ago

Things still left to do:

[x] Update URL-Tools
[x] Channels using default shortener should respect changes to the default
[ ] Logger support for Failures (Separate commit set?)
[x] Check portscans on redirect
[x] Redirect detection before handlers are run
- Keep a whitelist of URL shorteners that may be configured; a request should be created that doesn't handle redirects itself and should be handled manually until we get to a domain that isn't in the aforementioned whitelist

rakiru commented 8 years ago

A few more shorteners:

  - "gg.gg"
  - "dickensurl.com"
  - "5z8.info"  # shadyurl.com

v.gd exists too, but it gives you a preview page by default. It can be disable by adding a cookie called "preview" with a value of 1, but I don't know if you want to add something to the list that relies on that.

gdude2002 commented 8 years ago

It can be disable by adding a cookie called "preview" with a value of 1

Well, that Session doesn't use any cookie stores right now; but now that you mention this, since v.gd and is.gd are the same people, maybe that'd work on the dumb is.gd gated dropbox links?

rakiru commented 8 years ago

Another URL shortener: "tiny.cc"

Edit: Also "smarturl.it" and "hyperurl.co".

rakiru commented 8 years ago

:+1:

gdude2002 commented 8 years ago

No need to PEP8 this, flake8 says that everything's good.