Logs displays HTTPS referrers incorrectly, strips valid characters

[GoogleCodeExporter]

On Visitors Logs 'http://' is added in front of any referrer URL. The problem 
arises from the cleaning done in logit().

What logit() does, it:

* Removes http:// from the referrer.
* And runs the URL through clean_url().

This causes few issues which lead to invalid logged referrers:

* Ignores any other protocols.
* Removes any occurrence of 'http://', altering the referrer.
* Strips control characters. Control characters are removed to offer security, 
but can also cause transformations. Although we need to keep it unless we do 
actual real filtering.

Changes:

* Store the protocol with the URL.
* Update the filtering to work with different protocols.
* Block any referrer using different protocol than http or https. We don't want 
elevate exploitable binds through trusted sites.

Original issue reported on code.google.com by jukka.m.svahn on 11 Nov 2012 at 7:06

[GoogleCodeExporter]

This issue was closed by revision r4681.

Original comment by jukka.m.svahn on 11 Nov 2012 at 3:24

• Changed state: Fixed

[GoogleCodeExporter]

The current solution from r4681 breaks sorting by referrer on the 'Visitor 
logs' tab. 

It coalesces all 'https://.*' referrers and all 'http://.*' referrers into 
these two groups by protocol.

MySQL has no function to help us ignore the protocol part for sorting and 
searching.

I wonder if it wouldn't be wiser to add a separate 'protocol' column to txp_log 
and keep the stripped URL in the 'refer' column like it was in 4.5. 

This would also restore the protocol-less referrer display on the admin-side 
and thus reduce clutter, assuming that the dominant fraction of referrers will 
be 'http://.*' anyway so displaying the protocol part does not add any useful 
information.

Original comment by r.wetzlmayr on 17 Dec 2012 at 12:07

• Changed state: Assigned

[GoogleCodeExporter]

I don't believe it 'breaks' sorting. The protocol is part of the referrer, and 
as such it should sort by it too.

If you see that it needs amendments, feel free.

Original comment by jukka.svahn@rahinaa.biz on 17 Dec 2012 at 12:16

[GoogleCodeExporter]

Original comment by jukka.m.svahn on 17 Dec 2012 at 12:19

[GoogleCodeExporter]

This issue was closed by revision r5075.

Original comment by r.wetzlmayr on 17 Dec 2012 at 1:13

• Changed state: Fixed